fix(drift): flag new voice/audio model families beyond the "realtime" substring#292
Merged
Merged
Conversation
… substring
The ws-realtime known-models canary filtered GET /v1/models with
`models.filter((m) => m.includes("realtime"))`, so a NEW full-duplex voice
family whose id lacks the "realtime" substring (e.g. OpenAI's gpt-live-1 /
gpt-live-1-mini) never entered the unknown-model computation and slipped past
the canary silently. The general models.drift.ts canary only detects
deprecation of already-referenced models, not the arrival of new ones — this
was the blind spot.
Extract the detection into a side-effect-free module (voice-models.ts) with a
broadened voice/audio family matcher (realtime|audio|live|transcribe|whisper|
voice|tts) and a knownVoiceModels allowlist. Both the live canary and a new
unit test drive the SAME detectVoiceModelDrift code path. The canary's critical
-vs-crash markers (NO_GA_REALTIME_MODELS= / UNKNOWN_REALTIME_MODELS=, added in
1e3e638) are preserved verbatim, so the drift collector still emits exit-2
critical drift rather than crashing.
Generalizes beyond gpt-live: any unseen voice/audio family is flagged the first
time it appears; chat/image/embedding models never become false positives.
commit: |
… false positives The broadened voice/audio matcher correctly widened coverage to catch new families like gpt-live, but it compared full model ids against a known-ID set. OpenAI ships dated snapshots of already-known families constantly, so the live Drift Tests run (28968203340) flagged 7 legitimate snapshots as critical drift (tts-1-1106, tts-1-hd-1106, gpt-audio-2025-08-28, gpt-4o-mini-transcribe-*, gpt-4o-mini-tts-*). Appending each new snapshot never converges — the daily job would go red every day and bury a real new-family signal. Fix: normalize each candidate id to a FAMILY KEY by stripping trailing dated snapshot (-YYYY-MM-DD) and build-tag (-NNN/-NNNN) suffixes, then compare the normalized family against a set of known FAMILIES. Dated snapshots of a known family collapse onto it and stay green; a genuinely new family (gpt-live, whose single-digit -1 is deliberately not stripped) still normalizes to an unknown family and is flagged. The NO_GA_REALTIME_MODELS= / UNKNOWN_REALTIME_MODELS= markers in ws-realtime.drift.ts are preserved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The ws-realtime known-models drift canary (
src/__tests__/drift/ws-realtime.drift.ts) originally filtered theGET /v1/modelsresult withmodels.filter((m) => m.includes("realtime")), so a NEW full-duplex voice family whose id lacks the"realtime"substring — e.g.gpt-live-1/gpt-live-1-mini— never entered theunknown-model computation and slipped past silently.The first fix broadened the family matcher to a general voice/audio vocabulary (
realtime | audio | live | transcribe | whisper | voice | tts). That correctly widened coverage to catch new families — but it compared full model ids against a known-id set. OpenAI ships dated snapshots of already-known families constantly, so a LIVE dry-run of the drift suite on this branch (run 28968203340) failed as critical drift on 7 legitimate snapshots:These are just dated snapshots of families already conceptually known (
tts-1,tts-1-hd,gpt-audio,gpt-4o-mini-transcribe,gpt-4o-mini-tts). Merged as-is, the daily scheduled drift job would go RED every day on false positives and bury a real new-family signal. Appending each new snapshot to a known-id set never converges.Fix — FAMILY NORMALIZATION (not whack-a-mole)
src/__tests__/drift/voice-models.ts:normalizeVoiceModelFamily(id)strips trailing version/snapshot suffixes from the END of the id, repeatedly:-YYYY-MM-DD(/-\d{4}-\d{2}-\d{2}$/)-NNN/-NNNN(/-\d{3,4}$/)A single-digit trailing
-N(as ingpt-live-1) is deliberately NOT stripped, sogpt-live-1normalizes togpt-live-1— still an unknown family.knownVoiceModelFamilies), built throughnormalizeVoiceModelFamilyso it stays idempotent/in-lockstep.detectVoiceModelDriftnow compares each candidate's normalized family against the known-family set;hasGAlikewise normalizes before matchinggaRealtimeModels(also reduced to family keys).NO_GA_REALTIME_MODELS=/UNKNOWN_REALTIME_MODELS=inws-realtime.drift.tsare preserved verbatim.Dated snapshots of a known family collapse onto it and stay green; a genuinely new family (
gpt-live) still normalizes to an unknown family and is flagged.Red-Green Proof
Drives the real
detectVoiceModelDrift/normalizeVoiceModelFamily— the same functions the live canary invokes — against a representativeGET /v1/modelslist containing the 7 real snapshots +gpt-live-*.RED (pre-normalization id-set matcher — CI failure reproduced locally)
All 7 real snapshots are falsely flagged (reproduces run 28968203340), and
gpt-live-1is flagged.GREEN (family normalization — snapshots pass, new family still caught)
The 7 real snapshots are NOT flagged (they normalize to known families);
gpt-live-1/gpt-live-1-miniARE still flagged as a new family;gpt-realtime-2.1/whisper-1/tts-1stay green.Unit suite (
src/__tests__/ws-realtime-canary.test.ts), including a regression guard on the 7 real ids and explicit normalization assertions:LIVE
Drift Testsre-verification (the real gate)Dispatched the live workflow on this branch after the fix:
driftjob conclusion: success (alsoagui-schema-drift: success,notify: success).OPENAI_API_KEYpresent in the drift step env, so the canary ran (not skipped); collector completed without crashing;drift-report.jsonentries: []— zero drift, the 7 previously-flagged snapshots no longer surface.No iteration was needed — the first live re-run after the fix was green.
Pre-push gates (all pass)
pnpm run format:check— All matched files use Prettier code stylepnpm run lint(eslint) — cleanpnpm run test— 146 files, 4334 tests passed (+2 new tests)pnpm run build(tsdown) — Build complete