Repository navigation
feat(conformance): verify server input and audio frame boundaries - #44
Merged
austinchennn merged 2 commits intoOct 17, 2026
Conversation
This was referenced Oct 17, 2026
This was referenced Oct 17, 2026
2 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds opt-in
server-input-validation-v1black-box coverage for invalid control messages and binary input audio boundaries. Seven cases exercise sessionless/bound error classification, exact type/direction matching, malformed binary frames, ignored pre-ready audio, and valid/empty PCM input over real WebSockets and independent JSONL driver processes.Issue association
Refs #17. This completes only a server input-validation substep, not the full issue or protocol certification. Updated onto upstream main
4ceddbf, including merged #42 heartbeat and #43 replay-after-ACK. Shared profile registration, dispatch, packaging and docs preserve both heartbeat and input-validation behavior; this independent PR still targets main.Behaviours Completed
server/input/control-pre,/control-post,/types,/binary-pre,/binary-post,/audio-before-ready,/audio-ready. Profile requires server mode plus explicit driver; default and existing profiles preserve their behavior.toString,__proto__, absent/non-string/unknown types, and server-onlypongare rejected as UNKNOWN_TYPE. Invalid messages produce no observed model text/audio or receive-cursor advance. Valid seq-1 text still executes, including a schema-permitted extra property.finishInputsemits one public model transcript marker: real outgoing sequence gaps are observable even when the inert model otherwise produces no traffic. An unsolicited ACK upTo 0 is permitted, since it does not identify/acknowledge an audio frame.Files Changed
conformance/scenarios/server-input.json,src/node/server-input.ts: seven required cases, unmodified shared vector loading, raw binary/text sends, explicit session boundaries and receipt/completion/close fences.conformance/src/input-oracle.ts: SDK-independent public-input snapshot validation and expected model effects; missing observations never count as zero.conformance/src/node/drivers/server-input.ts,drivers/typescript-server.ts: opt-in publicserve/fakeModel({onCommand})instrumentation, bounded snapshots, public model marker, idempotent close. No handcrafted wire frames in the official driver.conformance/src/node/ws-peer.ts: explicit binary send only; unexpected binary responses still fail text-only profiles.conformance/src/node/{runner,cli}.ts,src/report.ts: explicit profile selection, instrumentation checks, required cases, failure stage and incomplete behavior.conformance/src/node/fixtures/input-server.ts,test/server-input.test.mjs: independent real-wire faults, allowed alternatives/replay and lifecycle verification.conformance/build.mjs,test/consumer.mjs: include new assets/fixture, installed tarball success and missing-required-case check.README.md,conformance/{README.md,driver-protocol.md}: usage, additive language-neutral JSONL contract and exact scope/trust limits.Brief Explanation
Runner owns scenario orchestration and deadlines, transport owns actual WebSocket traffic, driver observes only public model APIs, and independent schema/state/effect checks judge behavior. Raw malformed frames never depend on SDK codecs. Existing capability gating and sequence/replay checks stay intact. No production source, normative schema/vector, dependency or lockfile changes.
A pong confirms ordered receipt of preceding frames, not arbitrary asynchronous completion. The instrumented model has no scripts/tools/background producers; finish drains attached continuations and emits a public input.transcript event. The runner observes its actual wire mapping, then fences/closes and validates the full trace. This finite trusted instrumentation detects observable input effects without pretending a quiet interval proves absence forever.
Testing
Local environment: macOS 26.6.2 arm64, Node 22.23.3. Commands used Node 22 first on PATH.
npm run test:conformance: 86 passed (65 existing + 21 new). Includes production build, conformance build/typecheck and all profiles merged on main.npm run test:conformance:consumer: actual tarball installed outside repository without SDK peers; all merged profiles plus this new profile pass. Removingserver/input/binary-postyields incomplete at inventory and exit 1. Schemas/control/binary vectors resolve from installed assets.npm test: 465 passed across 26 SDK files, including fix(protocol): reject incomplete replay after ACK deletion #43's six reliability regressions.npm run typecheck,npm run check:spec: passed (23 schemas, 1 capability, 156 control vectors, 9 audio vectors).npm run build(via conformance pipeline),npm run check:budgets: passed. Production SDK size/dependency boundaries unchanged (protocol 2.16 KB min+gzip / 5 KB; core 5.11/15; client 5.26/20).npm run test:e2e: 7 passed.git diff --check: passed.Unsure About
Implementation Order and Acceptance — #17
The issue was re-read before delivery. Upstream access is read-only, so the author's issue body was not modified. Checked subitems below mean locally implemented and verified, pending merge, not already delivered on main. Maintainer-copyable progress, retaining the mixed parent unchecked:
#42 heartbeat and #43 replay repair are merged on upstream main; this PR remains pending review/merge. Existing supervisor/contributor issue wording and checkmarks were preserved; no broad parent or issue was marked complete.