Skip to content

feat(conformance): verify server input and audio frame boundaries - #44

Merged
austinchennn merged 2 commits into
EmergentExit:mainfrom
wangyeyu1129:feat/conformance-server-input-validation
Oct 17, 2026
Merged

austinchennn merged 2 commits into
EmergentExit:mainfrom
wangyeyu1129:feat/conformance-server-input-validation

Conversation

@wangyeyu1129

@wangyeyu1129 wangyeyu1129 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds opt-in server-input-validation-v1 black-box coverage for invalid control messages and binary input audio boundaries. Seven cases exercise sessionless/bound error classification, exact type/direction matching, malformed binary frames, ignored pre-ready audio, and valid/empty PCM input over real WebSockets and independent JSONL driver processes.

Issue association

Refs #17. This completes only a server input-validation substep, not the full issue or protocol certification. Updated onto upstream main 4ceddbf, including merged #42 heartbeat and #43 replay-after-ACK. Shared profile registration, dispatch, packaging and docs preserve both heartbeat and input-validation behavior; this independent PR still targets main.

Behaviours Completed

  • Stable IDs: server/input/control-pre, /control-post, /types, /binary-pre, /binary-post, /audio-before-ready, /audio-ready. Profile requires server mode plus explicit driver; default and existing profiles preserve their behavior.
  • Client envelope vectors run before and after session binding, with additional invalid text fields/seq-0 text/numbered ping. Expected priority is MALFORMED → UNSUPPORTED_VERSION → UNKNOWN_TYPE → INVALID_FIELD (§9.1). Errors remain recoverable; sessionless errors use 0 and bound errors enter the contiguous sequenced stream (§4.1).
  • toString, __proto__, absent/non-string/unknown types, and server-only pong are rejected as UNKNOWN_TYPE. Invalid messages produce no observed model text/audio or receive-cursor advance. Valid seq-1 text still executes, including a schema-permitted extra property.
  • Binary lengths 0–4, kinds 0/3/255 and server-received kind 2 yield recoverable MALFORMED without model audio. The kind-2/max-seq golden vector remains structurally valid but is explicitly wrong for this receiver direction (§8.1).
  • Pre-hello input audio is ignored without creating a model session and never buffered into the later session. A distinct post-ready marker arrives once. Valid PCM payload/model-facing negotiated format are checked; empty payload may be ignored or forwarded empty once. No sample-alignment SHOULD is promoted to a MUST.
  • Audio cannot advance the control receive ACK cursor or silently consume a control sequence. finishInputs emits one public model transcript marker: real outgoing sequence gaps are observable even when the inert model otherwise produces no traffic. An unsolicited ACK upTo 0 is permitted, since it does not identify/acknowledge an audio frame.
  • Final decisions use ordered receipt fences, bounded model observations, controlled-work drain, a live completion fence and the complete transcript through close. Same-seq identical retransmission passes; conflicting retransmission fails. Required inventory missing from the installed package yields incomplete/exit 1. fullSuite remains incomplete.

Files Changed

  • conformance/scenarios/server-input.json, src/node/server-input.ts: seven required cases, unmodified shared vector loading, raw binary/text sends, explicit session boundaries and receipt/completion/close fences.
  • conformance/src/input-oracle.ts: SDK-independent public-input snapshot validation and expected model effects; missing observations never count as zero.
  • conformance/src/node/drivers/server-input.ts, drivers/typescript-server.ts: opt-in public serve/fakeModel({onCommand}) instrumentation, bounded snapshots, public model marker, idempotent close. No handcrafted wire frames in the official driver.
  • conformance/src/node/ws-peer.ts: explicit binary send only; unexpected binary responses still fail text-only profiles.
  • conformance/src/node/{runner,cli}.ts, src/report.ts: explicit profile selection, instrumentation checks, required cases, failure stage and incomplete behavior.
  • conformance/src/node/fixtures/input-server.ts, test/server-input.test.mjs: independent real-wire faults, allowed alternatives/replay and lifecycle verification.
  • conformance/build.mjs, test/consumer.mjs: include new assets/fixture, installed tarball success and missing-required-case check.
  • README.md, conformance/{README.md,driver-protocol.md}: usage, additive language-neutral JSONL contract and exact scope/trust limits.

Brief Explanation

Runner owns scenario orchestration and deadlines, transport owns actual WebSocket traffic, driver observes only public model APIs, and independent schema/state/effect checks judge behavior. Raw malformed frames never depend on SDK codecs. Existing capability gating and sequence/replay checks stay intact. No production source, normative schema/vector, dependency or lockfile changes.

A pong confirms ordered receipt of preceding frames, not arbitrary asynchronous completion. The instrumented model has no scripts/tools/background producers; finish drains attached continuations and emits a public input.transcript event. The runner observes its actual wire mapping, then fences/closes and validates the full trace. This finite trusted instrumentation detects observable input effects without pretending a quiet interval proves absence forever.

Testing

Local environment: macOS 26.6.2 arm64, Node 22.23.3. Commands used Node 22 first on PATH.

  • npm run test:conformance: 86 passed (65 existing + 21 new). Includes production build, conformance build/typecheck and all profiles merged on main.
  • Official server passes all seven cases. Thirteen faults are caught at stable input case IDs: priority, invalid-control effects, invalid-binary effects, recoverable-error close, pre-ready buffering, invalid receive-cursor advance, bound/sessionless seq errors, audio advancing ACK/counter, conflicting replay, unexpected binary response and missing observations.
  • Independent correct wire peers pass, including ignoring empty payload, unsolicited ACK 0 and identical same-seq marker retransmission. This guards against over-restrictive oracles.
  • No-response ready/finish, abnormal exit, early close and successful repeated cleanup are bounded; all seven directly spawned PIDs per lifecycle variant are absent after the runner returns. Close/EOF are idempotent.
  • npm run test:conformance:consumer: actual tarball installed outside repository without SDK peers; all merged profiles plus this new profile pass. Removing server/input/binary-post yields incomplete at inventory and exit 1. Schemas/control/binary vectors resolve from installed assets.
  • npm test: 465 passed across 26 SDK files, including fix(protocol): reject incomplete replay after ACK deletion #43's six reliability regressions.
  • npm run typecheck, npm run check:spec: passed (23 schemas, 1 capability, 156 control vectors, 9 audio vectors).
  • npm run build (via conformance pipeline), npm run check:budgets: passed. Production SDK size/dependency boundaries unchanged (protocol 2.16 KB min+gzip / 5 KB; core 5.11/15; client 5.26/20).
  • npm run test:e2e: 7 passed. git diff --check: passed.
  • No separate lint/format script is configured. Python, standalone Workers suite and production SDK packaging/release dry-run were not rerun locally because this diff is conformance/docs only; standalone conformance packing was exercised above. No package published. Hosted checks are separate from local evidence.

Unsure About

  • Review the additive readInputs/finishInputs instrumentation, PCM16/24000 fixture requirement and final model marker before foreign-language drivers adopt them. Missing instrumentation or unsupported fixture format is incomplete, not global wire nonconformance.
  • ACK upTo 0 can be harmless and cannot be causally attributed to audio from the wire alone. This profile permits it, checks ACK bounds and detects hidden outgoing sequence consumption through the model marker. It does not invent a general ban on idle ACKs or legitimate server state messages.
  • Big-endian/uint32-max golden layout is independently checked without SDK codecs, but model sendAudio does not expose the informational audio seq. This does not prove the receiver's seq decoding, frame-loss diagnostics or a full wrap. No new audio dedup/gap rule is imposed.
  • Public model snapshots/drain are trusted bounded instrumentation, not dishonest-driver detection, arbitrary-work supervision, process-tree isolation or rollback. Input byte/chunk counts/formats describe this controlled direct-PCM fixture, not device/resampling certification.
  • Early closure prevents this required input scenario from finishing and is a bounded failure here; no universal claim about every unrelated peer disconnect is made. Error text is not required to match, only normative code/recoverability/state behavior.
  • feat(conformance): verify server heartbeat and idle retention #42 is now merged. Its heartbeat dispatch, metadata checks, CLI timing restrictions, assets and consumer checks are preserved alongside this profile. feat(conformance): verify client invalid control handling #45 remains a separate contribution.
  • Full bidirectional resume/replay/lost results, client input-error handling, opening-state audio races, output audio/playback/devices, remaining binary matrices and Python driver remain outside this profile. No production defect was discovered requiring a separate fix.

Implementation Order and Acceptance — #17

The issue was re-read before delivery. Upstream access is read-only, so the author's issue body was not modified. Checked subitems below mean locally implemented and verified, pending merge, not already delivered on main. Maintainer-copyable progress, retaining the mixed parent unchecked:

#42 heartbeat and #43 replay repair are merged on upstream main; this PR remains pending review/merge. Existing supervisor/contributor issue wording and checkmarks were preserved; no broad parent or issue was marked complete.

@austinchennn
austinchennn merged commit 26aa7f3 into EmergentExit:main Oct 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants