Skip to content

feat(conformance): verify client invalid control handling - #45

Merged
austinchennn merged 3 commits into
EmergentExit:mainfrom
wangyeyu1129:feat/conformance-client-control-validation
Oct 17, 2026
Merged

austinchennn merged 3 commits into
EmergentExit:mainfrom
wangyeyu1129:feat/conformance-client-control-validation

Conversation

@wangyeyu1129

@wangyeyu1129 wangyeyu1129 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds opt-in client-control-validation-v1: five black-box cases verify that a ready client reports malformed control input, does not execute it or pollute sequence/epoch/call state, and still executes a subsequent legal tool call exactly once. The suite uses real WebSockets and independent JSONL driver processes.

Issue association

Refs #17. This is a partial client invalid-control substep, not completion of the issue or full compatibility certification. Updated onto upstream main 26aa7f3, including merged #42 heartbeat, #43 ACK/replay repair and #44 server input/audio boundaries. This independent PR still targets main. An additive merge preserves both server input validation and client control validation; #46/#47 and fork Draft #1 are not included.

Behaviours Completed

  • Stable IDs: client/control/framing, /version, /types, /fields, /state-call. Explicit client/driver-only profile; old ready/finish forms and profiles preserve their behavior.
  • After baseline hello/ready, driver ready confirms public subscriptions are installed. Only then does the runner inject invalid text. Shared server-envelope vectors are unchanged; local additions cover missing/string versions, combined priority faults, exact toString/proto and direction matching, invalid numeric/typed seq, high-epoch invalid state and malformed calls (§§4.1–4.3, 9.1).
  • Every injected invalid input is reported through the public application error interface with the correct classification/recoverability. Observation snapshots require no business events and zero tool starts/commits. Missing fields never count as zero.
  • Legal turn.state seq 1 and tool.call seq 2 follow the invalid seq-bearing inputs. The valid call uses the same callId as malformed calls, has a schema-permitted extra property, and is sent twice with identical seq/content. It starts/commits once and produces one successful logical result.
  • A sequenced public transcript marker fences prior network input. One public interrupt() after the completed tool probes the actual observed epoch on wire: it must be 1, detecting hidden epoch pollution even if the legal tool could still run.
  • Finish drains the inert synchronous probe's continuations, closes the SDK and awaits its public closed event. Runner checks the entire client-output trace through WebSocket close, final application events/effects, identities, ACK bound and result uniqueness. Same-seq result replay passes; conflicting same-seq and different-seq duplicate results fail.
  • Instrumentation is distinct from wire capabilities. Missing/malformed declarations yield incomplete/nonzero. Missing required installed scenarios also yield incomplete/nonzero. Reports retain protocol/profile/capabilities, stable ID, expected/actual, failure stage and fullSuite=incomplete.

Files Changed

  • conformance/scenarios/client-control.json, src/node/client-control.ts: five cases, unmodified server-vector loading, raw invalid sends, legal state/call/retransmission, transcript fence and epoch probe.
  • conformance/src/client-control-oracle.ts: independent normalized application observations and logical tool-result checks; no SDK decoder/Inbox/Outbox/gate dependency.
  • conformance/src/node/drivers/client-control.ts, drivers/typescript-client.ts: opt-in public connect/events/tool instrumentation, bounded snapshots and explicit closed-event fence. Existing cancellation/default implementation stays on its original path.
  • conformance/src/node/fixtures/control-client.ts, test/client-control.test.mjs: independent actual wire/control faults, allowed replay and lifecycle checks.
  • conformance/src/node/{runner,cli}.ts, src/report.ts, build.mjs, test/consumer.mjs: profile/inventory selection, package fixture/scenarios, installed success and missing-case verification.
  • README.md, conformance/{README.md,driver-protocol.md}: usage, additive language-neutral controls, normative distinctions and scope limits.

Brief Explanation

Runner orchestrates deadlines and barriers; transport remains unchanged; the official driver observes only public SDK APIs; independent schema/state/effect checks judge target output and application observations. Deliberately invalid server injections are not incorrectly passed through an oracle demanding valid input. Expected classifications come from spec/shared vectors, never the tested decoder.

The transcript receipt fence follows the real legal tool result and repeated call, so final effects cover prior input. The post-completion interrupt is only an observed-epoch probe, not a new cancellation scenario. The driver has no asynchronous probe, model or external side effect; finite event-loop drain plus public closed notification and wire close fence establish this controlled scenario's end. No timed sleeps or private state inspection are introduced.

Testing

Local environment: macOS 26.6.2 arm64; Node 22.23.3. Commands used Node 22 first on PATH.

  • npm run test:conformance: 102 passed (86 latest-main tests + 16 client-control tests), including production/conformance builds and conformance typecheck. Official client passes all five cases.
  • Ten fault tests fail at stable client IDs: missing observations, wrong priority, unreported errors, illegal tool execution, invalid receive-cursor advance, invalid callId reservation, hidden epoch pollution, premature close, different-seq duplicate result and conflicting same-seq result. These faults occur through real WebSocket/control paths, never fabricated runner reports.
  • Independent valid client and identical same-seq result replay pass. Every reference case also retransmits the same valid call; its effects stay one. Missing/malformed instrumentation is incomplete.
  • Lifecycle variants cover normal completion, no ready, no error notification, held legal result, held finish, startup exit, exit after invalid wire input and premature close. Each variant's five recorded direct-child PIDs are absent after runner return. Close/EOF are idempotent; pending waits reject on observed close or runner terminates the unresponsive direct process boundedly.
  • npm run test:conformance:consumer: actual tarball installed outside the repository with no SDK peer install; all merged profiles and this profile run. Removing client/control/state-call yields incomplete at inventory and exit 1. Schemas and server vectors resolve from installed assets.
  • npm test: 465 passed, 26 files, rerun for this integration refresh; fix(protocol): reject incomplete replay after ACK deletion #43 regressions are included. feat(conformance): verify server input and audio frame boundaries #44's conformance profile is included in the separate full conformance run.
  • npm run typecheck, npm run check:spec: passed (23 schemas, 1 registered capability, 156 control vectors, 9 audio vectors).
  • npm run build (via conformance pipeline), npm run check:budgets: passed, including dependency/import boundaries; production sizes unchanged (protocol 2.16 KB min+gzip / 5 KB; core 5.11/15; client 5.26/20).
  • npm run test:e2e: 7 passed, rerun for this integration refresh. git diff --check: passed. No standalone lint/format script is configured.
  • Python, standalone Workers suite, production SDK tarball/consumer and release dry-run were not rerun locally because no production code changed; standalone conformance tarball was tested above. Hosted CI is separate evidence. No npm publication.

Validation environment for this refresh: because native esbuild file opens previously stalled in the Desktop checkout, the full build/conformance/SDK/e2e/type/spec/budget/installed-tarball checks ran in /private/tmp/vrt45-merge-verify-w0e4swah. Every tracked source file was byte-compared identical to the target checkout; workspace dependencies resolve to the mirrored packages, external dependency versions are unchanged. Temporary esbuild copies match their hosts (0.25.10 repository build/budget, 0.28.2 Vite). No workaround, dependency change or local path was committed. Direct Desktop native-build completion is not claimed.

Unsure About

  • Review the normalized diagnostics/events and additive command names before foreign-language drivers adopt this contract. Public connection-state events should not be confused with normalized vRT turn.state observations. Diagnostics preserve classification/recoverability, not exact text.
  • §7.1 does not require clients to detect valid callId reuse with a different seq. This profile does not require that: malformed frames cannot reserve callId, so the later call is its first valid use. Same-seq call retransmission must not re-execute (§4.3); conflicting-result checking belongs to the independent output oracle.
  • Business arguments are validated as required wire fields; optional application JSON Schema validation is not promoted to MUST. The legal probe intentionally uses object arguments; no business-schema implementation is added.
  • The epoch probe calls public interrupt() after completed work and a receipt marker, checking wire epoch without reading the gate. It adds one expected local interrupted notification, not rollback/forced-cancellation guarantees.
  • Bounded observations/drain are trusted instrumentation, not protection against dishonest drivers, arbitrary delayed work, process trees or external effects. No scripts or background producers exist in this fixture.
  • feat(conformance): verify server heartbeat and idle retention #42 and feat(conformance): verify server input and audio frame boundaries #44 are now merged. Their heartbeat and server-input registration, driver dispatch, CLI restrictions, assets, docs and installed consumer checks are preserved alongside this PR. Other pending profiles remain absent.
  • Pre-ready application subscription, binary/audio, automatic heartbeat/reconnect, full recovery, remaining fault matrices and Python driver remain unverified. fullSuite stays incomplete. No production defect requiring a separate fix was discovered.

Implementation Order and Acceptance — #17

The current issue checklist was re-read before delivery. Upstream repository access is read-only, so its author's issue body was not modified. Checked subitems below mean locally implemented and verified, pending merge, not upstream delivery. Maintainer-copyable progress:

#42/#43/#44 are merged on upstream main. This PR remains pending review/merge. No existing supervisor/contributor issue text or checkmark was rewritten, and no mixed parent was marked complete.

Integration refresh after #44: latest main merged without rewriting existing commits; all eight conflicting integration/docs files preserve both profiles. This is maintenance of the existing client-control substep, not an additional conformance feature or a completed recovery/security parent. Upstream issue checkmarks remain unchanged.

@austinchennn
austinchennn merged commit 771d79f into EmergentExit:main Oct 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants