Skip to content

feat(conformance): verify normal server resume and replay - #46

Merged
austinchennn merged 2 commits into
EmergentExit:mainfrom
wangyeyu1129:feat/conformance-server-resume-replay
Oct 17, 2026
Merged

austinchennn merged 2 commits into
EmergentExit:mainfrom
wangyeyu1129:feat/conformance-server-resume-replay

Conversation

@wangyeyu1129

@wangyeyu1129 wangyeyu1129 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds opt-in server-resume-replay-v1 black-box coverage for normal server recovery: unacknowledged suffix replay, resumeFrom 0, an empty replay suffix and replacement of a still-active connection. Real WebSockets and independent JSONL drivers verify both server output history and deduplication of already handled client context.

Issue association

Refs #17. This is a server recovery substep, not automatic client recovery, full bidirectional recovery or completion of the issue.

Merged latest upstream main 771d79f, including #42/#43 and now-merged #44 server input/#45 client control validation. All merged profiles, driver dispatch, CLI constraints and packaging checks are preserved. This independent PR still targets main; #47 and fork Draft #1 remain separate increments.

Behaviours Completed

  • Stable IDs: server/resume/suffix, /from-zero, /caught-up, /replace. Explicit server+driver-only profile; existing defaults/profiles and heartbeat-specific flags retain their behavior.
  • Each case processes client context seq 1 through public onContext, injects a public model transcript and deliberately withholds server ACKs to retain history. A physical disconnect is confirmed only after the driver delivers the public SDK connection.closed() callback, before offline model events.
  • Suffix/full-history cases generate two distinct transcripts while disconnected and require the complete ordered replay with original sequence/content. Captured wire bytes are distinct from application processing: from-zero captures the original transcript but deliberately leaves its processing cursor at 0.
  • Current-cursor recovery requires no old sequenced suffix. A new public model marker proves continued numbering and reveals missing last/intermediate messages; neither ready, first replay nor silence is considered completion.
  • Resumed ready preserves session identity/formats/baseline capabilities and reports the server's actual handled client cursor (1). Context seq 1 is resent, then seq 2 with new data; exactly two original callback values remain, with one model connection across both sockets.
  • Live replacement closes the old connection with 4000 on this direct loopback path; only the new peer receives subsequent model traffic. Both connection traces/close information survive into reports, including failures.
  • Independent schema/capability/sequence validation sees the combined session history; resumed replay need not start at seq 1. Semantic checks require the whole expected suffix. Identical same-seq replay is one logical message; conflicting replay and replayed transient ready fail.
  • Runner's new-peer ownership is explicit: it disposes the second peer in finally on every path; general runner cleanup owns first peer/driver. Missing instrumentation or required scenarios is incomplete/nonzero. fullSuite stays incomplete.

Files Changed

  • conformance/scenarios/server-resume.json, src/node/server-resume.ts: four cases, two independent cursors, context/close/model barriers and multi-peer failure evidence/cleanup.
  • conformance/src/resume-oracle.ts: SDK-independent effects, session/format/cursor checks and complete replay expectations.
  • conformance/src/node/drivers/server-resume.ts, drivers/typescript-server.ts: public Hub.attach/Connection.receive/closed, onContext, fakeModel events/sessions and bounded snapshots; no handcrafted expected wire frames in the official driver.
  • conformance/src/node/fixtures/resume-server.ts, test/server-resume.test.mjs: independent actual wire faults, same-seq positive control and bounded lifecycle verification.
  • conformance/src/node/{runner,cli}.ts, src/report.ts, build.mjs, test/consumer.mjs: profile/inventory selection, package assets/fixture, actual installed success and missing-required-case checks. Merged heartbeat dispatch/configuration is preserved.
  • README.md, conformance/{README.md,driver-protocol.md}: usage, minimal language-neutral JSONL contract, ownership and scope limits.

Brief Explanation

Runner controls scenarios/deadlines, transport records real traffic, driver observes only public SDK/model/edge APIs, and the oracle independently judges sequence/history and application effects. A public WebSocket edge is used around Hub because serve() does not expose per-connection close delivery; no private parked/session/socket/inbox/outbox state is inspected. Driver shutdown terminates physical peers, awaits their SDK close callbacks, then closes the Hub so parked expiry/heartbeat timers are cleared.

Prefix model events plus an ordered pong establish the capture window. After reconnect, actual context handling and a new public model marker fence replay, followed by live pong, second wire close, SDK close confirmation and final drain. Pongs alone are not treated as proof of arbitrary async completion. The fixture is inert: no scripts/tools/background producers, only synchronous public model injection and context observations. Deadline is bounded local test policy, not a normative latency requirement.

No production SDK, normative spec/schema/vector, dependency or lockfile changes. The new recovery profile does not use heartbeat clock controls or exercise ACK-deleted history, even though those independent contributions are now on its base.

Testing

Local environment: macOS 26.6.2 arm64, Node 22.23.3. Commands used Node 22 first on PATH.

  • Final integration npm run test:conformance: 121 passed (102 latest-main tests + 19 recovery tests). All four official server cases pass; build and conformance typecheck are part of this command.
  • Thirteen real wire/control faults fail at stable resume case IDs: missing middle/last replay, changed content, changed seq, wrong resumedFrom, replacement session/model, duplicate context application, missing/wrong old close, replayed original transient ready, conflicting same-seq content and missing observations. No runner failure report is fabricated by fixtures.
  • Independent valid server and identical same-seq replay pass. Missing/malformed instrumentation is incomplete/nonzero.
  • Lifecycle variants cover success, no ready/context/close confirmation, held replay/finish, startup exit, exit during model injection and early second-peer close. All four recorded direct-child PIDs per variant are absent after return. Failures with a second peer retain both traces; close/EOF/repeated cleanup is bounded.
  • npm run test:conformance:consumer: final actual tarball installed outside the repository, without SDK peer installation; all merged profiles plus recovery pass. Removing server/resume/suffix yields incomplete at inventory/exit 1. Schemas/scenarios resolve from installed assets. Re-run after final CLI parameter-isolation adjustment: passed.
  • Additional targeted CLI regression: recovery rejects heartbeat-only timeout configuration; typecheck and final standalone CLI rebuild passed. This preserves merged feat(conformance): verify server heartbeat and idle retention #42's flag boundary.
  • Rebased npm test: 465 passed, 26 SDK files, including fix(protocol): reject incomplete replay after ACK deletion #43's new regressions. npm run test:e2e: 7 passed.
  • npm run typecheck, npm run check:spec: passed (23 schemas, 1 registered capability, 156 control vectors, 9 audio vectors).
  • Production build (via conformance pipeline) and npm run check:budgets: passed, including zero-dependency/import rules; production sizes unchanged relative to rebased main (protocol 2.16 KB min+gzip / 5 KB; core 5.11/15; client 5.26/20).
  • git diff --check: passed. No separate lint/format script is configured. Python, standalone Workers and production SDK pack/consumer/release dry-run were not rerun locally because this diff is conformance/docs; standalone conformance packaging was exercised above. Hosted CI is separate evidence. No package published.

Unsure About

  • Review the minimal emitResume/waitResumeContext/waitResumeClosed/finishResume instrumentation before foreign-language implementations adopt it. Physical close acknowledgement is after the public SDK callback, not a private-state query. Model count includes all connected model sessions, not only active ones; context instrumentation never deduplicates on behalf of the SDK.
  • Recovery from zero distinguishes capture from application processing and intentionally withholds the receiver's recommended ACKs. This tests server retention, not a claim that the raw test client is a fully conforming SDK.
  • Replay semantics require the entire retained suffix; known original frames are compared canonically and offline markers against independent script expectations. The controlled inert model has no other sequenced producers. This is not an oracle for arbitrary production model outputs.
  • Close code 4000 is checked on a direct loopback connection alongside identity, cursor, model count, context effects and complete traffic; it is not the sole evidence of recovery (§9.3).
  • Snapshot/drain promises are trusted finite instrumentation, not dishonest-driver detection, process-tree supervision or arbitrary delayed-work guarantees. A model/context marker and ordered interactions complement, rather than replace, the complete final trace.
  • feat(conformance): verify server input and audio frame boundaries #44/feat(conformance): verify client invalid control handling #45 are merged and included. Their input/binary/client-control checks, assets, public driver routing and close behavior are preserved alongside recovery. Other pending features are not imported.
  • ACK-deleted/evicted history, capacity/expiry failure, principal isolation, lost tool results and side-effect idempotency, full epoch/tool/capability preservation, automatic client/bidirectional recovery, heartbeat/binary/audio matrices and Python driver remain out of scope. No production defect requiring a separate fix was discovered.

Implementation Order and Acceptance — #17

The current issue checklist was re-read before delivery. Upstream access is read-only, so its author's issue body was not modified. Checked subitems below mean locally implemented and verified, PR pending merge, not upstream delivery. Maintainer-copyable progress retaining the mixed parent unchecked:

Latest status: #42/#43/#44/#45 are merged and included in the tested base; this PR and #47 remain pending. Fork Draft #1 remains stacked on this branch. No supervisor/contributor issue text or checkmark was rewritten and no mixed parent was marked complete.

Integration refresh: merged main without rewriting review history; nine conflicting common files were resolved additively. Full build/conformance/SDK/e2e/type/spec/budget/tarball checks were rerun in /private/tmp/vrt-open-46-_7uzs_xj against byte-identical tracked sources, using same-version temporary esbuild binaries (0.25.10 build/budget; 0.28.2 Vite) and workspace links to the matching mirrored packages, to avoid the previously observed Desktop native file-open stall. No environment workaround or dependency change is committed. Direct Desktop native-build completion is not claimed. This is integration maintenance, not new #17 feature acceptance; upstream issue text/checkmarks remain unchanged.

@austinchennn
austinchennn merged commit c55e752 into EmergentExit:main Oct 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants