Repository navigation
Baton2: recover lost retained custody and checkpoint stream observation #696
Description
A retained provider receive blocked recovery after its keeper disappeared. Two receive attempts each used more than 11 CPU minutes while replaying the same 39 MB stdout artifact. Direct continuation was refused because the recorded retained execution remained active.
Observed evidence from provider683-deepseek-impl-20261006:
- The recorded native PID was absent, and the attempt had no open files. The canonical session lock was acquired before manual recovery.
- The artifact contained 149,025 progress frames and no terminal event. Its SHA-256 was
ad0372a35b80ac55a9264557762a1d230b19f2359ac359b51fbbbadd77da772a. - Five pending input messages remained unacknowledged.
- A complete database backup and before/after receipt were retained. The exact old execution was marked
unknown after keeper loss, and the same recorded native conversation resumed. Original output, manifests, message bodies, and receipts were preserved. No completion was fabricated.
Implement native recovery using actual custody and process identity, with a durable incremental observation checkpoint. Restart must reconcile unread output and pending input without parsing all previously observed progress frames. Native output and terminal report evidence must survive client and coordinator loss. Default trace capture needs bounded generation and artifact retention under #686, including repeated streaming snapshots. A stale heartbeat alone cannot authorize replacing a live owner.
Required tests cover keeper loss without a terminal frame, a surviving provider after receiver loss, large retained progress streams, repeated recovery, same-conversation continuation, pending input during recovery, report conflict, and checkpoint corruption or partial writes. Measure replay work and generated bytes before and after. Operative laws and mutations must cover the actual recovery admission and checkpoint behavior.
Coordinate #685 pending-input recovery, #676 shared native custody, and #686 logging. The native-receive conductor owns receive admission; the shared-instance implementer owns host custody. Root will coordinate the observer checkpoint source scope. Complete reviewed source, remote exact-source validation, publication, integration, and installed-runtime evidence before closing.
Tracking: Baton todo.
Current work tracking: Open-task TODO.
Activity
flip-engineer commentedon Oct 11, 2026
Current operator-raised follow-up work is linked to public in-harness tasks in the existing audit-root Orchestra. Owners retain design choices and can return findings, alternatives or revised proposals. Concrete regression checks apply where behavior is established.
Complete interrupted original-input recovery through original owners
Original inputs can remain unfinished after provider terminal events, retained output processing or receiver/keeper loss. The historical Dirty caller failed schema migration before consuming retained frames. Original native and receipt state must remain interpretable through recovery.
#696, #705, #706, #708 and #685 retain the original cases. The Dirty diagnosis preserves original stdout, manifest and native identity; compatibility source has been published, while qualified installation and the original public recovery remain separate work.
Purpose: Let original actors continue and handle their own retained input with their native conversations, source, history and explicit stops preserved.
Use the existing receive/adoption/recover-observer lifecycle where sufficient. Report which original input is admitted, handled or still owed and its actual obstruction. Distinguish provider failure, native exit, observation, input receipt and assignment confirmation. Investigate unresolved custody causes without assigning an invented writer or completion.
In-harness task
root-gap-original-input-recovery-20261011: senderaudit-root, recipientnative-receive-conductor. Existing Root todo IDs:recover-failed-seats,all-harness-continuation,provider694-observer,project-session-resume,coordinator-completion.The canonical Orchestra database is
.scratch/baton2-audit-followups-20261004/orchestra.dbin the root-delivery checkout. Read a linked task withbaton2 DATABASE delivery TASK_ID. Each task links back to this issue. Root retains current integration, qualification and delivery authority.Recover released custody through the selected observer
flip-engineer commentedon Oct 15, 2026
The release branch now contains
b7497f24(parentdcbd7c7d), a scoped selected-observer recovery accessor. The new path reads the current released custody while preserving the existing ACK, launch and native-start-error exclusions. Ordinary receive admission and direct-turn recovery keep their existing unreleased selection.The regression fixture retains the complete multiline Unicode result, original report identity, original native conversation, queued continuation, immutable manifest/native birth/status and final physical acknowledgement. It also checks that a later public recovery request refuses the acknowledged attempt and starts no native process. Static review caught and corrected an inherited-stderr wait cycle in the fixture before qualification.
This change is committed and source-reviewed. Remote compilation and native fixture execution remain pending. Current active adoption still uses the original owner and has not been reported as complete. Complete historical output must be retained before physical acknowledgement, and the actual old owner and recovery children must finish before adoption is accepted.
Cross-reference: release and active adoption, completed-attempt reconciliation, retained custody recovery, and the tracked work.
flip-engineer commentedon Oct 15, 2026
Postrelease 2.0.0 observation on October 15: ordinary
serveelected owner epoch 2, emitted{"generation":2,"cursor":0,"gap":true}, then exited 2 while resolving the historicalnative-instance-muse-implementerattemptreceive:native-instance-muse-implementer:26507:c120995c0a8e0491321a6cce693c402d. The installed owner remained active with zero attempts and zero observers.Read-only inspection found the attempt directory, a structurally valid BATONRP2 manifest and
launch. The database recordsphase=exitedandstatus=start failed. The attempt has nonative.birth,native.pid,native-start-error,releasedoracknowledged. Receive.acquired_recorded callsrecovery_argvfor the retained execution. br_recovery validates the manifest and then readsnative.birth; that missing file explains the reportedkeeper ...: No such file or directoryboundary. The historical cause of the incomplete start remains unproven.Track this failed-start recovery refusal here, with lifecycle reporting in #712. The original execution, files and command streams are preserved in
/tmp/baton-v200-active-serve-owner-20261015.{json,stdout,stderr}and/tmp/baton-v200-postrelease-serve-attempt-files-20261015.json. This remains a postrelease recovery followup; active-owner adoption completed.flip-engineer commentedon Oct 17, 2026
Reboot recovery with missing native terminal status
On 2026-10-17, five OMP sessions in the native fframes port campaign retained executions in
runningstate after a host reboot. Their native process births were gone, and their retained attempt directories had no terminalstatusfile. Each session also had an outstanding wake claim. Ordinary resume could not adopt these attempts: pinned sourcee6a56295f0f407ee6993d9d99ba158696d28f7bereturnsENODATAfrombr_owner_recoverwhen the recorded native birth has ended without a terminal status (source). The receive path still sees recovery arguments while the attempt directory remains available (br_recovery).The operator recovered the five sessions under their native per-session locks. The recovery archived the original execution pointers and wake rows, cleared each current execution's directory pointer, and recorded
phase=exited,status=unknown-after-reboot. Original attempt directories and their evidence remained untouched. The sessions resumed with their existing native conversation IDs and original inbox history. No terminal status, release marker, acknowledgement marker, provider success, or completion was fabricated.Recovered sessions and current attempts:
simulation: oldreceive:simulation:4:5cfa9d7db8ec6e3b290d5e2449c725b2; resumed asreceive:simulation:583:7e72e93dcf2e1a8988a51f68df7c5c94; native ID01a14713-d3b8-7000-b15a-112318059965.material-shader: oldreceive:material-shader:6:563e65e11ecab767a5ce5967eece04a9; resumed asreceive:material-shader:619:451215774a6b5c913e749f8369366836; native ID01a14713-dfd0-7000-9bf5-be76ef60a71f.native-gpu: oldreceive:native-gpu:8:95717bcbdef7bfd34ee8370463016013; an intermediate recovery attempt exited 143 and was retained; current attemptreceive:native-gpu:624:3533e1ac4493a5f0b0d613ad3b91539a; native ID01a14713-fb44-7000-b3ae-9acc9f4987b3.quality: oldreceive:quality:10:a0f69acaf5e583d2ebb48df6d7ff4322; resumed asreceive:quality:621:c5599846b3d8a88b9fdeeee71171b272; native ID01a14714-293d-7000-b23e-1d3738750b6e.build-runtime: oldreceive:build-runtime:12:8b6c0b0dd1ac94c77585c21bc019bca8; resumed asreceive:build-runtime:620:9a4595c458df54d15b926390d0c1d85d; native ID01a14714-4cb0-7000-8ea0-426dab0f6c96.The recovery evidence bundle is at
/home/atari2036/render-fframes-port-20261016/evidence/reboot-recovery-94650f62-e8b3-44ee-bfb9-7ab23994d642/unknown-native-recovery-20261017T010850Zon the host. This operator recovery addressed the live sessions; it does not provide a normal supported recovery operation for future reboot cases.Add a supported recovery path for an old-boot native attempt with no terminal status. It must prove the old native lifetime ended, retain the original attempt and inbox evidence, record the outcome as unknown, resume the same native conversation, and preserve recipient-owned message receipts. Tests should cover a stale wake claim, missing status after reboot, repeated resume, and a recovery attempt that exits before its successor starts.
flip-engineer commentedon Oct 17, 2026
October 17 recovery follow-up
The portfolio Orchestra on boot
644c8e32-c8b4-45a9-973e-518954b52678has seven sessions with retained executions marked running while their native PIDs are absent:accounts-deepseek,runtime-deepseek,logging-deepseek,logging-glm,runtime-wonder,release-wonder, andknowledge-wonder. Each raw stdout tail contains NUL bytes after the last validated checkpoint. A supported receive attempt on pinned Baton 2.0.0 exited with NUL22 before starting a provider. These seven inputs remain pending; their original logs, checkpoints, native UUIDs, and receipt state are preserved. They have not been retried against the corrupt tails.A source-reviewed follow-up is on
luna/769-invalid-tail-recovery, based on0880cf3dddee47c68dd98c6411a3570fc53749b7. Current hashes areturn.bend0217bf65870c5a156fe7ad8840d4cfbbfef7c0f56bd1f48ccf8117ce853f13d0andreceive.py6fdae696d797912d7a55b1a4363aaa752eca643ac7eebc5c46f7107ca0dd03a5. The change preserves the raw tail and checkpoint, records an unknown terminal outcome only after custody proves the old native and keeper are gone, and keeps owed message receipts pending. The crash-shaped fixture has not yet been built or run.The separate receive-memory source commit
0880cf3dddee47c68dd98c6411a3570fc53749b7passed focused compiler checking. Its emergency-O0Main11 build is still in C generation underbaton-main11-0880cf3-memory-o0.service(Bun PID 133396); no C file or executable artifact exists yet. The OMP 17.4 producer delta-filter correction is deployed at the saved endpoint (SHA-2560d22b0eaaf713cc16874181737368967dfe871d3ec7f6711328b7cb632be9f35). Build and producer details remain tracked under #769, with receive retry behavior linked to #786.A separate clean-tail recovery succeeded for fframes: attempt
receive:fframes-lead:694:5406ae95ed37c37072b03b332da2d0c5is running with PID 189235 and the original native UUID01a1475d-d987-7bf2-8968-b31165d91a60. The original 3.55 MB stream had no NUL tail, and the new attempt recorded 14 owned reads. The official fframes video skill, API reference, and upstreamAGENTS.mdwere read.Record retained-tail recovery and build status for #696
flip-engineer commentedon Oct 17, 2026
2026-10-17 recovery receipt
The clean-tail
knowledge-deepseeklaunch gap was resolved separately from the seven NUL-tail sessions. Its prior attempt had exited 127 before provider launch, with no native UUID and an empty stream. Using Baton’s supportedreceiveraction through the liveaccounts-wonderactor registered the existing OMP 17.4 endpoint (SHA-2560d22b0eaaf713cc16874181737368967dfe871d3ec7f6711328b7cb632be9f35) while preserving the recorded workspace and pending input. The original dispatchdispatch:bb56091c-a309-43eb-a490-fb89f4dc0909now has an owned read under native UUID01a147c2-99b1-7000-bcb7-cbaea84d2e53; attemptreceive:knowledge-deepseek:42:8e458cd46f641dd816c5b02d89530874is running with Baton driver PID 223556. The input has not been acknowledged.The fframes lead continues its original attempt
receive:fframes-lead:694:5406ae95ed37c37072b03b332da2d0c5with native UUID01a1475d-d987-7bf2-8968-b31165d91a60. Its current route remains inheritedmuse-wonder-full-1.0; no model change was made during the active attempt. Operator guidancemessage:operator:48b2eac01a4535020a9e25439a45909fasks the lead to finish naturally and wait for a standardmuse-wonder-1.0route update before any future turn.The failed Kimi report remains preserved and unacknowledged. Its quota-failure scope was forwarded through Baton to the existing DeepSeek worker as guidance
message:rewrite-tooling:a8c03fce4bfa7a5adebc8d2ab5660f1f; the same native UUID01a14666-43c8-7000-bab0-d9c2e2bb2890read the guidance in attempt 941, currently running under driver PID 226782. The original Kimi report ID isreceive:rewrite-kimi:893:4e9ee3cd15600e2468722b117b5d22b8; its report delivery failed with exit 1 and receipt remains NULL. No Kimi retry was made.Farm sample at 02:51 UTC: user units
acp-render-farm.serviceandacp-render-farm-gpu1.servicewere active. Four assigned containers were running with restart count 0; CPU readings ranged from about 102% to 156%. The boundednvidia-smisample showed 0% GPU utilization on the three enumerated devices. GPU0 admission logged power-brake/thermal slowdown code0x20; GPU3 remained user-held. No GPU controls were used. This confirms active queue processes, not completed GPU rendering.The combined memory and invalid-tail Main11 build is compiling source commit
0724077c03f4802f2a396a6fecec5d717c423c32underbaton-main11-0724077c-final-memory-nul-o0.service(Bun PID 227093); no executable artifact is available yet. Full build and runtime acceptance remain open under #769, with retry behavior linked in #786.flip-engineer commentedon Oct 17, 2026
2026-10-17 update: the existing
knowledge-deepseeksession owns the original dispatch under UUID01a147c2-99b1-7000-bcb7-cbaea84d2e53; its receipt remains pending. The fframes session completed attempt 694, then its route was guardedly changed to standardmuse-wonder-1.0at high effort without changing UUID01a1475d-d987-7bf2-8968-b31165d91a60. One continuation was accepted on that UUID and attempt 735 is running; its owned read is not recorded yet. The exact-hash #769 review update was admitted to existingrewrite-deepseek; attempt 953 is running, but the update has not been read yet. Current review and build state is in the open-task cross-reference commitbd2d21f33.flip-engineer commentedon Oct 17, 2026
October 17 continuation and NUL-tail boundary
The exact-hash review is read under the existing rewrite-DeepSeek native, attempt
receive:rewrite-deepseek:984:691184f811a59c6b8a14f0707ae2ba0e. Follow-up source tracing corrected the initial checkpoint concern: the full-spool witness gates successful input settlement; it does not gate process-attempt retirement. A nonempty NUL-corruption error keeps the owed inbox input unsettled.A separate preservation constraint remains. The normal keeper exit-zero acknowledgement path in
process-spawn.c:1100-1111unlinks stdout; the owner-loss path preserves it. The approved pure Bend2 recovery scope therefore keeps an exit-zero NUL tail incomplete. Custody-proven owner loss with no status marker can retire the dead process attempt while retaining the original raw spool, checkpoint, native UUID and owed input. Failed-observation retirement after exit zero remains a separate process-acknowledgement task linked to #769 and #786. The reviewed helper passed focused type checking; build and crash-shaped runtime evidence remain open.The fframes lead has continued under standard
muse-wonder-1.0at high effort on UUID01a1475d-d987-7bf2-8968-b31165d91a60, attemptreceive:fframes-lead:735:5e1acf5a17c271f828ce9dce9bef577d. Its latest report says native-gpu is actively editing and should remain undisturbed; material-shader tool calls are repeatedly skipped by queued-message preemption; simulation and quality had Kimi quota failures. At terminal attempts 743 and 681, both Kimi routes were guardedly changed to DeepSeek Flash while preserving their OMP endpoints and native UUIDs. Their same-UUID continuations are now running and own 11 and 2 retained inputs respectively. Their output is still active work, not completed validation. The open-task cross-reference records the exact execution IDs and PID births.flip-engineer commentedon Oct 17, 2026
At 2026-10-17 04:29 UTC, the retained Main executable endpoint was atomically replaced with the scoped host-C handover and SQLite projection fix. The installed SHA-256 is
9cc011e95da93526c433de7887863f3ef8d10df55043011125fba2ba52d602c5. The previous executable remains as a same-directory rollback hardlink; its hash, inode, owner and mode were independently verified after installation.The executable uses cached C from source
9bdc172, with exact replacement of the two-form process-spawn fragment (93326470…) and SQLite fragment (02c18aa8…). It does not include the Bend callback changes in PR #795.The public
receivefixture exposed a distinct custody form: it holds the attempt stdout and session lock, and does not holdobserver.log. Its guard is being corrected and tested separately. Existing internal/recovery observer rotation provides temporary containment while the full Bend callback build continues in C generation. The callback fix, public-form qualification and normal source consolidation remain open under #769, #776 and PR #795.