Skip to content

Keep retained Codex assistant frames in spool spans - #869

Closed
flip-engineer wants to merge 3 commits into
mainfrom
codex/codex-large-frame-span-20261018
Closed

flip-engineer wants to merge 3 commits into
mainfrom
codex/codex-large-frame-span-20261018

Conversation

@flip-engineer

@flip-engineer flip-engineer commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Large Codex assistant and failed-terminal frames pass through repeated Bend String and SQL construction. An 8 MiB frame exhausted a 768 MiB recovery scope before the observer cursor advanced.

This change reads retained Codex frame spans, stores span references in checkpoint state, and persists the exact terminal event and report through a SQLite transaction. Generic observations omit the large assistant text and error message fields; the retained spool supplies the complete result. Failed terminals establish tagged span state, including when recovery starts from a legacy assistant checkpoint.

Validation: an earlier revision of the focused production entry passed Bend check-only remotely. The previous candidate failed the 768 MiB runtime gate. Corrected Bend source is published at ed9fba70af00fb0c305a5de59f8cca934ca1de5c, with the test-only timing correction at PR head d461a5611b2b2819e1e7803c8b92ff0da5f9c9c9; a self-hosted 16 GiB, zero-swap focused build reached its 1800-second deadline with no generated C or binary. Its launcher pinned earlier turn/fixture hashes (425244… / 030de5…), while the worktree later contained 57f53c… / 3f0bc0…; that invocation cannot validate the published source. The scope reported timeout and no OOM events; sampled compiler RSS ranged from about 216 to 441 MiB. Its compiler phase remains unclassified because the invocation did not enable stage diagnostics. A short diagnostic run requires a frozen current source closure and a separate manifest. Diff checks and Python fixture syntax checks passed. Runtime qualification is pending for 8 MiB assistant and failed-terminal frames, checkpoint reopen, exact persisted native event/report, cursor advancement, native identity, custody, and acknowledgment state. This PR remains a draft until those gates pass.

@flip-engineer

Copy link
Copy Markdown
Collaborator Author

A separate 120-second diagnostic was run with the current source pinned and rechecked: HEAD d461a5611b2b2819e1e7803c8b92ff0da5f9c9c9, tree 78d45451005a67dcceb44d2e13a86091bd463db2, turn prefix 57f53cd2, fixture prefix 3f0bc03a, and recovery entry prefix 766c647e.

Scope codex-span-diag-d461-retry1-20261018, invocation ffeb7521134441f698ebe7ce9e875240, had a 120-second runtime limit, 16 GiB memory limit and zero swap. It reached the runtime limit and produced no C file. Stdout and stderr were empty despite the diagnostic environment setting. The stage receipt contains the start time only; the timeout interrupted the footer. No peak-memory measurement was available. This diagnostic does not qualify the candidate or establish a compiler phase bottleneck. Compiler diagnostic configuration needs inspection before another attempt.

The earlier 30-minute run retains the source-attribution correction already recorded. No candidate runtime or deployment claim is made. #776 tracks the compiler blocker; #870 and draft #876 track the separate checkpoint restore path; #875 and #877 track guarded observer recovery.

@flip-engineer

Copy link
Copy Markdown
Collaborator Author

The new legacy-checkpoint fixture needs a source correction before runtime qualification. retained_output now requests spanned reads for every retained Codex frame. Frame.decode(..., True) produces SpanFrame, and codex_frames_span_role stores assistant frames using codex-span-v1. Therefore test_codex_large_failed_terminal_replaces_legacy_checkpoint cannot expect a raw JSON assistant line after consuming it through the new runtime.

Create a genuine old-format checkpoint with a fixture or previous runtime, preserving its valid header, cursor and custody. Then reopen it with the new runtime and process the large failed terminal frame. Keep the exact result/report, native identity, pending-input and memory checks. This finding does not establish a production failure in the new span path.

@flip-engineer

Copy link
Copy Markdown
Collaborator Author

The final16 log proves two legacy reopen tests using the retained PR835 acceptance binary. It does not qualify the Codex span implementation in this PR.

I checked the retained build receipt and source objects directly. Binary ad9ee93a3977d9df30f0c22318ae054c7f8a419a694bba77980224a49ac36c8c is built from source 3fe6380de75432442b179987324fab2f7084fadc, whose parent is 788b806db8aefbe8453511bf19c615a1efa48bc1. Its turn.bend SHA-256 is 9d7637b824cd9d0771761dae9f12f5c5ba460f7b6a4026c378c72738ff3303f7; that source has no CodexSpanState or codex_span_observe_fields. The receipt identifies the PR835 retained-output acceptance scope.

PR head d461a5611b2b2819e1e7803c8b92ff0da5f9c9c9 has turn.bend blob 0433182ddd00416be2209b67d3fdf06acbbf3d7f, which includes those Codex changes. The final16 test log remains valid evidence for its two assertions. Runtime qualification of this PR's 8 MiB Codex assistant and failed-terminal paths remains pending on a binary built from the matching source closure. The PR remains a draft.

@flip-engineer flip-engineer left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Escalated to the root runtime/compiler review under #943 and #965. Prior capped or timed-out compiler attempts do not qualify this source. The PR835 final16 binary also does not contain this Codex span implementation. Correct the legacy-checkpoint fixture so it creates an actual old-format checkpoint before reopening it. Then build the exact current candidate using the selected compiler with the merged #967/#968 changes and qualify the large assistant and failed-terminal paths, checkpoint reopen, persisted result/report, cursor, custody and native identity. Preserve all original failure evidence. This draft remains open for development and root review.

Review disposition recorded 2026-10-19. Completed and validated work proceeds through root review, merge and deployment; remaining work stays assigned for development or escalation.

flip-baton-claude Bot added a commit that referenced this pull request Oct 24, 2026
The candidate and target checks of the previous head failed in prepare-native-compiler.py with 'Unknown compiler archive SHA256': the branch's verifier did not list the pinned compiler archive 6609a2de that main now lists.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@flip-engineer

Copy link
Copy Markdown
Collaborator Author

Root source review is complete. Current main contains this work through merged PR #1088 (3c4f9a0458c922417b1e83f8a0cf4f2fcd52166a). An independent native reviewer traced the retained Codex span/checkpoint, small-frame continuation, monitor-only replay and custody, invalid-checkpoint handling, instance-owner entry, and result/report requirements against current source, tests and documentation.

turn.bend classifies complete retained JSON spans through SQLite and preserves codex-span-v1 state. Current observer code and observer-monitor.py cover cursor commits, latest assistant and failed-terminal data, legacy reopen, metadata ordering, monitor custody, unknown native exit and unread input. The exact escaped-backslash boundary regression is merged in #1148. Root checked that test's 5300-byte fixture and escaped boundary; the focused native observer test has not been rerun here.

This older proposal is superseded by the integrated source. Its truncated JSON-prefix classifier can reject valid escaped input at byte4096 and must remain superseded. Root is closing #1047, #998, #869 and #901 on this source disposition. Replay-throughput work remains open in #1046; exact current-package runtime qualification and installed wake/chat-injection proof remain separate work in #1054 and the active critical package assignment. Closing these obsolete PRs does not claim those delivery results.

@flip-engineer
flip-engineer deleted the codex/codex-large-frame-span-20261018 branch October 27, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants