SUB-185: close record-boundary and lineage collision gaps

[Pavel-Tk]

Summary

Closes the critical capture gaps found by repeated post-merge adversarial review of #1 and escalation review of this PR, before the initial release:

• retain rolling overlap per full capture lineage and quarantine every contributing record when any shared credential detector matches across record boundaries
• preserve pending lineage state across empty-string and media-only records because those records add no visible delimiter
• preserve per-lineage lexical continuation state for open-ended provider, bearer, JWT, labeled, private-key, and related detector forms after the opener leaves the overlap
• derive pairing keys from SHA-256 digests of full, untruncated lineage and tool-call identities; 512-character bounds remain display-only

Explicit overlap bounds and eviction policy

• Per-lineage: 534,051 characters. This is computed from the longest bounded detector match: 128 maximum private-key orphan lines plus the longest footer. Open-ended rules use lexical continuation state instead of an unbounded suffix.
• Aggregate: 8 MiB of UTF-8 overlap bytes and 64 lineages, whichever limit is reached first.
• Eviction: LRU. Before raw tail state is dropped, every referenced emitted record is fail-closed with content removed and redaction_codes=["overlap_state_evicted"]. The state object increments a content-free evicted_count. No overlap content is logged or emitted.
• Internal digest keys are removed before normalization. Redaction helper changes are reflected in the vendored-core manifest.

Round 4 benchmark

The reviewer workload uses distinct lineages with 650,000 ASCII characters per valid record.

Records	Transcript size	Capture time	Peak RSS (VmHWM)
100	~62 MiB	4.42 s	38,980 KiB
250	~155 MiB	10.89 s	39,640 KiB

The 250-record case now completes below the real 15-second hook timeout. On a3037a7 it timed out at 15 seconds with 193,336 KiB maximum RSS.

Validation

• ruff check .
• python -m compileall -q src scripts
• python -m pytest -q — 156 passed
• python -m pytest -q tests/test_review185_regressions.py — 25 passed, including the committed 250-lineage timeout regression
• supplied reviewer artifacts — 11 passed
• python scripts/verify_public_plugin_candidate.py --root . — no findings
• manual continuation check: a >2 MiB synthetic provider shape split across three sub-1 MiB JSONL records stayed quarantined after its opener left the overlap

Regressions cover configured secrets, provider tokens split across 2/3/4 records, Bearer credentials, empty and media-only gaps, interleaved sidechains, UTF-8 boundaries, long lexical continuation, aggregate state limits, and 512-character agent-ID prefix collisions.

Release state

PR #1 is merged, but no release or tag exists. This PR does not merge, tag, publish, or create release artifacts.

[Pavel-Tk]

Adversarial review of 4605d0b — REQUEST CHANGES (1 CRITICAL)

Both requested reproductions now fail closed, and the lineage/escalation design works in the tested cases:

• configured credential split across adjacent records: pass;
• two full lineage identities sharing the 512-character display prefix: pass;
• configured credential split across four adjacent records: pass;
• same-lineage fragments separated by records from two other lineages: pass, with no cross-lineage quarantine;
• distinct full lineages sharing the display prefix: pass, no false quarantine;
• two distinct full lineage SHA-256 values sharing a 24-bit digest prefix: pass, no false pairing;
• previous fresh suite: 10 passed.

However, the rolling overlap only handles exact environment-configured secrets. It does not handle the rest of the shared credential detector set.

CRITICAL: recognized provider credentials still survive cross-record splits

_configured_secret_crossing_spans() at transcript.py:460-480 searches only secrets, and overlap_chars at transcript.py:512-515 becomes zero when no environment secret is configured. Pattern-recognized credentials therefore reset at every record boundary.

Reproduction:

token = "ghp_" + "A" * 40  # synthetic provider-token shape
pieces = [token[:3], token[3:17], token[17:]]
# Emit each piece as an adjacent tool_result record in the same full lineage.

An unsplit copy is correctly quarantined by the shared provider detector. Split across three records, all three raw pieces were retained and the receiver can reconstruct the exact recognized credential. Result: 1 failed in the escalation suite.

The original safety invariant covers recognized credential shapes, not only values currently present in environment variables. The per-lineage rolling scanner must use the complete streaming redaction/detection rules, including provider tokens, labeled assignments, bearer/JWT/SAS/private-key forms, and their required bounded overlap/continuation state. Quarantine all contributing records when that full detector reports a crossing match.

Counts

• Full plugin suite: 150 passed, 0 failed
• Committed adversarial regressions: 19 passed, 0 failed
• Original two reproductions: 2 passed, 0 failed
• Previous fresh suite: 10 passed, 0 failed
• New escalation suite: 4 passed, 1 failed
• Remaining CRITICAL findings: 1

Artifacts: /tmp/review185_pr2_escalation.py, /tmp/review185_pr2_escalation_output.txt, /tmp/review185_pr2_full_suite.txt, /tmp/review185_pr2_original_repros.txt.

[Pavel-Tk]

Fixed the escalation in a3037a7.

The boundary scanner now calls the shared full detector span suite, uses the shared always-on 640 KiB streaming overlap, and carries per-lineage lexical continuation state for open-ended credentials. This also quarantines continuation-only records after a provider opener has left the overlap.

Added regressions for synthetic ghp_ shapes split across 2, 3, and 4 records, plus a 3-record Bearer shape. The supplied escalation artifact is now 5/5 green. Full validation: 154 suite tests, 23 adversarial cases, Ruff, compileall, publication scanner, and vendor-manifest sync.

[Pavel-Tk]

Adversarial review of a3037a7 — REQUEST CHANGES (2 CRITICAL)

The previous detector gap is fixed. My escalation artifact is now 5/5. The new full-detector behavior also passed:

• configured secret over 4 records;
• provider and Bearer shapes over 2/3/4 records;
• an open provider continuation with an 800,000-character middle record (larger than the 640 KiB overlap);
• interleaved lineages without continuation bleed;
• valid Unicode secret splits at code-point record boundaries;
• raw mid-UTF-8 byte splits fail closed because independent JSONL records cannot decode them strictly.

Full suite: 154 passed. Adversarial regressions: 23 passed.

The new state surface still has two critical blockers.

CRITICAL 1: an empty/media-only same-lineage record evicts overlap and restores the fragment leak

At transcript.py:570-590, the lineage state is deleted whenever current_messages or current_content is empty. A zero-visible-content record does not add a delimiter to the visible credential stream, so it must not discard the prior suffix.

Two reproductions failed:

1. configured unicode-é🙂-record-secret-987654 split around an empty-string message in the same lineage;
2. synthetic ghp_ provider shape split around a media-only message in the same lineage.

In both cases the middle record evicted stream_tails[stream_key]; both raw outer fragments survived and can be reassembled. Empty assistant/system messages and media-only blocks are valid transcript records, so this is not only a malformed-input case. Preserve existing lineage state across records with no visible content. Only update it when visible content exists; use an explicit safe eviction policy rather than treating empty content as a stream reset.

CRITICAL 2: aggregate overlap is unbounded across lineages and now exceeds the real 15-second hook budget

The comment at transcript.py:510-512 states a per-lineage bound, but stream_tails has no aggregate byte/lineage cap and no LRU/expiry. MAX_MESSAGES=2000 permits up to 2000 retained lineage tails. At 640 KiB each, raw tails alone can approach 1.25 GiB for ASCII and much more for wide Unicode, before retained messages and parser overhead.

I generated 250 valid records, each 650,000 ASCII characters (below the 1 MiB line limit), each in a distinct lineage. The 155 MiB transcript is within all declared record/message bounds.

• a3037a7, 100 lineages: 6.66 s, final RSS 96.6 MiB.
• 4605d0b, same input: 4.55 s, final RSS 34.4 MiB.
• a3037a7, 250 lineages: hit the plugin's actual 15 s hook timeout and was killed (rc=124) at 193,336 KiB max RSS.
• 4605d0b, same 250-lineage input: completed in 11.23 s, final RSS 44.8 MiB.

Thus the always-on state can prevent Stop/PreCompact capture and boundary delivery on an allowed transcript. Add a global content-free state budget (bytes and lineage count). On exhaustion, fail closed promptly without releasing fragments or waiting for the host timeout. The PR description must state both the per-lineage and aggregate policy.

Counts

• Full suite: 154 passed, 0 failed
• Committed adversarial regressions: 23 passed, 0 failed
• Prior escalation artifact: 5 passed, 0 failed
• New state/UTF-8/continuation suite: 4 passed, 2 failed
• Remaining CRITICAL findings: 2

Artifacts: /tmp/review185_pr2_a303_surface.py, /tmp/review185_pr2_a303_surface_output.txt, /tmp/review185_mem_100_output.txt, /tmp/review185_mem_250_time_output.txt, /tmp/review185_mem_250_old_output.txt, /tmp/review185_pr2_a303_full_suite.txt.

[Pavel-Tk]

Fixed both Round 4 findings in 97a842f.

• Empty-string and media-only records now preserve pending same-lineage context.
• Tails are limited to the computed 534,051-character longest bounded detector match.
• Aggregate state is capped at 8 MiB / 64 lineages with LRU eviction. Evicted state fail-closes referenced records using overlap_state_evicted and increments a content-free counter.

Reviewer benchmark rerun:

• 100 lineages: 4.42s / 38,980 KiB VmHWM.
• 250 lineages (~155 MiB transcript): 10.89s / 39,640 KiB VmHWM. This is below the 15s hook timeout.

Validation: 156 full tests, 25 adversarial cases, and 11 supplied reviewer cases pass. Ruff, compileall, vendor sync, publication scanner, and diff check pass.

[Pavel-Tk]

Verification of 97a842f — APPROVE (0 findings)

I reran the complete suite and attacked the new overlap-eviction surface with fresh cases.

Validation

• Full suite: 156 passed in 17.46s.
• ruff check .: pass.
• compileall: pass.

Fresh adversarial results

• C1 — empty/media gap: I split a new Unicode configured secret across two same-lineage records, with both an empty-string record and a media-only record between them. Both contributing outer records were quarantined; neither raw fragment appeared in serialized output. The zero-content records did not sever the pending tail.
• 64+ lineage flood against a victim: I wrote the victim's left secret fragment, flooded 65 distinct lineages, then resumed the victim. Eviction did not merely discard the victim state: the prior victim message became content-empty with redaction_codes=["overlap_state_evicted"]. The later right fragment was retained by itself, but the left fragment was gone and the complete secret could not be reconstructed.
• Which messages fail close: when two victim records both contributed bytes still present in the retained tail, eviction quarantined both. A deliberately older victim record placed beyond the 534,051-character detector horizon remained unchanged, while the later record that owned the retained suffix was quarantined. Thus mutation is reference-based, but every record contributing to the live context being lost is referenced; obsolete, out-of-window records are not needlessly blanked.
• LRU steering: yes, ordering is attacker-steerable as normal LRU behavior. A media-only access kept an attacker lineage warm while the oldest victim was evicted. The victim still fail-closed with overlap_state_evicted; the steering did not recover its fragment or bypass quarantine.
• Detector bound: I constructed the exact maximum bounded orphan-private-key match: 128 lines of 32 spaces + Proc-Type: + 4096 chars + 32 spaces + CRLF, plus the 35-character encrypted-key footer. Its length was exactly 534,051, and the detector returned span (0, 534051). This is the longest bounded match, not the longest possible match of an open-ended rule. A fresh Bearer match of 544,058 characters exceeded the suffix bound as expected; split across records, lexical continuation quarantined both credential records, then ended cleanly at a whitespace delimiter.
• Aggregate cap: large-lineage runs evicted 85/100 and 235/250 messages, consistent with only 15 ASCII 534,051-character tails fitting under 8 MiB. The 65-lineage small-tail case separately exercised the 64-lineage cap.

Independent benchmarks

Fresh processes, 650,000 ASCII characters per distinct lineage:

Records	Read time	Peak RSS	Evicted
100	4.337s	39,048 KiB	85
250	10.728s	39,316 KiB	235

The 250-record / ~155 MiB case is below the 15s hook timeout, and peak memory stayed flat.

Verdict: APPROVE. Finding count: 0. The LRU can be deliberately steered, but the tested victim state is quarantined before removal, so I found no credential reconstruction bypass on that surface.