trace: git push hangs on long AI sessions because secret redaction is slow #3498

Activity

1. migmartri

   added

   bugSomething isn't workingSomething isn't working

   component/CLI

   on Sep 30, 2026
2. migmartri commented on Sep 30, 2026

   

   migmartri

   on Sep 30, 2026

   Last edited by migmartri

   MemberAuthor

   More actions

   Research

   I reproduced the push pipeline offline with the reported session: ParseSession, evidence encoding, crafter validation, and redaction. The upload was not included. Machine: 8 cores.

   Where the time goes

   Step	Time
   Parse the session, encode the evidence (190 MB), decode and validate it	approximately 8 s (heap peak 1.3 GB)
   One betterleaks scan pass, stdlib regexp (current code)	10m23s
   Full Redact (3 passes for this session), stdlib regexp	approximately 31 min (estimated: 3 × one pass)
   Full Redact, RE2 engine	5m20s (measured)

   In the CPU profile, 98% of the time is in regexp.(*Regexp).FindAllStringIndex called from detect.(*Detector).detectFragmentWithRule. The scan uses only one core.

   Root causes

   1. Slow regex engine. The betterleaks CLI selects the RE2 engine (github.com/betterleaks/go-re2, which runs on wazero) by default. When betterleaks is used as a library, the default engine is stdlib regexp. We use the library, so we get stdlib regexp.
   2. The document is scanned as one fragment. betterleaksScanner.Scan sends the whole indented document as one sources.Fragment. The betterleaks keyword prefilter selects a rule when one of its keywords occurs anywhere in the fragment. A 190 MB fragment contains almost all keywords, so almost all rules run their regex over all 190 MB. The betterleaks file source reads input in 100 KB chunks for this reason.
   3. Each pass rescans the full document. A clean result needs 2 passes, and this session needed 3. Each pass encodes the full document again and scans all of it. Encoding and rewriting add approximately 19 s per pass.
   4. Much of the transcript is repeated content. attachment entries are 72 MB of the 160 MB raw transcripts (45%). The command field of hook_success attachments is 36 MB: one third-party hook command of 2,966 bytes occurs 12,778 times with the same text. Other examples are total_tokens_reminder, prompt_snapshot and skill_listing.

   Per-rule timings (100 KB chunks): generic-api-key uses approximately 31% of the scan time. Approximately 10 rules with common keywords (key, token, secret, ...) run on almost every chunk (1,951 of 1,952).

   Measured options (one scan pass, 190 MB)

   Configuration	Time	Findings
   stdlib, 1 fragment (current)	10m23s	319
   stdlib, 100 KB chunks	1m41s	319
   stdlib, 100 KB chunks, 8 workers	26.5 s	—
   RE2, 1 fragment	1m27s	319
   RE2, 100 KB chunks	13.7 s	319
   RE2, 100 KB chunks, 8 workers	3.95 s	—

   The findings column shows counts from our scanner, which includes composite components. The parallel runs used Detector.DetectString, which does not include components, so their counts are not comparable and are not shown.

   RE2 adds approximately 4.5 MB to the binary. It works with CGO_ENABLED=0.

   Proposed changes, by impact

   • A. Use the RE2 engine. Call regexp.SetEngine(re2.RE2{}) before the detector is built. Result: 7x faster, with the same findings.
   • B. Scan in chunks. Split the document at record or leaf boundaries, so that no secret crosses a chunk boundary. Keep each key next to its value, because the generic rules need that context. With A: 13.7 s per pass.
   • C. Scan chunks in parallel. Use one detector for each worker, because the detector keeps state for each run. With A and B: approximately 4 s per pass.
   • D. Rescan only what changed. After the first pass, scan only the chunks that had replacements, and rewrite only the leaves that had findings. This removes most of the 19 s encoding and rewrite cost for each pass.
   • F. Cache scan results. Keep a cache keyed by the content hash of each record in the trace state, or scan records incrementally from the agent hooks. Each push scans the same growing session again, so with a cache only new records are scanned. This is necessary only if A–D are not sufficient.
   • G. Do not block git push. Run the attestation in the background, so that a large session cannot cause the remote to close the connection.

   Recommendation: do A, B and C first. They are a small change in internal/redaction and change the worst case from approximately 30 minutes to seconds. Then do D, and G as a safeguard.

   Removing the repeated content (root cause 4) is not proposed. Each entry is a real transcript event with its own timestamp, and the evidence stores the transcript as it was captured. Item E, which proposed this, was withdrawn. F reduces the scan cost of repeated content without changing the transcript.

   Compaction does not help. The Claude Code transcript only appends entries, so old entries stay in the file.

   Related: #3481 / #3489 changes the rewrite step (redactLeaf). That change is about correctness, not speed, but it will conflict with D.

   🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri
3. migmartri

   self-assigned this

   on Sep 30, 2026
4. linear-code

   added

   performance

   on Sep 30, 2026
5. migmartri

   added 7 commits that reference this issue on Sep 30, 2026

   fix(redaction): always re-read the last line of a chunk

   ...

   ba82b8d

   perf(redaction): scan AI sessions in parallel chunks with RE2

   ...

   141b781

   fix(redaction): always re-read the last line of a chunk

   ...

   f4c954c

   perf(redaction): consult the path filter only for leaves with a secret

   ...

   31856f6

   perf(redaction): rescan only the chunks that changed between passes

   ...

   665212f

   refactor(redaction): simplify the rewriter and hash chunks in parallel

   ...

   70d8a14

   fix(redaction): honour cancellation when every chunk is unchanged

   ...

   94cb638
6. migmartri

   closed this as completedon Oct 2, 2026