Repository navigation
test(fuzz): skip arithmetic inputs the diagnostic would echo back - #2667
Merged
Merged
Conversation
An arithmetic error names the expression and the unparsed rest verbatim, so the script's own text returns in stderr -- real bash does the same. When that text happens to contain a UNIVERSAL_BANNED substring, assert_no_leak trips on an echo rather than a leak: the tokenizer's Tok enum is never formatted. Fuzz run 271 found `:A>>=::TTA:::Tok::::`, whose diagnostic reads `... (error token is ":A>>=::TTA:::Tok::::")`, and the arithmetic diagnostic is not one of the real-shell templates strip_real_shell_error_lines knows. arithmetic_fuzz now pre-filters with input_echo_would_trip, the defense glob_fuzz and cpython_fuzz already use, keeping the leak detector strict for genuine internals. Of the seven targets in the nightly fuzz matrix this was the only gap: glob and cpython already filter, and parser, lexer, analyze and cpython_http do not run the leak check. Two regressions cover it: the run 271 input is recognized by the pre-filter and its diagnostic is still a bounded, faithful echo, and the same expression without the banned text still goes through the unfiltered leak check, so the filter is not blanket-suppressing this path. Claude-Session: https://claude.ai/code/session_01BegucbcUgCEsPZiAzt6GRx
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
bashkit | 7371f36 | Commit Preview URL Branch Preview URL |
Oct 10 2026, 10:43 AM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
arithmetic_fuzzpre-filters its input withinput_echo_would_trip, thedefense
glob_fuzzandcpython_fuzzalready use, so inputs whose own textwould come back inside the diagnostic no longer trip the leak detector.
No production code changes — fuzz harness and tests only.
Why
An arithmetic error names the expression and the unparsed rest verbatim, so the
script's own text returns in stderr. Real bash does the same. When that text
happens to contain a
UNIVERSAL_BANNEDsubstring,assert_no_leakfires on anecho rather than a leak — the tokenizer's
Tokenum is never formattedanywhere.
Fuzz run 271 (on current main) found a 20-byte input that does exactly this:
testing.rsalready anticipates this class —Tok::is the example instrip_real_shell_error_lines's own doc comment — and provides two independentdefenses: a diagnostic that matches a recognized real-shell template gets
stripped, or the target pre-filters at the input layer. Arithmetic diagnostics
are not one of those templates (
SHELL_ERROR_SUFFIXEScoverscommand not found, the errno templates and redirect refusals), andarithmetic_fuzzhad neither defense. That is the whole bug.Of the seven targets in the nightly fuzz matrix this was the only gap:
globand
cpythonalready pre-filter, andparser,lexer,analyzeandcpython_httpnever call the leak check.Before / After
Reproduced locally through the CLI (output above), then covered by two
regressions, both passing with the 9-case scaffold suite:
run_271_banned_shape_comes_from_the_input— asserts the pre-filterrecognizes the input, and that the diagnostic is still a faithful,
bounded echo, so the skip hides a false positive rather than a real leak.
same_shape_without_the_banned_text_is_still_checked— the same expressionshape with
X::instead ofTok::is not filtered and still goes throughthe unfiltered leak check, proving the filter is not blanket-suppressing this
code path.
Risk
are exactly the ones whose echo is indistinguishable from a leak. The second
regression pins that the surrounding path is still checked.
UNIVERSAL_BANNEDis untouched, so a shape bashkit really emits still fails.
Checklist
https://claude.ai/code/session_01BegucbcUgCEsPZiAzt6GRx
Generated by Claude Code