Skip to content

test(fuzz): skip arithmetic inputs the diagnostic would echo back - #2667

Merged
chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-fasrpn
Oct 10, 2026
Merged

chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-fasrpn

Conversation

@chaliy

@chaliy chaliy commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

What changed

arithmetic_fuzz pre-filters its input with input_echo_would_trip, the
defense glob_fuzz and cpython_fuzz already use, so inputs whose own text
would come back inside the diagnostic no longer trip the leak detector.

No production code changes — fuzz harness and tests only.

Why

An arithmetic error names the expression and the unparsed rest verbatim, so the
script's own text returns in stderr. Real bash does the same. When that text
happens to contain a UNIVERSAL_BANNED substring, assert_no_leak fires on an
echo rather than a leak — the tokenizer's Tok enum is never formatted
anywhere.

Fuzz run 271 (on current main) found a 20-byte input that does exactly this:

$ bashkit -c 'echo $((:A>>=::TTA:::Tok::::))'
bash: line 1: :A>>=::TTA:::Tok:::::  syntax error: operand expected
(error token is ":A>>=::TTA:::Tok::::")
                              ^^^^^^ the banned shape `Tok::`, supplied by the input

testing.rs already anticipates this class — Tok:: is the example in
strip_real_shell_error_lines's own doc comment — and provides two independent
defenses: a diagnostic that matches a recognized real-shell template gets
stripped, or the target pre-filters at the input layer. Arithmetic diagnostics
are not one of those templates (SHELL_ERROR_SUFFIXES covers
command not found, the errno templates and redirect refusals), and
arithmetic_fuzz had neither defense. That is the whole bug.

Of the seven targets in the nightly fuzz matrix this was the only gap: glob
and cpython already pre-filter, and parser, lexer, analyze and
cpython_http never call the leak check.

Before / After

Before: arithmetic_fuzz run 271 -> deadly signal, exit 77
        "stderr leaks banned shape `Tok::`" on the input's own text
After:  the input is skipped at the input layer; the leak detector stays
        strict for shapes bashkit actually emits

Reproduced locally through the CLI (output above), then covered by two
regressions, both passing with the 9-case scaffold suite:

  • run_271_banned_shape_comes_from_the_input — asserts the pre-filter
    recognizes the input, and that the diagnostic is still a faithful,
    bounded echo, so the skip hides a false positive rather than a real leak.
  • same_shape_without_the_banned_text_is_still_checked — the same expression
    shape with X:: instead of Tok:: is not filtered and still goes through
    the unfiltered leak check, proving the filter is not blanket-suppressing this
    code path.

Risk

  • Low. Test-only; no production code, limits or diagnostics change.
  • The pre-filter costs coverage of inputs containing banned substrings, which
    are exactly the ones whose echo is indistinguishable from a leak. The second
    regression pins that the surrounding path is still checked. UNIVERSAL_BANNED
    is untouched, so a shape bashkit really emits still fails.

Checklist

  • Tests added or updated
  • Backward compatibility considered

https://claude.ai/code/session_01BegucbcUgCEsPZiAzt6GRx


Generated by Claude Code

An arithmetic error names the expression and the unparsed rest verbatim, so
the script's own text returns in stderr -- real bash does the same. When that
text happens to contain a UNIVERSAL_BANNED substring, assert_no_leak trips on
an echo rather than a leak: the tokenizer's Tok enum is never formatted. Fuzz
run 271 found `:A>>=::TTA:::Tok::::`, whose diagnostic reads
`... (error token is ":A>>=::TTA:::Tok::::")`, and the arithmetic diagnostic
is not one of the real-shell templates strip_real_shell_error_lines knows.

arithmetic_fuzz now pre-filters with input_echo_would_trip, the defense
glob_fuzz and cpython_fuzz already use, keeping the leak detector strict for
genuine internals. Of the seven targets in the nightly fuzz matrix this was
the only gap: glob and cpython already filter, and parser, lexer, analyze and
cpython_http do not run the leak check.

Two regressions cover it: the run 271 input is recognized by the pre-filter
and its diagnostic is still a bounded, faithful echo, and the same expression
without the banned text still goes through the unfiltered leak check, so the
filter is not blanket-suppressing this path.

Claude-Session: https://claude.ai/code/session_01BegucbcUgCEsPZiAzt6GRx
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit 7371f36 Commit Preview URL

Branch Preview URL
Oct 10 2026, 10:43 AM

@chaliy
chaliy merged commit 28a7f02 into main Oct 10, 2026
33 checks passed
@chaliy
chaliy deleted the eve/clever-mccarthy-fasrpn branch October 10, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant