Skip to content

fix(tests): cap bounded-child RSS instead of address space - #2701

Merged
chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-rxj7d8
Oct 12, 2026
Merged

chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-rxj7d8

Conversation

@chaliy

@chaliy chaliy commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

What changed

awk_field_depth_bounded's supervised-child launcher no longer caps the child's
address space. Memory containment moves to a parent-side peak-RSS watchdog that
samples the live child, kills it on breach, and asserts the peak afterwards, so
growth still fails closed. CPU, stack, core, output-drain and wall bounds are
unchanged.

A static scan test now fails any test launcher that reintroduces an
address-space cap, in either the shell spellings (ulimit -v, ulimit -d) or a
direct setrlimit (RLIMIT_AS, RLIMIT_DATA), with an escape hatch
(// rlimit-as-ok: <reason>) for a launcher that provably never builds with a
sanitizer.

Why

The nightly AddressSanitizer job went red on run 257 while required CI stayed
green on the same commit (fa2262d). The launcher set ulimit -v 2097152;
AddressSanitizer reserves ~14 TiB of virtual address space for its shadow map
during startup, so the instrumented child died in ReserveShadowMemoryRange
before executing a line of test code. The test then failed on the launcher's own
cap rather than on the depth bound it asserts — the same divergence shape as the
CLI-path bug in run 252, and awk_field_depth_tests.rs was the only harness
still using an address-space cap.

RSS is also the better proxy for the runaway allocation the bound exists to
catch, and it costs nothing in virtual space, so one mechanism holds on
instrumented and native builds alike.

Before / After

Root cause, reproduced standalone (the failure is in ASAN startup, independent of
bashkit code):

$ gcc -fsanitize=address -o t t.c   # int main(void){return 0;}
$ /bin/sh -c 'ulimit -t 10 && ulimit -s 8192 && ulimit -c 0 && ulimit -v 2097152 && exec "$@"' x ./t
==524==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)
==524==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v
exit=134

$ /bin/sh -c 'ulimit -t 10 && ulimit -s 8192 && ulimit -c 0 && exec "$@"' x ./t
exit=0

The reservation size 0xdfff0001000 is byte-identical to the one in the nightly
log, confirming the same mechanism.

Before, under the real nightly invocation (-Z build-std, -Z sanitizer=address):

thread 'awk_field_depth_tests::awk_field_depth_bounded' panicked at
  crates/bashkit/tests/integration/awk_field_depth_tests.rs:82:5:
child status exit status: 1: ==15767==ERROR: AddressSanitizer failed to allocate
  0xdfff0001000 (15392894357504) bytes ...
test result: FAILED. 2409 passed; 1 failed

After, same invocation and flags on this branch:

$ RUSTFLAGS="-Z sanitizer=address" ASAN_OPTIONS="detect_stack_use_after_return=1:detect_leaks=0" \
  cargo +nightly test -Z build-std --target x86_64-unknown-linux-gnu \
  -p bashkit --test integration awk_field_depth -- --test-threads=1
test awk_field_depth_tests::awk_field_depth_bounded ... ok
test awk_field_depth_tests::awk_field_depth_child ... ok
test awk_field_depth_tests::finite_field_operands_match_gawk ... ok
test result: ok. 3 passed; 0 failed

The replacement bound is not vacuous: the child's measured peak RSS is 29,216 KiB
over ~4 polls in a 45 ms lifetime, against the 2 GiB ceiling (the same 2 GiB
figure as the old cap, now measured on resident rather than virtual memory), so
an ASAN build fits comfortably while multi-gigabyte growth still trips it.

Guard verified by mutation — reintroducing the cap fails the scan with the
offending line:

address-space caps break sanitizer-instrumented children; cap RSS from the parent
instead (knowledge/operations/testing.md):
crates/bashkit/tests/integration/awk_field_depth_tests.rs:82: let limits = "... ulimit -v 2097152 ..."

Risk

  • Low. Test-harness only; no production code changes.
  • The child keeps its CPU, stack, core, output-drain and wall bounds, and gains
    an RSS bound it did not have. The removed cap is the one that could not
    coexist with a sanitizer.
  • Residual: RSS is sampled on a 10 ms poll, so an allocation spike that both
    starts and ends between two polls is unmeasured. status.success(), the stack
    and CPU limits and the wall watchdog still fail such a run, and the depth guard
    under test is what bounds the recursion in the first place.

Checklist

  • Tests added or updated
  • Backward compatibility considered (internal test harness)

https://claude.ai/code/session_013mA6kgZvfp234w1L2gLint


Generated by Claude Code

The `awk_field_depth_bounded` launcher capped the child's address space
with `ulimit -v 2097152`. AddressSanitizer reserves roughly 14 TiB of
*virtual* address space for its shadow map during startup, so the
instrumented child died in `ReserveShadowMemoryRange` before executing a
line of test code. The test then failed on the launcher's own cap rather
than on the depth bound it asserts, which is why nightly run 257 went red
while required CI, which is not instrumented, stayed green on the same
commit -- the same divergence shape as the CLI-path bug in run 252.
`awk_field_depth_tests.rs` was the only harness still using such a cap.

Memory containment moves to a parent-side peak-RSS watchdog that samples
the live child each poll, kills it on breach and asserts the peak
afterwards, so a child that breached on its final poll and then exited on
its own still fails closed. CPU, stack, core, output-drain and wall bounds
are unchanged. RSS is also the better proxy for the runaway allocation the
bound exists to catch and costs nothing in virtual space, so one mechanism
holds on instrumented and native builds alike; it is the convention the
other bounded-child harnesses already use.

The replacement bound is not vacuous: the child's measured peak is
29,216 KiB over roughly four polls in a 45 ms lifetime, against a 2 GiB
ceiling (the same figure as the old cap, now measured on resident rather
than virtual memory), which an ASAN build fits while multi-gigabyte growth
still trips it.

A static scan now fails any launcher that reintroduces an address-space
cap, in either the shell spellings or a direct `setrlimit`, with a
`// rlimit-as-ok: <reason>` hatch for one that provably never builds with a
sanitizer.

Verified by running nightly's own invocation (`-Z build-std`,
`-Z sanitizer=address`) on this change: 3 passed, 0 failed, where
`awk_field_depth_bounded` previously failed. The mechanism was reproduced
standalone with a 3-line ASAN binary, which fails under the old cap with
the byte-identical `0xdfff0001000` reservation from the nightly log and
exits 0 without it. Reintroducing the cap fails the new scan.

Claude-Session: https://claude.ai/code/session_013mA6kgZvfp234w1L2gLint
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit dd6bea0 Commit Preview URL

Branch Preview URL
Oct 12 2026, 09:21 AM

@chaliy
chaliy merged commit 46e7d8f into main Oct 12, 2026
33 checks passed
@chaliy
chaliy deleted the eve/clever-mccarthy-rxj7d8 branch October 12, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant