[mattlorimor]

Fixes #344.

The cut sweep can't see a check that holds a count to the bytes after it, when the writer keeps that count small (#332). This PR audits every such check in the library, adds edited-claim tests for the four nothing caught, and adds a roster test so a new check can't go untested. It's test-only, plus a TESTING.md note, and follows the coordinator's plan on #344, approved by Matt.

The audit

The roster comes from the code: a parse of the library with comments and strings blanked. It found 31 RequireHolds calls, plus the 7 checks inside PayloadReader that hold a claimed length to the bytes left. Each check was dropped through scripts/mutate.sh and run against the whole suite at 6e2e5ec.

• 34 of the 38 drops were killed. The full table, with each check's file, line, what it holds and how many tests failed, is in ccd3f49's message.
• CountSketch, IBLT, DDSketch and QuotientFilter call no RequireHolds. Their runs are held by BeginRun and Skip, and both drops are killed.

The four survivors:

Check	What it holds	Why the sweep missed it
HeavyKeeper:686	element count	The count is at most k (500 in the sample), and the only cuts that keep it come after 80 KB of buckets, where the bound has room for a heap of k elements.
TopK:408	element count	The same, with k = 1,000, after a 110 KB nested sketch.
SublimeCountMinSketch:1166	current rows' counters	Each block is checked again where it begins, so every cut it refuses is refused later anyway. An edited depth isn't: at depth 64, without the check the price runs to offset 32,838 and the read allocates 118,024 bytes against 82,768.
PayloadBuffer:974 (Slice)	a nested envelope's length against its run	No cut can make the two disagree. Dropped, a held payload's slice throws ArgumentOutOfRangeException in place of InvalidDataException.

None of the four is unreachable, so none is proposed for removal.

The tests

All are in TestReadsAllocateOnlyWhatTheyHold.cs.

• TestAClaimNoCutCanSeeIsRefusedBeforeItIsPricedOrAllocatedWhenEdited: three edited-claim cases, in Test DpswSketch's substream price check with an edited count (#332) #340's pattern. They are HeavyKeeper's and TopK's element counts, with k raised to the heap's limit, and Sublime's depth.
  • Each claim is edited in the sweep's own sample, checksum resealed: once to one past what the bytes hold, once to the largest the field allows.
  • The price must stop exactly where the check refuses (offsets from FORMAT.md), having priced exactly what a cut just before the field prices.
  • The read is refused for the missing bytes under its own price, the default budget and none. It allocates no more than that cut's read, plus the envelope's difference and 1 KiB.
  • Control: the most the bytes can hold gets past the check, and its read allocates 15,472 to 154,848 bytes past the bound. Test DpswSketch's substream price check with an edited count (#332) #340's envelope + Fixed bound was too loose here, since Sublime's control fell inside 16 KiB.
• TestANestedEnvelopeClaimingMoreThanItsRunIsRefusedAsData: the nested envelope's length is edited to the least its run can't hold. Its roster is the structures whose price calls PriceNested, taken from IL: TopK and ScalableBloomFilter.
• TestEveryHoldsCheckIsHeldByAnEditedClaimOrNamesTheTestThatHoldsIt: the roster is the library's IL, every RequireHolds call. Each check must be one of the edited claims, or be listed in HeldElsewhere with the test that kills its drop, verified to be a real test.
  • Stale entries fail.
  • The edited claims' own checks must be found, so a scan that finds nothing can't pass.

A sweep over FORMAT.md or StructureId wasn't feasible. FORMAT.md's layouts are prose code blocks, with version conditions and nested groups that no test parses. StructureId doesn't name checks. The IL is the roster that can be derived.

Watched failing, each drop applied by hand:

• HeavyKeeper's price priced 116,536 bytes and stopped at 91,961, not 81,961.
• TopK's priced 232,496 and stopped at 128,827, not 108,827.
• Sublime's, at depth 10, priced 22,048 and stopped at 5,190, not 32.
• Slice: ArgumentOutOfRangeException from both nesting structures.

Mutation of the new tests: 20 mutants, all killed, against the whole suite and by the three new tests alone. They are:

• the four drops;
• each bound a byte looser, and a byte stricter;
• each element-count check moved after its price, or after its charge;
• for the rosters: a check added to BloomFilter.Price, the TopK case removed, a stale entry added, and a nesting structure left out.

TESTING.md

A note under Persistence: the cut sweep can't see a check on a count the writer keeps small, so such a check needs an edited claim. It also says what this audit found.

Left for later

An envelope's claimed length is held only by the read budget. A held 128,827-byte TopK payload whose header claims 2^30 bytes allocates 1,073,743,704 bytes with no budget. Under the default budget it's refused for its budget, not as malformed. Nested envelopes behave the same way. Matt has decided that held lengths should be checked against the bytes held, and a separate round will do it.

Verification

• Clean Release build with -warnaserror: exit 0, 0 warnings.
• Full suite: 2,529 passed, 0 failed, run by the author and again by the coordinator in a clean clone.
• The new tests, and Test DpswSketch's substream price check with an edited count (#332) #340's, also pass under the segments GC, as macOS runs it.
• A trial merge with QuotientFilter read leftovers: Build's throwaway table, 72 bytes uncharged, a possibly untested price #343's branch is clean.
• No CHANGELOG entry, as for Test DpswSketch's substream price check with an edited count (#332) #340.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx