Skip to content

Test DpswSketch's substream price check with an edited count (#332) - #340

Merged
mattlorimor merged 2 commits into
masterfrom
test/332-dpsw-substream-price-check
Nov 6, 2026
Merged

Test DpswSketch's substream price check with an edited count (#332)#340
mattlorimor merged 2 commits into
masterfrom
test/332-dpsw-substream-price-check

Conversation

@mattlorimor

@mattlorimor mattlorimor commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Fixes #332.

DpswSketch's price and reader both check a window's substream count against the bytes after it. Until now no test saw the price's check: dropping it (mutant A2) passed the suite. This PR adds one test that does see it. It's test-only, and no library code changes.

Why no test reached the check

The check is reachable. The sweep's cuts do stop inside the substreams with the count intact, and no shape check refuses first. The gap is in what A2 costs.

Measured on the sample TestReadBudget cuts, with window 2,000, width 1,024, depth 3, after 5,000 adds:

  • The payload is 4,600,567 bytes and holds 11 substreams.
  • Every cut from 48 to 179 bytes keeps the count of 11, with fewer than the 132 bytes it needs behind it. The check refuses them all at a price of 1,792.
  • With A2, those cuts price at 2,152 or 3,944. The sweep's bound there is at least 16,648, so the extra cost is well inside it.
  • Expire drops substreams that leave the window, so no payload this library writes has a count large enough for the bound to notice.

The test

TestADpswSubstreamCountItDoesNotHoldIsRefusedBeforeItIsPricedOrAllocated is in TestReadsAllocateOnlyWhatTheyHold.cs. It edits the substream count in the sweep's own sample, at payload offset 44 (from FORMAT.md), and reseals the checksum.

  • Claims of 383,377 and 4,294,967,295. 383,377 is one more than the bytes after the count could hold.
    • The price comes to exactly what a cut just before the count prices (1,792), and refuses at the count.
    • The read is refused for the missing substreams, not for its budget. That holds under no budget, the default budget, and a budget of exactly its own price.
    • It allocates no more than the envelope's cost plus Fixed: 4,605,256 bytes, against 4,600,728 + 16,384.
  • Control: a claim of 383,376, which the bytes can hold. It is priced past the check, at 13,040,696, and its read allocates 9,252,224 bytes. That shows the test's bound can see a substream being allocated.

Watched failing

  • A2 applied by hand: the test fails at 3 checks. At 383,377 the price came to 13,038,904 bytes more than the cut before the count.
  • The reader's own check dropped by hand: the test fails at 9 checks, after allocating 9,251,856 bytes.

Mutation: 14 mutants through scripts/mutate.sh. Each of the price's and the reader's checks was dropped, moved after its charge, set to 8 or 13 bytes a substream, set to count − 1 or count + 1, or made to refuse any count.

The table is in 75e1e63's message.

Verification

  • The branch was written on 05a5669, and master is merged in. Both sides had added tests at the same spot in TestReadsAllocateOnlyWhatTheyHold.cs, so the merge keeps both.
  • Clean Release build with -warnaserror: exit 0. Full suite: 2,503 passed on the author's base. After the merge, the coordinator ran it in a clean clone: 2,512 passed, 0 failed.
  • TestReadsAllocateOnlyWhatTheyHold also passed under the segments GC (DOTNET_GCName=libclrgc.so DOTNET_PROCESSOR_COUNT=3), as macOS runs it.
  • No CHANGELOG entry, as for earlier test-only fixes.

Left for later

  • TESTING.md could note that the cut sweep can't see a check on a count the writer keeps small, so such a check needs an edited claim.
  • A RequireHolds in another structure, whose writer's claim stays small next to the 16 KiB slack, would be just as invisible to the sweep. Not investigated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx

mattlorimor and others added 2 commits November 5, 2026 21:06
@mattlorimor @claude
Test that a DPSW substream count its bytes do not hold is refused bef…
75e1e63 
…ore it is priced (#332)

No test saw DpswSketch's price check on its count of substreams: a
mutant dropping it (A2, found in #329) passed the whole suite. Why,
measured on the sample TestReadsAllocateOnlyWhatTheyHold cuts
(TestReadBudget's: a window of 2,000, width 1,024, depth 3, after 5,000
adds; a 4,600,567-byte payload of 11 substreams of 205 items, 17
sketches each):

- The sweep's cuts do stop inside the substreams with the count intact.
  Every cut from 48 to 179 bytes holds the writer's count of 11 with
  fewer than the 132 bytes the check asks for behind it. The check
  refuses all 132 at a price of 1,792, what the shape alone costs.
- No shape check refuses first. Nothing reads or checks the count
  between its u32 and the RequireHolds.
- The claim is the writer's own, and small. With A2 those cuts price
  2,152 (cuts 48 to 59: the list of 11, 360 bytes) or 3,944 (60 to 179:
  the list and the first substream's segment list). The bound there is
  4 a byte and 16 KiB, at least 16,648. The tightest room over every
  cut short of whole was 13,232 bytes before A2 and 12,752 with it.
  Expire drops each substream that leaves the window, so a writer's
  count is the few a window spans, and no cut of a payload this library
  writes makes the claim costly enough for the bound to see. (The
  prices are measured at cuts 48, 49, 59, 60 and 179; the rooms over
  every cut.)

So the count is edited in that sample, with the checksum resealed, as
#322's per-structure tests edit theirs. The test sits beside Sublime's
in TestReadsAllocateOnlyWhatTheyHold. Measured on this commit:

- Claims of 383,377, one more than the 4,600,519 bytes after the count
  could hold at 12 each, and of 4,294,967,295: each prices exactly the
  1,792 a cut before the count prices, and stops at the count with the
  check's refusal. Each read is refused for the substreams it lacks
  under no budget, under the default, and under a budget of exactly its
  own price (4,602,520). It allocates 4,605,256 and 4,605,288 bytes:
  the envelope's cost of 4,600,728 and about 4.5 KB, held to the
  envelope's cost and Fixed.
- The control claims 383,376, which the bytes could hold. It is priced
  13,040,696, at least the shape and the list it claims. Its read gets
  past the check and is refused where the substreams run out, having
  allocated 9,252,224 bytes, past the envelope's cost and Fixed, so that
  bound does show a read that allocated a substream.

Red. A2 applied by hand fails the test at 3 checks: claiming 383,377,
the price priced 13,038,904 bytes past what a cut before the count does
and stopped at 4,600,567; claiming 4,294,967,295, it priced
68,724,127,344 past it; and that window was refused for the default
budget of 1,073,741,824. The reader's own check dropped fails it at 9:
under no budget and the default, the 383,377 claim is read into and
refused where the payload ended, having allocated 9,251,856 bytes, and
under its own price it is refused for that budget; the 4,294,967,295
claim likewise, and for the default budget as well.

Mutation, through scripts/mutate.sh in two copies of this commit's
tree, of both checks on the count, the price's and the reader's. Run A
is the new test alone; run B is the whole suite without it, 2,502
tests, to show which only this test sees.

  mutant                                 A: new test     B: the rest
  P1 (A2) price check dropped            killed          SURVIVED
  P2 price check after pricing           killed          SURVIVED
  P3 price: 8 bytes a substream          killed          SURVIVED
  P4 price: 13 bytes a substream         killed          SURVIVED
  P5 price: count - 1                    killed          SURVIVED
  P6 price: count + 1                    killed          SURVIVED
  P7 price refuses any count             killed          killed (6)
  R1 reader check dropped                killed          SURVIVED
  R2 reader check after charging         killed          SURVIVED
  R3 reader: 8 bytes a substream         killed          SURVIVED
  R4 reader: 13 bytes a substream        killed          SURVIVED
  R5 reader: count - 1                   killed          killed (3)
  R6 reader: count + 1                   killed          killed (3)
  R7 reader refuses any count            killed          killed (51)

Run A: 14 killed, 0 survived. Which check of the test failed was not
recorded per mutant. By construction, a stricter check (P4, P6, P7, R4,
R6, R7) can fail only the control, and a looser or missing one only the
claims past the bytes. Run B's survivors are what the suite missed
before this test. None is left unkilled.

There is no external anchor: the layout is this repository's own
format. The offsets are FORMAT.md's DpswSketch section, read
2026-11-05 and held to the code by TestFormatDocumentation. The test
checks them by behaviour as well: a cut before the count stops its
price there, and each edit is read back as the count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx
@mattlorimor @claude
Merge master into test/332-dpsw-substream-price-check
4fa3c2e 
Both sides added tests at the same spot in TestReadsAllocateOnlyWhatTheyHold.cs; both are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx
This was referenced Nov 6, 2026
Merged
Merged
Closed
@mattlorimor
mattlorimor merged commit ea8feb1 into master Nov 6, 2026
9 checks passed
@mattlorimor
mattlorimor deleted the test/332-dpsw-substream-price-check branch November 6, 2026 06:23
This was referenced Nov 6, 2026
Closed
Merged
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

No test sees DpswSketch's price check on its substreams (dropping it passes the suite)

1 participant

@mattlorimor