[mattlorimor]

Fixes #347.

Matt's decision on #347 was "Check held lengths". An envelope's claimed length is now checked against the bytes actually held, before anything is priced or allocated. A short payload with an inflated header is refused as malformed (InvalidDataException) under any budget, as C16's rule asks. Before, a held 128,827-byte TopK payload claiming 2^30 bytes allocated about 1.07 GB with no budget, and under the default budget it was refused for its budget.

The change

All four points follow the coordinator's plan on #347.

1. Top level. PersistenceFormat.Read checks an ordinary envelope's length against what a seekable stream has left after the header, with the checksum, before EnvelopeCost is priced or charged. A byte array is read through a MemoryStream, which can seek.
2. Nested. For a held payload, PriceNested checks a nested envelope's length against its run before pricing the envelope. The nested read reads from a MemoryStream over the run, so (1) holds it too, and the price and the read stop in the same place.
3. One helper, RequireHeld, makes both checks. The large form's existing check (ReadLarge) now calls it too, with its message unchanged.
4. A stream that can't seek is still held by the budget, since its length isn't known. The comments in Read and PriceNested say so. The comment that called the old behaviour intended now says it of that stream. FORMAT.md's limits and read-budget sections say so too.

The check refuses when fewer bytes than the checksum's four follow the header, before it compares what's left with the claim. That keeps the unsigned subtraction from wrapping.

Tests

New file TestHeldEnvelopeLengths.cs, in #340's and #345's edited-claim pattern. One field of a payload the library wrote is edited, and the checksums are resealed, so the length is the only lie.

• TestEveryHeldPayloadClaimingMoreThanItHoldsIsRefusedBeforeItIsPriced:
  • covers every StructureId, in the ordinary envelope, as a byte array and from a seekable stream;
  • claims are one past what the bytes hold, and Array.MaxLength;
  • each is refused for the bytes it lacks under no budget, the default and its own price, having allocated no more than a claim refused at the header, plus 1 KiB;
  • controls: the length the bytes hold reads, including from a stream with 64 more bytes after the envelope.
• TestEveryLargeFormPayloadClaimingMoreThanItsStreamHoldsIsRefusedBeforeItIsPriced: the same in the large form, which already had this check (green on arrival).
• TestAPayloadFromAStreamThatCannotSeekIsHeldToItsBudgetInstead: from a stream that can't seek, Array.MaxLength is refused for the default budget.
• TestANestedEnvelopeClaimingMoreThanItsRunHoldsIsRefusedBeforeItIsPriced: its roster is every structure whose sample holds a nested envelope, checked against the structures whose Price calls PriceNested in IL (TopK and ScalableBloomFilter).
• TestEveryPayloadCutShortOfItsChecksumIsRefusedBeforeItIsPriced: each payload cut to 0 and to 3 bytes after the header, which reaches the checksum half of the check. A boundary control, an empty payload with its 4 checksum bytes, must not be refused as short.

Red, on 6e2e5ec plus the test file:

• Ordinary envelope: all 33 structures fail. A claim of L + 1 was read into and refused as "The stream ended after 3 of the 4 bytes the checksum needs". TopK allocated 130,736 bytes doing it. Array.MaxLength was refused for its budget, asking 2,147,603,592 bytes for BloomFilter.
• Nested: TopK's L + 1 claim was priced 370,360 bytes past a claim refused at the header, and its read allocated 348,592. Array.MaxLength was priced 2,147,483,752 bytes past it before Slice refused it.
• Cut short of the checksum: all 33 fail in the ordinary envelope.

Changed: TestPayloadWriterGrowth's TestALengthOfExactlyTheLargestArrayIsRefusedForItsCostNotItsSize pinned the old budget refusal for a held payload. It's there to show that CheckHeader lets exactly the largest array through. It now reads that payload from a stream that can't seek, where the length still reaches the default budget, and held, where it gets past its size and is refused as too few bytes.

Mutation of RequireHeld, against TestHeldEnvelopeLengths alone: 6 mutants, all killed, by 33 to 101 tests each. The boundary control is what kills held < 4 made <=; before it, that mutant survived this class. The table is in 52b5698's message.

Merging with #345

#345's TestANestedEnvelopeClaimingMoreThanItsRunIsRefusedAsData expects Slice's message for a nested claim of one byte past its run. With this change, PriceNested refuses earlier, with the held-check message. The claim is still refused as data under every budget, and only the expected message changes. The two merge cleanly as text. The author found this by reading the code, and has the one-line change ready for when master has #345.

Verification

• Clean Release build with -warnaserror: exit 0.
• Full suite on 52b5698: 2,625 passed, 0 failed, run by the author and again by the coordinator in a clean clone.
• Trial merges with Hold every holds-check to an edited claim or the test that kills its drop (#344) #345 and QuotientFilter read leftovers, and a byte left over refused as malformed under its own price (#343) #348 are clean as text; see above for Hold every holds-check to an edited claim or the test that kills its drop (#344) #345.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx