Repository navigation
QuotientFilter reads its table straight into storage, holding it once (#328) - #337
Merged
Merged
Conversation
TestAReadHoldsItsTableOnce reads a filter of 2^16 slots at 1% (81,936 bytes of words, holding 30,000 items) and holds what the read allocates, counted in a no-GC region, and what it is charged, to its payload and one copy of its table plus 4 KiB: in the ordinary envelope, and in the large form, where the payload is not held, to the table alone. Each is held from below too, to the payload and the table, so that a measurement of nothing cannot pass. The table is longer than a heap piece of a run (64 KiB), so the large form's buffer for its pieces is a whole piece, past the slack. Red on 05a5669, at the first check: the read allocated 246,248 bytes, 82,332 more than its payload and its table That is PayloadReader.BeginWords' copy of the run (81,960 bytes with its header) and 372 bytes besides. The test prints every figure before it asserts any, and on 05a5669 they are: payload 81,956, table 81,960 ordinary read 246,248, 82,332 past payload + table ordinary charge 246,104, 82,188 past payload + table large-form read 147,968, 66,008 past the table large-form charge 147,824, 65,864 past the table each past the 4 KiB slack: the ordinary read by the copy, the large form by its 64 KiB piece buffer (65,560 bytes with its header). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx
…ces (red) (#328) TestAQuotientFiltersTableIsReadFromTheLargeFormInPiecesStraightIntoItsTable holds the table as TestAnInfiniFiltersTableIsReadFromTheLargeFormInPieces- StraightIntoItsSegment holds an Infini filter's (#313): the budget sample's table (2^17 slots at 0.1%, 213,008 bytes), found as the bytes its words are written as, with its run's u64 length just before them, is read in pieces of the shipped 16 KiB through the stream's span overload, one after another from its first byte to its last, with nothing among them, and in pieces of eight words with the piece lowered. Then the run, claimed a byte past one array at its prefix, is refused for the bytes the stream lacks and not as more than one array holds. Red on 05a5669, at the first piece: the table's 213,008 bytes at 46, with the piece at 16,384, were read in 4 reads, the first of 65,536 bytes through the array overload, and not in 14 pieces through the span overload That is PayloadReader.BeginWords' buffer of a heap piece (64 KiB). The last check passes on 05a5669, where BeginWords already holds a streamed run to what remains; it is there for the fix, whose new way of beginning the run has to keep that. The fix will take the quotient filter out of TestARunItsStructureKeepsInStorageOfItsOwnIsReadInPiecesIntoIt, which counts array reads, and this test holds it instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx
The reader began the table's run with PayloadReader.BeginWords, which from a held payload copies the run out whole (ReadBytes), and then unpacked the copy into the table it built, so a read held, and was charged for, the table twice besides the payload: three times in all. From the large form, which is not held, BeginWords read the run through a buffer of a heap piece (64 KiB, or the run where shorter). The run is now begun with a new PayloadReader.BeginWordsInPlace, which reads its length after the checks BeginWords makes of it (held to one array where the read holds the payload, to what remains where it holds none) and charges nothing, and the words are read straight into the table with a new PayloadReader.ReadWords(Span<ulong>) overload, in pieces through a stack buffer of BlockPieceFor (16 KiB) with ReadBlock, as #289 does for CountMinSketch and #313 and #230 do for the Infini and cuckoo filters. The overload is the body of InfiniFilter.ReadWords and CuckooBloomFilter.ReadWords, which #329 replaces with it. The table is still charged (TableCost) before it is allocated, after the run's length is held to the payload (#322). The price skips the run with BeginWordsInPlace and prices TableCost alone, as the reader charges it, so price and charge still come to one sum (TestReadBudget holds them equal for every sample). Measured on the red test's filter (2^16 slots at 1%, a table of 81,936 bytes, 81,960 with its header), in a no-GC region: 05a5669 this ordinary read 246,248 164,288 ordinary charge 246,104 164,144 large-form read 147,968 82,408 large-form charge 147,824 82,264 What is left past the payload and one table is the 372, 228, 448 and 304 bytes that were past the copy, or the piece, before. Documents moved by it, through the capacity checks: - README: the default budget now reads 0.1% up to a table of 2^28 slots, 201,326,592 items (was 100,663,296); 1% stays at 2^28. - CAPACITY.md: "A read is charged three times the table" is now twice, in the payload and the table its words are read into, with two claims: both charge lines' slopes are 16 bytes a word (were 24), and the default budget reads up to 2^28 slots at 1% and at 0.1%, from the fitted lines. - FORMAT.md: QuotientFilter's row of read charges says its words are read straight into the table. TestARunItsStructureKeepsInStorageOfItsOwnIsReadInPiecesIntoIt counts the array reads a structure makes of a run in the large form, and the quotient filter now makes none of its table: it is taken out of the structures #228 names there, with a check that it reads no run through the array overload, and the red test before this one holds its pieces. Mutation, scripts/mutate.sh, MUTATION_JOBS=2, at this commit's tree (f7b2f0c, before this table was added to its message), whole suite unless a test is named in brackets: begin: always held to what remains killed (1) begin: always held to one array killed (1) begin: branches swapped killed (2) begin: length held to nothing killed (9) words: pieces of one word killed (1) words: buffer not cut to the run SURVIVED (all 2,504) words: last piece read whole killed (18) words: last piece not read killed (44) words: last word of a piece dropped killed (16) words: read big-endian killed (42) reader: the old read (copy), price now killed (7) reader+price: both as at 05a5669 killed (5) price: as at 05a5669 (SkipWords) killed (5) reader: copy charged too killed (5) reader: table not charged killed (2) reader: table not read killed (44) [holds once] both as at 05a5669 killed (1) [holds once] copy charged too killed (1) [pieces] both as at 05a5669 killed (1) [pieces] pieces of one word killed (1) [pieces] begin: always held to one array killed (1) "begin" is BeginWordsInPlace's one condition, this.streamed ? ReadHeldRunLength() : BeginRun(): dropped to either arm, the arms swapped, and both arms replaced by a bare read of the length, held to nothing. Held to what remains everywhere, a held payload's run past one array is refused as the payload ending rather than as more than one array holds, which one test sees; held to one array everywhere, a large-form run past one array is refused as if it were read into one, which the pieces test's last check alone sees ([pieces]). Held to nothing, a payload cut inside the table is charged, and has the table allocated, before its words are found missing (#322). "words" is the ReadWords(Span<ulong>) overload. "both as at 05a5669" puts back the old reader and price whole, BeginWords and its copy and SkipWords, and each red test alone kills it. "the old read, price now" is the old reader under the new price. The survivor is the one #289 and #313 adjudicated for this loop, and equivalent in all but stack use: the read loop cuts every piece to Math.Min(left, piece.Length), and with left never above the run's length that is Math.Min(left, BlockPieceBytes) whether the buffer was cut to the run or not, so the same bytes are read in the same calls. Nothing a test can see measures stack. The cut is kept so a small table takes a small buffer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #328.
QuotientFilter's reader no longer copies its table out of the payload before copying it into the filter. It now reads the words straight into the table, in 16 KiB pieces. A read is charged for one copy of the table, not two. This follows the coordinator's plan on #328, approved by Matt.The change
PayloadReadermembers:BeginWordsInPlace()reads the run's length withBeginWords' own checks, and charges nothing. It isthis.streamed ? ReadHeldRunLength() : BeginRun(), so the "single array" refusals stay where they were.ReadWords(Span<ulong>)reads the words straight into the table, in 16 KiB stack pieces viaReadBlock.Measured in
NoGCRegion, on 2^16 slots at 1% (an 81,936-byte table):Docs, through
CapacityClaims:Tests
Each is red on master.
TestAReadHoldsItsTableOnce: "the read allocated 246,248 bytes, 82,332 more than its payload and its table".TestAQuotientFiltersTableIsReadFromTheLargeFormInPiecesStraightIntoItsTable: "4 reads, the first of 65,536 bytes through the array overload, and not in 14 pieces through the span overload".TestARunItsStructureKeepsInStorageOfItsOwnIsReadInPiecesIntoIt: counts reads through the array overload, and QF now makes none. So QF comes off that test's list, with a guard, and the pieces test above holds it instead.Mutation: 20 killed, 1 survivor. The table is in
a950eba's message.Verification
-warnaserror: exit 0.ReadWordsoverload, byte for byte. A trial merge both ways was clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01KK3MspqAcczxT45xCfLgyx