[WIP][Spec] PFOR encoding with delta support. - #617
Draft
prtkgaur wants to merge 6 commits into
Draft
Conversation
PFOR is an integer compression encoding (encoding number 11) for INT32 and INT64 columns. It compresses by subtracting the minimum value (FOR), selecting an optimal bit width via a histogram-based cost model, bit-packing the deltas, and storing outlier values as exceptions with their positions. Adds the full encoding specification in Encodings.md and the PFOR = 11 enum entry in parquet.thrift.
Remove stray colon from "Patched Frame of Reference: (PFOR = 11)" to be consistent with other headings like "Delta Encoding (DELTA_BINARY_PACKED = 5)".
A PFOR writer may now pack the differences between successive values rather than the values, decided per vector and recorded in bit 7 of the bit_width byte. A vector in that mode carries a StartValue between its info block and its packed values, so it still decodes without reading the vector before it, and the frame of reference, the bit width and the exceptions are all chosen over the differences exactly as they would be over the values. The two ordering rules a reader has to get right are stated as requirements: the exception patch runs before the prefix sum, since an exception in a delta vector holds a difference, and both the differencing and the sum run on the unsigned bit patterns. Bit 7 also means the width is masked to seven bits, not six -- an INT64 vector needing the full 64 bits would otherwise read back as a constant vector with no error. Because "delta" now names the differencing, the values that reach the packed stream are called residuals throughout the section. Adds an INT64 timestamp example with and without an exception, and the StartValue term to the size formula.
prtkgaur
commented
Sep 4, 2026
| is subtracted, whether the frame was subtracted from a value or from a | ||
| difference. | ||
|
|
||
| #### Overview |
Contributor
Author
There was a problem hiding this comment.
Will be moved to a separate file (just like ALP)
Encodings.md now carries the same short entry for PFOR that the other encodings have -- the supported types, what the encoding does including the optional delta mode, and a pointer to the details -- while the page layout, the encoding and decoding procedures, the worked examples and the constants move to PforEncoding.md. This matches how ALP is split between Encodings.md and AlpEncoding.md, and it keeps Encodings.md readable as a survey of the encodings. The moved text is unchanged apart from heading depth and one word: the paragraph defining a residual now says it holds throughout the file rather than the section. Two links into Encodings.md are written relative to it, which fixes one of them along the way: the reference to DELTA_BINARY_PACKED used the anchor #DELTA, and the anchor defined in that file is #DELTAENC.
CurtHagenlocher
added a commit
to clast-project/engineered-wood
that referenced
this pull request
Sep 5, 2026
FSST's proposal asks for encoding 10 and does not get it. ALP claimed 10 too, shipped here first, and has since been merged into parquet.thrift on parquet-format main -- so 10 is settled and not FSST's. FSST took 11 here, which is what the arrow-rs proof-of-concept predicted would happen once ALP landed. 11 is no longer free either. apache/parquet-format#617 proposes PFOR (Patched Frame of Reference) as encoding 11, and unlike the FSST proposal it arrives with two implementations behind it: apache/parquet-java#3775 and apache/arrow-rs#10977, both of which write 11. So FSST moves to 12 and 11 is reserved. The collision is not one a reader can detect and report: a decoder reads the encoding byte, believes it, and misreads the page body -- there is no magic or length that disagrees. That makes it worth vacating the slot now rather than after files exist. The number lives only on the enum member; every other site goes through Encoding.Fsst, so this is a one-line format change plus its documentation. Breaking for anyone who has persisted the numeric value, which the [Experimental] attribute on the member has always warned about. Adds EncodingWireNumberTests to pin all of the numbers, including that nothing answers to 11. Round-trip tests cannot see a renumber: this library would write and read its own files happily either way and only disagree with other implementations, which is exactly the failure mode that produced the move. Also corrects a stale README claim that FSST_16 is unimplemented and rejected; both symbol table widths have shipped since 2026-08-13. Claude-Session: https://claude.ai/code/session_01UX8ZZxcXf5q4EqqwhDntNA Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The specification said the frame of reference is the minimum of the vector, or the minimum of the differences in the delta mode, and that all residuals are therefore non-negative. That forbids what a writer should do on a tight cluster with a few low outliers, where the minimum forces a bit width wide enough to reach the far outlier and exceptions cannot help, because with the frame at the minimum only values above the packed window ever exceed it. The frame is now any value of the column's type, chosen by the writer. A value below it wraps under the modular subtraction to a residual too large for bit_width, which is the same unsigned test a value above the window fails, so it is patched like any other exception and the stored exception value is the original one. No field changes, no new mechanism, and no reader change: the frame already travels in the vector info at full width and the decode steps add it before overwriting the exception positions, so they reconstruct either choice. The bit_width == 0 text needed care in the delta mode. Such a vector steps by the frame from start_value, with patched positions breaking the step; it is only a vector with no exceptions at all that forces a frame of 0, because d[0] is 0 and nothing overwrites it. The decode fast path's justification is restated on that basis rather than on the frame being a minimum. Example 3's arithmetic was inconsistent with its own conclusion -- it framed at the sentinel and still claimed 11 bits -- and now names the cluster frame that makes 11 bits correct. Example 4 gains the page a searching writer actually produces for a constant-step column: bit_width 0 with the leading difference as its one exception, 29 bytes against the 403 a frame of 0 gets.
The short entry still described PFOR as subtracting the minimum value, which the specification body no longer requires. It now names the frame and says what the freedom buys: outliers on either side of the packed window are patched, not just the ones above it.
CurtHagenlocher
added a commit
to clast-project/engineered-wood
that referenced
this pull request
Sep 5, 2026
…239) * fix(parquet)!: move FSST off encoding 11, which PFOR now claims FSST's proposal asks for encoding 10 and does not get it. ALP claimed 10 too, shipped here first, and has since been merged into parquet.thrift on parquet-format main -- so 10 is settled and not FSST's. FSST took 11 here, which is what the arrow-rs proof-of-concept predicted would happen once ALP landed. 11 is no longer free either. apache/parquet-format#617 proposes PFOR (Patched Frame of Reference) as encoding 11, and unlike the FSST proposal it arrives with two implementations behind it: apache/parquet-java#3775 and apache/arrow-rs#10977, both of which write 11. So FSST moves to 12 and 11 is reserved. The collision is not one a reader can detect and report: a decoder reads the encoding byte, believes it, and misreads the page body -- there is no magic or length that disagrees. That makes it worth vacating the slot now rather than after files exist. The number lives only on the enum member; every other site goes through Encoding.Fsst, so this is a one-line format change plus its documentation. Breaking for anyone who has persisted the numeric value, which the [Experimental] attribute on the member has always warned about. Adds EncodingWireNumberTests to pin all of the numbers, including that nothing answers to 11. Round-trip tests cannot see a renumber: this library would write and read its own files happily either way and only disagree with other implementations, which is exactly the failure mode that produced the move. Also corrects a stale README claim that FSST_16 is unimplemented and rejected; both symbol table widths have shipped since 2026-08-13. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UX8ZZxcXf5q4EqqwhDntNA * feat(parquet): PFOR encoding, plain and delta, behind EWPARQUET0005 Implements PFOR (Patched Frame of Reference) for INT32 and INT64 as proposed in apache/parquet-format#617, reader and writer, both modes. Frame of reference plus bit-packing, with the values that do not fit the chosen width stored separately as exceptions -- so one outlier stops widening the packing for everyone. Each 1024-value vector independently chooses whether to pack values or the differences between them, which is the difference from DELTA_BINARY_PACKED: a column sorted only in stretches gets the delta treatment on those stretches and frame-of-reference on the rest. The writer measures the result and falls back to PLAIN per page, so enabling the setting cannot make a file bigger. The page layout is close enough to ALP's that PforDecoder is shaped like AlpDecoder and the two read together: a 7-byte header, an offset array whose offsets are measured from its own start, then self-describing vectors. What PFOR adds is the delta flag in bit 7 of the width byte, a per-vector start value, and a prefix sum -- which must run AFTER the exceptions are patched, since an exception in a delta vector is a difference like any other. TWO PLACES WHERE THE SPEC CONTRADICTS ITSELF, both found by measuring. First, the frame. PforEncoding.md says it is the column's minimum. On the shape PFOR exists for -- a tight cluster with a low sentinel -- that is the wrong answer by a wide margin: the sentinel takes the frame, every ordinary value sits tens of thousands above it, and the width is set by the gap rather than the cluster. The spec's own Example 3 is that column and quotes a width only a frame ABOVE the minimum can produce. So the minimum is a candidate here, not the rule: the encoder buckets the range, slides a window over the bucket counts per candidate width, lowers the winner onto the smallest value it covers, and costs that frame exactly. The minimum is always among the candidates and the only one costed from a real histogram, so the search cannot do worse than the naive rule. arrow-rs does the same; parquet-java does not. MEASURED, on 200k INT64 date keys with a null sentinel: 0.65x against DELTA_BINARY_PACKED with the naive frame, 5.31x with the search. Second, width 0 in the delta mode. The spec says a reader may fill with the start value and "get the same answer for any frame". It does not: fill and the general path agree only when the frame is 0. No conforming writer can emit anything else, so it is unreachable from real data -- but arrow-rs takes the fill path and its own test pins an answer the general path disagrees with. This decodes by the general path, which is what the numbered decode steps say, and pins that. Compression, 200k INT64 per shape, uncompressed, vs DELTA_BINARY_PACKED: date keys + sentinel 5.31x, sorted in stretches 3.11x, tight cluster 1.74x, sorted ids 1.20x, timestamps 1.20x, sequence with gaps 1.06x, random 1.03x. With zstd the picture reverses everywhere except the outlier shapes, because DBP's output on clean sequential data is far more compressible than PFOR's tightly packed output. doc/parquet-pfor.md carries both tables and says so plainly rather than quoting only the flattering half. Tests sweep EVERY bit width, 0 to 32 and 0 to 64, at a vector length that is a multiple of eight and one that is not, rather than sampling: this is the class of bug that shipped once already in the RLE decoder (#236), and it is invisible at every width but the broken one. Three byte-for-byte golden vectors from arrow-rs's tests are the only assertions that could catch us writing a self-consistent page nobody else can read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UX8ZZxcXf5q4EqqwhDntNA * fix(parquet): strip the UTF-8 BOMs the PFOR commit introduced Three files were rewritten through a utf-8-sig encoder, which prepends a BOM. CI's "Check for UTF-8 BOMs" step is exactly there to catch it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UX8ZZxcXf5q4EqqwhDntNA * fix(parquet): bound a PFOR vector by the next offset, and reuse encoder scratch Both from Copilot's review of #239. A vector was sliced from its own offset to the END OF THE PAGE rather than to the next vector's offset. Every per-vector truncation check then measured against the wrong extent: a vector shorter than its own header claims read on into the following vectors' bytes, satisfied every length check, and returned their contents as data. Non-monotonic offsets were not checked at all, so a decreasing offset silently decoded overlapping vectors. Neither is a memory-safety problem -- the reads stay inside the page -- but both are silent. The value count comes from the page header, so the output is the right shape and the wrong data, with nothing raised to say so. The two new tests fail on the old code with "No exception was thrown", which is the whole point. The first attempt at the truncation test did not discriminate: shifting the next offset backwards also breaks the FOLLOWING vector, which threw for an unrelated reason and made the test pass against the bug. It now widens the first vector's declared bit width instead, leaving every offset alone, and asserts the precondition that makes it a regression test -- short measured against the next offset, long enough measured against the end of the page. Also makes the encoder's per-page scratch thread-static. Encoding a page allocated differences, residuals, exception positions, a histogram and the frame-search buckets every time, about eleven kilobytes of garbage per page of every integer column. Thread-static rather than pooled because column chunks encode in parallel and each thread encodes one page at a time, which is why ColumnChunkWriter.t_valuesBuffer is thread-static too. Not taken: the reviewer also suggested writing vectors into one contiguous page buffer instead of a byte[] per vector. That is a real cost, but the byte[]-per-vector shape is AlpEncoder's as well, and changing it in one encoder leaves the two inconsistent. Worth doing across both with a benchmark behind it, not blind in this PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UX8ZZxcXf5q4EqqwhDntNA * fix(parquet): drop the "11 is reserved" comment now that PFOR occupies 11 Merge fallout. #238 left a comment above the FSST member reserving 11 for PFOR; this branch replaced it with the member itself, and merging main back in reinstated the comment alongside the member it describes. "Reserved rather than reused" now sits directly above Pfor = 11. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UX8ZZxcXf5q4EqqwhDntNA --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: CurtHagenlocher <904803+CurtHagenlocher@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
What changes are included in this PR?
Do these changes have PoC implementations?