cl/persistence, cl/antiquary: store builder pending payments history as epoch base plus deltas by domiwei · Pull Request #25234 · erigontech/erigon

domiwei · 2026-10-22T09:35:49Z

Stacked on #25231 (→ #25230 → #25212): review only the last commit.

Problem

The state antiquary writes the full builder_pending_payments vector for every processed Gloas block, compressed with zstd: 2 × SLOTS_PER_EPOCH entries of 52 bytes, about 2.3 KB per row with realistic data. On mainnet that is about 73.7 KB per epoch, or about 6 GB a year. Most blocks change only a few entries: one bid insertion, one parent payment settlement, some attestation weights. The format has to change before Gloas reaches Hoodi and mainnet.

Fix

kv.BuilderPendingPaymentsTable stores one base per epoch plus a delta per block. It is the same table and the same snapshot type. The other per-slot tables (ExecutionPayloadAvailability, PtcWindow, LatestExecutionPayloadBid) are unchanged.

Rows are raw and little-endian, and use the same 14-byte preamble as the builders rows from #25231:

00 00 00 00 | 00 00 00 00 | "GLOA" | 01 (version) | kind
kind 02, base:  the full vector SSZ, exactly 2*SLOTS_PER_EPOCH*52 bytes
kind 03, delta: count u16, then count × (index u16, 52-byte BuilderPendingPayment SSZ)

The preamble code and ErrHistoryRebuildRequired move to a shared history_row.go. The builders decoder keeps its behaviour and error messages.
Writer (PendingPaymentsWriter): it keeps the pre-block vector as SSZ bytes and compares entries by value.
- It writes a base on the first row of a new writer, when the epoch changes, and when there is no baseline. A new collector is created on every IncrementBeaconState call and after every commit batch, so a restart or backoff also starts with a base.
- Otherwise it writes a delta with every changed index, in increasing order. The delta has count 0 when nothing changed.
- The epoch rotation in process_builder_pending_payments happens in the first block of an epoch, so it always lands in a base.
- A vector longer than math.MaxUint16 is rejected, because delta indices are uint16.
Decoder (ApplyPendingPaymentsRow): strict. It checks the preamble, version, kind, the exact length, and that indices are strictly increasing and in range. It decodes into temporary storage, so the output stays unchanged on any error. Legacy zstd rows fail at the preamble.
Reader (ReadBuilderPendingPayments): it scans back from the target slot to the epoch start and stops at the latest base, then applies the base and the deltas in slot order.
- A processed block (one with kv.SlotData) that has no row returns ErrHistoryRebuildRequired.
- A missed slot is skipped.
- No base in the epoch returns ErrHistoryRebuildRequired.

Size

Per row, from BenchmarkPendingPaymentsRowSize:

Vector	Legacy zstd	Base	Typical delta	Empty delta
dense (64 realistic entries)	2302	3342	286 (1 bid, 1 settlement, 3 weights)	16
empty	15	3342	–	16
one entry	62	3342	–	16

Per mainnet epoch with a dense vector: about 12.2 KB, against 73.7 KB before.
When the vector is almost empty, for example on a network where most proposers build their own blocks (bid value 0 records no payment), the new format costs more: about 3.8 KB per epoch against 0.5 KB, at most about 0.75 MB a day.
Base rows are not compressed, because Caplin state snapshots store words uncompressed.

Upgrade note

The same notes as #25231 apply; that PR has the release note. An old-format row in the DB makes the antiquary back off and re-antiquate. Rows frozen in Caplin state snapshot files by older releases (Sepolia) are not repaired. An older release fails on new rows instead of reading wrong data.

Tests

Codec:
- exact byte layout of a base and a delta;
- writer lifecycle: first row is a base, then a delta, a base on epoch change, a base without a baseline, an empty delta;
- entries are compared by value after an in-place mutation;
- table-driven mutation round trip: bid insertion, settlement, weight updates, epoch rotation, proposer slashing, combined changes;
- 21 rejected rows, each with no mutation;
- the vector length limit.
Reader:
- base at the target; base plus deltas; a missed slot; a processed-block hole; a malformed later delta; a delta without a base;
- a base only in the previous epoch; the latest base after a restart;
- exact fork slot, missed fork slot, missed epoch start, fully missed previous epoch;
- the old readCompressedSSZ path fails on new rows without changing its output.
Antiquary:
- the collector driven through preStateTransitionHook across the Fulu → Gloas fork and a mid-epoch collector restart, with the row kinds checked, flushed to mdbx and read back against the vectors that were written;
- the seam test with real snapshot files and pruning, reading epochs that cross the 1000-slot file boundary and the 2000 snapshot/DB boundary.

TDD: the codec, reader and collector tests were written first and failed on missing symbols or wrong row kinds. The vector length test failed before its guard was added. Reverting the reader or the collector to zstd makes the reconstruction tests fail.

Validation

go test ./cl/persistence/... ./cl/antiquary/... -count=1
make lint, two runs, 0 issues
make erigon integration
Dual review (Claude + Codex) until converged: two review rounds, fix-delta reviews and a fresh adversarial pass.

Copilot

🔵 Needs a closer look

The incompatible persisted-history format, unchanged snapshot version, and frozen-history upgrade limitations require final human validation.

No open issues 🎉

What changed in this PR

This PR reduces Caplin’s Gloas builder pending payments history storage by replacing full compressed vectors with epoch bases and per-block deltas. It extends the preceding builders-history work while keeping the existing table and snapshot type.

Changes:

Adds the base/delta codec and shares history-row validation with builders diffs.
Connects the antiquary writer and historical reader, including restart bases and missing-row detection.
Adds codec, compatibility, restart, and snapshot-boundary tests, plus row-size benchmarks.

File	Description
`cl/persistence/state/pending_payments.go`	Adds base/delta writing and strict decoding.
`cl/persistence/state/pending_payments_test.go`	Tests layout, lifecycle, mutations, and invalid rows.
`cl/persistence/state/pending_payments_bench_test.go`	Compares base and delta sizes with legacy compression.
`cl/persistence/state/history_row.go`	Centralizes the preamble and rebuild-required error.
`cl/persistence/state/historical_states_reader/pending_payments_reader_test.go`	Tests reconstruction, missing rows, and legacy-reader rejection.
`cl/persistence/state/historical_states_reader/historical_states_reader.go`	Reconstructs payments from the latest base and subsequent deltas.
`cl/persistence/state/historical_states_reader/gloas_roundtrip_test.go`	Updates round-trip fixtures to the new format.
`cl/persistence/state/builders_diff.go`	Reuses shared history-row helpers.
`cl/antiquary/state_prune_reader_test.go`	Tests payment replay across snapshot and database boundaries.
`cl/antiquary/state_antiquary.go`	Propagates pre-transition initialization errors.
`cl/antiquary/beacon_states_collector.go`	Integrates the pending payments writer into collection.
`cl/antiquary/beacon_states_collector_test.go`	Tests collection across fork initialization and collector restarts.

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The on-disk format change leaves legacy frozen snapshots unreadable, so release compatibility and recovery implications require final human validation.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The incompatible persisted-history format and unchanged legacy frozen snapshots require human validation of upgrade compatibility before release.

No open issues 🎉

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The persistent consensus-history format change leaves legacy frozen snapshots unreadable, requiring human sign-off on upgrade compatibility.

No open issues 🎉

_{Review effort: Balanced}

lystopad

LGTM. Reviewed the last commit (caba93ce10); the description matches the change.

The delta chain is sound: the hook initializes from the previous processed block's post-state, a base is written on every new epoch (so the rotation never lands in a delta), on collector restart and at the fork, and deltas compare by SSZ bytes, so they don't rely on copy-on-write. The decoder is strict and leaves the output unchanged on error. cl/persistence/... and cl/antiquary/... pass with -race.

Non-blocking, for later if sparse networks matter: a near-empty vector now costs ~3.3 KB per base vs ~15 bytes zstd before; a sparse-base row kind (non-zero indices only, same layout as a delta) would make that case about as cheap as an empty delta.

Stack note: needs #25230 and #25231 in the same release.

…lap (#25230) Fixes #25205 Stacked on #25212: review only the last four commits. The last commit updates one test of this PR for the `ReadRequiredQueueSSZ` signature from #25212. This PR must ship in the same release as #25231 (builders per-index diff) and #25234 (builder pending payments base plus deltas), which are stacked on it. Without them, a Gloas node writes `builders` rows that rewrite the whole list on every in-place change. ## Problem The state antiquary stores per-slot diffs of the Electra queues (`pending_deposits`, `pending_consolidations`, `pending_partial_withdrawals`) and the Gloas lists (`builders`, `builder_pending_withdrawals`, `payload_expected_withdrawals`) with `SSZQueueEncoder`. The encoder remembered only the previous size and the previous last element. It searched the new list backwards for that element and wrote "truncate N from the front" plus the elements after it. Two cases broke this: - `builders` changes in place (withdrawals, deposits, exits). When a builder that is not the last one changed, the diff was "truncate 0, append nothing", so the change was lost until the next dump. - When an appended entry equalled the previous last entry, the search picked the appended copy, the truncate count went negative and wrapped in `uint32`, and replay emptied the queue. ## Fix The diff byte format and the replay format are unchanged. - `Initialize` keeps a copy of the previous list's element pointers in a slice the encoder owns. - `WriteDiff` first tries a fast path based on pointer identity: queue elements are copy-on-write, so equal pointers hold equal values. If the previous list's tail and the new list's head hold the same pointers, it writes the diff directly. This covers unchanged queues, append-only slots and front pops plus appends. - Otherwise it finds the longest suffix of the previous list that equals a prefix of the new list with KMP, which is linear in the list sizes, and writes `truncate = prevLen - overlap` plus `new[overlap:]`. An in-place builder change becomes a full rewrite of the list for that slot. - `ApplySSZQueueDiff` returns the new `ErrCorruptQueueDiff` when the truncate count is larger than the queue length, before it changes the queue. The readers wrap it, and the antiquary resume backs off on it like it does for `ErrMissingGloasData`. ## Upgrade note There is no migration: the stored bytes of a wrong diff cannot be repaired, and each wrong window ends at the next dump (at most 1535 slots). Re-antiquation does not repair rows already frozen in Caplin state snapshot files, and nothing regenerates a frozen range automatically. On Sepolia the Gloas range written by older releases stays wrong. Electra ranges are affected only in the rare case where two equal queue entries were appended in a row. ## Tests - Collector tests: an in-place change to builder 0 is kept, and an appended duplicate builder pending withdrawal no longer empties the queue. - Encoder tests: in-place `Set` on the same list, duplicate last append, front cut plus postponed re-append, the fast path and its fallback when equal values have different pointers, `truncate == len` accepted and `truncate > len` rejected without changing the queue, an exhaustive round trip over all small lists with duplicates, and a check that the overlap search makes a linear number of equality calls. - Reader and antiquary tests: the corrupt-diff error is wrapped by both readers, and the antiquary backoff includes it. - Benchmark in `ssz_queue_bench_test.go`: at 100k `pending_deposits` entries, a diff takes about 0.24 ms. TDD: the tests were written first and failed on the old code for the intended reason, except the `truncate == len` check and the linear-call check, which also pass on the old encoder and guard the new one. The fast-path test failed before the fast path was added. ## Validation - `go test ./cl/persistence/... ./cl/antiquary/... -count=1` - `make lint`, two runs, 0 issues - `make erigon integration` ## Follow-ups (not in this PR) - `builders` gets its own per-index row format in #25231, so an in-place builder change no longer rewrites the whole list. - No end-to-end test drives the antiquary backoff through a stored corrupt row. - Copy-on-write of queue elements is a convention, not enforced by the type.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…as epoch base plus deltas

domiwei · 2026-10-22T21:20:37Z

Closing. The premise of this PR was wrong.

After each Gloas block, builder_pending_payments holds about one non-default entry. The block first settles its parent's payment and clears that entry (ApplyParentExecutionPayload), and only then records its own bid. Only payments for withheld payloads stay in the vector and gain weight. The legacy per-slot row is the full vector compressed with zstd, so in normal operation it is about 63 B.

This PR writes a 3342 B base per epoch plus 124 B deltas (each block changes two entries), which is about 3.6x larger. Per mainnet epoch, from a simulation with the real writers:

Withheld payloads	Legacy	This PR
0%	2010 B	7186 B
5%	5105 B	7316 B
10%	7933 B	7481 B
100%	49260 B	10534 B

The new format only wins when about 10% or more of payloads are withheld for a long time, and the legacy worst case is about 4 GB a year. The "6 GB a year" in the description came from a benchmark with all 64 entries filled, which the state transition cannot produce. The legacy format stays as it is.

…iff (erigontech#25231) Stacked on erigontech#25230 (which is stacked on erigontech#25212): review only the last two commits. ## Problem The state antiquary writes one `kv.Builders` row per processed Gloas block with `SSZQueueEncoder`. Builders is a registry, not a queue: deposits, withdrawals and exits change builders in place. With the overlap encoder from erigontech#25205, any change to a builder that is not the last one rewrites the whole list for that slot, about 97 bytes per builder. At 200 builders that is about 19 KB per block, roughly 46–51 GB a year. The format has to change before Gloas reaches Hoodi and mainnet, because after that old rows would need a migration. ## Fix `kv.Builders` gets a per-index diff format. Same table, same snapshot type. `kv.BuildersDump` and the other five queues are unchanged. Row layout, little-endian, not compressed: ``` 00 00 00 00 | 00 00 00 00 | 47 4c 4f 41 ("GLOA") | 01 (version) | 01 (kind) newLength u64 | count u32 | count × (index u64, 93-byte Builder SSZ) ``` - The total length must be exactly `26 + 101*count`. A block with no builder change writes a 26-byte row. With one change at 200 builders a row is 127 bytes, against 19404 with the queue encoder. - The writer, `BuildersDiffWriter`, keeps the pre-block builders as copied values. It writes every changed index plus the appended indices, in increasing order. The first Gloas row after the fork (empty baseline) carries the full list as appends. A shrink is an error. - The decoder, `ApplyBuildersDiff`, checks the whole row before it changes the list. It rejects: - a wrong preamble, version or kind; - a wrong exact length; - a length above the registry limit, or a shrink; - indices that are not strictly increasing, or not below the new length; - appends that are not contiguous; - a builder that does not decode. - `ReadBuildersSSZ` replays the rows on top of the dump, using the same dump rules as `ReadRequiredQueueSSZ` (erigontech#25212). A processed block (one with `kv.SlotData`) at or after the fork slot that has no row is an error. A missed slot is skipped. The Gloas genesis state writes a no-change row at its own slot, so it can be read when its slot is not a dump slot. - Every rejection wraps the new `ErrHistoryRebuildRequired`, and the antiquary backs off on it like it does for `ErrMissingGloasData`. The first 8 zero bytes are for old binaries. An older reader takes them as "truncate 0", then a zero-length builder, and fails decoding the empty builder before it changes the list. It does not mistake the row for a corrupt queue diff, so it does not back off and rewrite new rows in the old format. ## Upgrade note (for the release notes) Hoodi operators must upgrade before the Gloas fork on 2026-10-26. This release changes the on-disk format of Caplin historical state rows for builders (per-index diff) and builder pending payments (epoch base plus per-block deltas, erigontech#25234). - The Caplin state snapshot types keep their file version. - A node that reads an old-format row from its DB backs off and re-antiquates from before the Gloas fork. - An older release that reads a new-format row fails with a decode error and does not read wrong data. Downgrading after new rows were written is not supported. - erigontech#25205 must not ship without this format change. - Sepolia is not repaired. Rows already frozen in Caplin state snapshot files, including the published Sepolia range 11250000–11300000, stay in the old format. Snapshot files are read before the DB and frozen ranges are not regenerated, so historical state reads that replay those rows keep failing until the range is reset or republished. - The same applies to a node that passed a Gloas fork on an older release and has already frozen Gloas state snapshots. ## Tests - **Codec:** - round trip over: no change, first, middle or last change, several changes, index reuse, append, reuse plus append, empty to one; - a change to each single non-balance field; - exact byte layout; - the baseline is a value copy; - shrink is an error; - about 20 rejected rows, each with no mutation; - both legacy queue decoders, before and after erigontech#25205, fail on a new row without changing the list. - **Reader:** exact fork slot, missed fork slot, dump plus change plus append, a processed-block hole at the target and between the dump and the target, a missed slot, a pre-fork slot, and a legacy row (sentinel, with table and slot in the error). - **Antiquary:** - backoff on the sentinel; - collector → mdbx → reader round trip; - the hook across the Fulu → Gloas fork; - a Gloas genesis state at a dump slot (0) and at a slot that is not a dump slot (32): the genesis row is the 26-byte no-change row, and both the genesis state and the next block's change read back; - a seam test with real 1000-slot snapshot files, pruning, and reads that cross the file boundary and the snapshot/DB boundary, compared with the lists actually written. - **Benchmark:** `BenchmarkBuildersRowSize`. At 200 builders with one change per block, Initialize plus WriteDiff takes about 1.7 µs, 7 allocs and 127 bytes per row. TDD: the codec, reader and backoff tests were written first and failed for the intended reason, either on undefined symbols or with the backoff predicate returning false. The genesis test also failed first: at slot 32 with `historical state row needs a rebuild: missing row from table Builders at slot 32`, and at slot 0 because the genesis slot had no row (an empty value where a 26-byte row was expected). Some later tests, the append-end check, the hole after a dump, and the single-field changes, guard existing behaviour and passed on first run. Each one was checked to fail when its target branch is disabled. The snapshot file version is deliberately not bumped, so no Caplin snapshot republish is needed. The raw preamble tells old and new rows apart. ## Validation - `go test ./cl/persistence/... ./cl/antiquary/... ./db/snapshotsync/... -count=1`, and `-race` on `cl/antiquary` - `make lint`, two runs, 0 issues - `make erigon integration` - Dual review until converged: two review rounds, a fix-delta review and a fresh adversarial pass. The genesis-row commit had its own dual review, with no blocker. ## Follow-ups (not in this PR) - Gloas required-queue history is unreadable when every slot from the fork through the next dump slot is missed: no Gloas dump is written at that slot. This is pre-existing and affects all three Gloas queues. - `LatestExecutionPayloadBidTable` stores a redundant full bid per slot. It can be changed after the Hoodi fork.

domiwei requested a review from sudeepdino008 as a code owner October 22, 2026 09:35

Copilot AI balanced review requested due to automatic review settings October 22, 2026 09:35

domiwei added Glamsterdam https://eips.ethereum.org/EIPS/eip-7773 Caplin Caplin: Consensus Layer, Beacon API type:bug labels Oct 22, 2026

domiwei self-assigned this Oct 22, 2026

domiwei requested review from lystopad and yperbasis October 22, 2026 09:35

This was referenced Oct 22, 2026

cl/persistence, cl/antiquary: store builders history as a per-index diff #25231

Merged

cl/persistence, cl/antiquary: write queue diffs from the longest overlap #25230

Merged

Copilot started reviewing on behalf of domiwei October 22, 2026 09:36

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 22, 2026 11:04

domiwei force-pushed the kewei/pending-payments-delta branch from 76e1ca8 to 94b6b49 Compare October 22, 2026 11:04

Copilot started reviewing on behalf of domiwei October 22, 2026 11:07

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 22, 2026 11:36

domiwei force-pushed the kewei/pending-payments-delta branch from 94b6b49 to 77f4903 Compare October 22, 2026 11:36

Copilot started reviewing on behalf of domiwei October 22, 2026 11:36

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

domiwei mentioned this pull request Oct 22, 2026

cl/persistence: read Gloas queues before the first Gloas dump #25212

Merged

domiwei force-pushed the kewei/builders-index-diff branch from cfa8786 to 58ace75 Compare October 22, 2026 14:00

domiwei requested a review from AskAlexSharov as a code owner October 22, 2026 14:00

Copilot AI balanced review requested due to automatic review settings October 22, 2026 14:00

domiwei force-pushed the kewei/pending-payments-delta branch from 77f4903 to caba93c Compare October 22, 2026 14:00

Copilot AI reviewed Oct 22, 2026

View reviewed changes

Copilot started reviewing on behalf of domiwei October 22, 2026 14:04

View session

lystopad approved these changes Oct 22, 2026

View reviewed changes

domiwei force-pushed the kewei/builders-index-diff branch from 58ace75 to 9616363 Compare October 22, 2026 17:31

Copilot AI balanced review requested due to automatic review settings October 22, 2026 17:32

domiwei force-pushed the kewei/pending-payments-delta branch from caba93c to 9c0946c Compare October 22, 2026 17:32

Copilot AI reviewed Oct 22, 2026

awskii approved these changes Oct 22, 2026

View reviewed changes

Base automatically changed from kewei/builders-index-diff to main October 22, 2026 20:08

cl/persistence, cl/antiquary: store builder pending payments history …

1dc0712

…as epoch base plus deltas

domiwei force-pushed the kewei/pending-payments-delta branch from 9c0946c to 1dc0712 Compare October 22, 2026 20:20

domiwei enabled auto-merge October 22, 2026 20:21

domiwei disabled auto-merge October 22, 2026 20:44

domiwei closed this Oct 22, 2026

Navigation Menu

Repository navigation

cl/persistence, cl/antiquary: store builder pending payments history as epoch base plus deltas - #25234

cl/persistence, cl/antiquary: store builder pending payments history as epoch base plus deltas#25234
domiwei wants to merge 1 commit into
mainfrom
kewei/pending-payments-delta

domiwei commented Oct 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

lystopad left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

domiwei commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

domiwei commented Oct 22, 2026

Problem

Fix

Size

Upgrade note

Tests

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

lystopad left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

domiwei commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants