Repository navigation
cl/persistence: read Gloas queues before the first Gloas dump - #25212
Merged
Merged
Conversation
When the Gloas fork slot is not a multiple of SlotsPerDump, the dump slot before the fork is pre-Gloas and the antiquary writes no builders, builder_pending_withdrawals or payload_expected_withdrawals dump for it. ReadRequiredQueueSSZ then returned ErrMissingGloasData for every state from the fork slot up to the next dump slot. Pass the Gloas fork slot to ReadRequiredQueueSSZ. When the dump is missing and the dump slot is before the fork slot, start from an empty list and apply the per-slot diffs. This is safe because the antiquary does not initialize the Gloas queue writers before the first Gloas pre-state, so the diff at the first Gloas slot holds the full list. A missing dump at or after the fork slot still returns ErrMissingGloasData. Fixes #25200
When the Gloas fork slot is a multiple of SlotsPerDump and its block is missed, the antiquary keeps the pre-Gloas state, so it writes no Gloas queue dump at that slot. The first Gloas block then writes a diff that holds the full list, because the queue encoders were never initialized. ReadRequiredQueueSSZ returned ErrMissingGloasData in this case, so historical states for the rest of that dump interval could not be read. Return the error only when the missing dump slot is after the fork slot.
A missing dump at or before the fork slot is read as an empty list, because the antiquary writes no Gloas dump for a pre-Gloas state and the first Gloas diff holds the full list. In that case the replay starts at the dump slot instead of the slot after it. For data the antiquary writes this gives the same result, because a pre-Gloas dump slot has no Gloas diff. When the dump at an aligned fork slot is lost while its full-list diff is present, starting at the dump slot keeps that diff.
domiwei
added
Glamsterdam
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The focused reader change preserves post-fork missing-data checks and has regression coverage for the affected replay boundaries.
No open issues 🎉
What changed in this PR
Fixes historical Gloas state reads between fork activation and the first Gloas dump, addressing #25200 without changing the writer or persistence format.
Changes:
- Treats missing queue dumps at or before the fork slot as empty bases and replays diffs from the dump slot.
- Preserves errors for missing post-fork dumps and adds regression coverage for fork boundaries and historical-state reconstruction.
| File | Description |
|---|---|
cl/persistence/state/historical_states_reader/historical_states_reader.go |
Passes the fork slot into queue reads and adjusts missing-dump handling and replay boundaries. |
cl/persistence/state/historical_states_reader/gloas_roundtrip_test.go |
Adds fork-boundary regression tests and extracts a shared historical-state fixture. |
Review effort: Balanced
This was referenced Oct 22, 2026
yperbasis
approved these changes
Oct 22, 2026
yperbasis
left a comment
yperbasis
left a comment
Member
There was a problem hiding this comment.
Two optional notes:
ReadRequiredQueueSSZ: when the dump slot is before the fork slot, the replay loop walks the pre-fork slots, which cannot hold Gloas rows. Starting at the fork slot instead (firstDiffSlot = forkSlot) is equivalent and states the invariant directly.- The fix relies on the first Gloas diff being a full list: a fresh
SSZQueueEncoderwrites the whole list andApplySSZQueueDiffrebuilds it on an empty list.ssz_queue_test.goonly covers an initialized encoder. A small round-trip test for the fresh encoder would protect this reader path when the encoder changes for #25205.
Member
Author
|
Thanks for the review! Both notes make sense, and neither changes behaviour, so I'll leave this PR as it is. The PRs stacked on it (#25230, #25231, #25234) move this logic into a shared |
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Oct 22, 2026
…lap (#25230) Fixes #25205 Stacked on #25212: review only the last four commits. The last commit updates one test of this PR for the `ReadRequiredQueueSSZ` signature from #25212. This PR must ship in the same release as #25231 (builders per-index diff) and #25234 (builder pending payments base plus deltas), which are stacked on it. Without them, a Gloas node writes `builders` rows that rewrite the whole list on every in-place change. ## Problem The state antiquary stores per-slot diffs of the Electra queues (`pending_deposits`, `pending_consolidations`, `pending_partial_withdrawals`) and the Gloas lists (`builders`, `builder_pending_withdrawals`, `payload_expected_withdrawals`) with `SSZQueueEncoder`. The encoder remembered only the previous size and the previous last element. It searched the new list backwards for that element and wrote "truncate N from the front" plus the elements after it. Two cases broke this: - `builders` changes in place (withdrawals, deposits, exits). When a builder that is not the last one changed, the diff was "truncate 0, append nothing", so the change was lost until the next dump. - When an appended entry equalled the previous last entry, the search picked the appended copy, the truncate count went negative and wrapped in `uint32`, and replay emptied the queue. ## Fix The diff byte format and the replay format are unchanged. - `Initialize` keeps a copy of the previous list's element pointers in a slice the encoder owns. - `WriteDiff` first tries a fast path based on pointer identity: queue elements are copy-on-write, so equal pointers hold equal values. If the previous list's tail and the new list's head hold the same pointers, it writes the diff directly. This covers unchanged queues, append-only slots and front pops plus appends. - Otherwise it finds the longest suffix of the previous list that equals a prefix of the new list with KMP, which is linear in the list sizes, and writes `truncate = prevLen - overlap` plus `new[overlap:]`. An in-place builder change becomes a full rewrite of the list for that slot. - `ApplySSZQueueDiff` returns the new `ErrCorruptQueueDiff` when the truncate count is larger than the queue length, before it changes the queue. The readers wrap it, and the antiquary resume backs off on it like it does for `ErrMissingGloasData`. ## Upgrade note There is no migration: the stored bytes of a wrong diff cannot be repaired, and each wrong window ends at the next dump (at most 1535 slots). Re-antiquation does not repair rows already frozen in Caplin state snapshot files, and nothing regenerates a frozen range automatically. On Sepolia the Gloas range written by older releases stays wrong. Electra ranges are affected only in the rare case where two equal queue entries were appended in a row. ## Tests - Collector tests: an in-place change to builder 0 is kept, and an appended duplicate builder pending withdrawal no longer empties the queue. - Encoder tests: in-place `Set` on the same list, duplicate last append, front cut plus postponed re-append, the fast path and its fallback when equal values have different pointers, `truncate == len` accepted and `truncate > len` rejected without changing the queue, an exhaustive round trip over all small lists with duplicates, and a check that the overlap search makes a linear number of equality calls. - Reader and antiquary tests: the corrupt-diff error is wrapped by both readers, and the antiquary backoff includes it. - Benchmark in `ssz_queue_bench_test.go`: at 100k `pending_deposits` entries, a diff takes about 0.24 ms. TDD: the tests were written first and failed on the old code for the intended reason, except the `truncate == len` check and the linear-call check, which also pass on the old encoder and guard the new one. The fast-path test failed before the fast path was added. ## Validation - `go test ./cl/persistence/... ./cl/antiquary/... -count=1` - `make lint`, two runs, 0 issues - `make erigon integration` ## Follow-ups (not in this PR) - `builders` gets its own per-index row format in #25231, so an in-place builder change no longer rewrites the whole list. - No end-to-end test drives the antiquary backoff through a stored corrupt row. - Copy-on-write of queue elements is a convention, not enforced by the type.
rayjun
pushed a commit
to rayjun/erigon
that referenced
this pull request
Oct 23, 2026
…iff (erigontech#25231) Stacked on erigontech#25230 (which is stacked on erigontech#25212): review only the last two commits. ## Problem The state antiquary writes one `kv.Builders` row per processed Gloas block with `SSZQueueEncoder`. Builders is a registry, not a queue: deposits, withdrawals and exits change builders in place. With the overlap encoder from erigontech#25205, any change to a builder that is not the last one rewrites the whole list for that slot, about 97 bytes per builder. At 200 builders that is about 19 KB per block, roughly 46–51 GB a year. The format has to change before Gloas reaches Hoodi and mainnet, because after that old rows would need a migration. ## Fix `kv.Builders` gets a per-index diff format. Same table, same snapshot type. `kv.BuildersDump` and the other five queues are unchanged. Row layout, little-endian, not compressed: ``` 00 00 00 00 | 00 00 00 00 | 47 4c 4f 41 ("GLOA") | 01 (version) | 01 (kind) newLength u64 | count u32 | count × (index u64, 93-byte Builder SSZ) ``` - The total length must be exactly `26 + 101*count`. A block with no builder change writes a 26-byte row. With one change at 200 builders a row is 127 bytes, against 19404 with the queue encoder. - The writer, `BuildersDiffWriter`, keeps the pre-block builders as copied values. It writes every changed index plus the appended indices, in increasing order. The first Gloas row after the fork (empty baseline) carries the full list as appends. A shrink is an error. - The decoder, `ApplyBuildersDiff`, checks the whole row before it changes the list. It rejects: - a wrong preamble, version or kind; - a wrong exact length; - a length above the registry limit, or a shrink; - indices that are not strictly increasing, or not below the new length; - appends that are not contiguous; - a builder that does not decode. - `ReadBuildersSSZ` replays the rows on top of the dump, using the same dump rules as `ReadRequiredQueueSSZ` (erigontech#25212). A processed block (one with `kv.SlotData`) at or after the fork slot that has no row is an error. A missed slot is skipped. The Gloas genesis state writes a no-change row at its own slot, so it can be read when its slot is not a dump slot. - Every rejection wraps the new `ErrHistoryRebuildRequired`, and the antiquary backs off on it like it does for `ErrMissingGloasData`. The first 8 zero bytes are for old binaries. An older reader takes them as "truncate 0", then a zero-length builder, and fails decoding the empty builder before it changes the list. It does not mistake the row for a corrupt queue diff, so it does not back off and rewrite new rows in the old format. ## Upgrade note (for the release notes) Hoodi operators must upgrade before the Gloas fork on 2026-10-26. This release changes the on-disk format of Caplin historical state rows for builders (per-index diff) and builder pending payments (epoch base plus per-block deltas, erigontech#25234). - The Caplin state snapshot types keep their file version. - A node that reads an old-format row from its DB backs off and re-antiquates from before the Gloas fork. - An older release that reads a new-format row fails with a decode error and does not read wrong data. Downgrading after new rows were written is not supported. - erigontech#25205 must not ship without this format change. - Sepolia is not repaired. Rows already frozen in Caplin state snapshot files, including the published Sepolia range 11250000–11300000, stay in the old format. Snapshot files are read before the DB and frozen ranges are not regenerated, so historical state reads that replay those rows keep failing until the range is reset or republished. - The same applies to a node that passed a Gloas fork on an older release and has already frozen Gloas state snapshots. ## Tests - **Codec:** - round trip over: no change, first, middle or last change, several changes, index reuse, append, reuse plus append, empty to one; - a change to each single non-balance field; - exact byte layout; - the baseline is a value copy; - shrink is an error; - about 20 rejected rows, each with no mutation; - both legacy queue decoders, before and after erigontech#25205, fail on a new row without changing the list. - **Reader:** exact fork slot, missed fork slot, dump plus change plus append, a processed-block hole at the target and between the dump and the target, a missed slot, a pre-fork slot, and a legacy row (sentinel, with table and slot in the error). - **Antiquary:** - backoff on the sentinel; - collector → mdbx → reader round trip; - the hook across the Fulu → Gloas fork; - a Gloas genesis state at a dump slot (0) and at a slot that is not a dump slot (32): the genesis row is the 26-byte no-change row, and both the genesis state and the next block's change read back; - a seam test with real 1000-slot snapshot files, pruning, and reads that cross the file boundary and the snapshot/DB boundary, compared with the lists actually written. - **Benchmark:** `BenchmarkBuildersRowSize`. At 200 builders with one change per block, Initialize plus WriteDiff takes about 1.7 µs, 7 allocs and 127 bytes per row. TDD: the codec, reader and backoff tests were written first and failed for the intended reason, either on undefined symbols or with the backoff predicate returning false. The genesis test also failed first: at slot 32 with `historical state row needs a rebuild: missing row from table Builders at slot 32`, and at slot 0 because the genesis slot had no row (an empty value where a 26-byte row was expected). Some later tests, the append-end check, the hole after a dump, and the single-field changes, guard existing behaviour and passed on first run. Each one was checked to fail when its target branch is disabled. The snapshot file version is deliberately not bumped, so no Caplin snapshot republish is needed. The raw preamble tells old and new rows apart. ## Validation - `go test ./cl/persistence/... ./cl/antiquary/... ./db/snapshotsync/... -count=1`, and `-race` on `cl/antiquary` - `make lint`, two runs, 0 issues - `make erigon integration` - Dual review until converged: two review rounds, a fix-delta review and a fresh adversarial pass. The genesis-row commit had its own dual review, with no blocker. ## Follow-ups (not in this PR) - Gloas required-queue history is unreadable when every slot from the fork through the next dump slot is missed: no Gloas dump is written at that slot. This is pre-existing and affects all three Gloas queues. - `LatestExecutionPayloadBidTable` stores a redundant full bid per slot. It can be changed after the Hoodi fork.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #25200
Problem
ReadHistoricalStatereadsbuilders,builder_pending_withdrawalsandpayload_expected_withdrawalsfrom the dump atslot - slot % SlotsPerDumpplus the per-slot diffs.ReadRequiredQueueSSZreturnedErrMissingGloasDatawhenever that dump was missing. The antiquary writes these dumps only for a Gloas state, so every state between the Gloas fork slot and the first Gloas dump slot could not be read. On Sepolia these are slots 11296768–11297279. The same happens for an aligned fork slot whose block is missed.Fix
The change is reader-only. There is no antiquary change and no persistence-format change.
ReadHistoricalStatepasses the Gloas fork slot toReadRequiredQueueSSZ.ErrMissingGloasData.Tests
TestReadRequiredQueueSSZ_MissedForkSlotBlock: aligned fork slot with a missed block. There is no dump, and the full diff is at the next slot.TestReadRequiredQueueSSZ_MissingDumpAtForkSlotWithBlock: the dump at an aligned fork slot is missing and the full diff is at the fork slot.TestReadRequiredQueueSSZ_MissingDump: adds a dump slot right after an unaligned fork, which still returns an error.TestReadHistoricalState_GloasQueuesBeforeFirstDump: end-to-end read with fork epoch 49, which is not a multiple of 48. The fork slot is 1568 and the pre-fork dump slot is 1536.TestReadHistoricalState_GloasQueuesMissingDumpAfterFork: end-to-end reverse case. The dump slot is after the fork and the dumps are missing, so the read returnsErrMissingGloasData.TDD: each new test was written first and failed on the previous code for the intended reason:
MissedForkSlotBlock,MissingDumpAtForkSlotWithBlockandGloasQueuesBeforeFirstDumpfailed withErrMissingGloasDataon the origin/main reader.MissingDumpAtForkSlotWithBlockreturned[b]instead of[a, b]before the replay-start fix.MissingDumpcase expects an error, so it also passes on origin/main. It guards the relaxed check against admitting a dump slot after the fork.ReadHistoricalStatetests fail under the mutationsgloasForkSlot := r.cfg.GloasForkEpochandgloasForkSlot := ^uint64(0). Both mutations passed the suite before these tests existed.Validation
go test ./cl/persistence/state/historical_states_reader/ ./cl/antiquary/ -count=1make lint, two runs, 0 issuesmake erigon integrationFollow-ups (not in this PR)
ErrMissingGloasDatauntil the following dump. This needs 33+ missed slots; on Sepolia it needs 513.GloasForkEpoch * SlotsPerEpochoverflows for a far-future fork epoch. It is reachable only when the reader's config differs from the config that wrote the DB, for example in capcli.