[domiwei]

Stacked on #25230 (which is stacked on #25212): review only the last two commits.

Problem

The state antiquary writes one kv.Builders row per processed Gloas block with SSZQueueEncoder. Builders is a registry, not a queue: deposits, withdrawals and exits change builders in place. With the overlap encoder from #25205, any change to a builder that is not the last one rewrites the whole list for that slot, about 97 bytes per builder. At 200 builders that is about 19 KB per block, roughly 46–51 GB a year. The format has to change before Gloas reaches Hoodi and mainnet, because after that old rows would need a migration.

Fix

kv.Builders gets a per-index diff format. Same table, same snapshot type. kv.BuildersDump and the other five queues are unchanged.

Row layout, little-endian, not compressed:

00 00 00 00 | 00 00 00 00 | 47 4c 4f 41 ("GLOA") | 01 (version) | 01 (kind)
newLength u64 | count u32 | count × (index u64, 93-byte Builder SSZ)

• The total length must be exactly 26 + 101*count. A block with no builder change writes a 26-byte row. With one change at 200 builders a row is 127 bytes, against 19404 with the queue encoder.
• The writer, BuildersDiffWriter, keeps the pre-block builders as copied values. It writes every changed index plus the appended indices, in increasing order. The first Gloas row after the fork (empty baseline) carries the full list as appends. A shrink is an error.
• The decoder, ApplyBuildersDiff, checks the whole row before it changes the list. It rejects:
  • a wrong preamble, version or kind;
  • a wrong exact length;
  • a length above the registry limit, or a shrink;
  • indices that are not strictly increasing, or not below the new length;
  • appends that are not contiguous;
  • a builder that does not decode.
• ReadBuildersSSZ replays the rows on top of the dump, using the same dump rules as ReadRequiredQueueSSZ (cl/persistence: read Gloas queues before the first Gloas dump #25212). A processed block (one with kv.SlotData) at or after the fork slot that has no row is an error. A missed slot is skipped. The Gloas genesis state writes a no-change row at its own slot, so it can be read when its slot is not a dump slot.
• Every rejection wraps the new ErrHistoryRebuildRequired, and the antiquary backs off on it like it does for ErrMissingGloasData.

The first 8 zero bytes are for old binaries. An older reader takes them as "truncate 0", then a zero-length builder, and fails decoding the empty builder before it changes the list. It does not mistake the row for a corrupt queue diff, so it does not back off and rewrite new rows in the old format.

Upgrade note (for the release notes)

Hoodi operators must upgrade before the Gloas fork on 2026-10-26. This release changes the on-disk format of Caplin historical state rows for builders (per-index diff).

• The Caplin state snapshot types keep their file version.
• A node that reads an old-format row from its DB backs off and re-antiquates from before the Gloas fork.
• An older release that reads a new-format row fails with a decode error and does not read wrong data. Downgrading after new rows were written is not supported.
• cl/antiquary: Gloas builder diffs lose changes in historical states #25205 must not ship without this format change.
• Sepolia is not repaired. Rows already frozen in Caplin state snapshot files, including the published Sepolia range 11250000–11300000, stay in the old format. Snapshot files are read before the DB and frozen ranges are not regenerated, so historical state reads that replay those rows keep failing until the range is reset or republished.
• The same applies to a node that passed a Gloas fork on an older release and has already frozen Gloas state snapshots.

Tests

• Codec:
  • round trip over: no change, first, middle or last change, several changes, index reuse, append, reuse plus append, empty to one;
  • a change to each single non-balance field;
  • exact byte layout;
  • the baseline is a value copy;
  • shrink is an error;
  • about 20 rejected rows, each with no mutation;
  • both legacy queue decoders, before and after cl/antiquary: Gloas builder diffs lose changes in historical states #25205, fail on a new row without changing the list.
• Reader: exact fork slot, missed fork slot, dump plus change plus append, a processed-block hole at the target and between the dump and the target, a missed slot, a pre-fork slot, and a legacy row (sentinel, with table and slot in the error).
• Antiquary:
  • backoff on the sentinel;
  • collector → mdbx → reader round trip;
  • the hook across the Fulu → Gloas fork;
  • a Gloas genesis state at a dump slot (0) and at a slot that is not a dump slot (32): the genesis row is the 26-byte no-change row, and both the genesis state and the next block's change read back;
  • a seam test with real 1000-slot snapshot files, pruning, and reads that cross the file boundary and the snapshot/DB boundary, compared with the lists actually written.
• Benchmark: BenchmarkBuildersRowSize. At 200 builders with one change per block, Initialize plus WriteDiff takes about 1.7 µs, 7 allocs and 127 bytes per row.

TDD: the codec, reader and backoff tests were written first and failed for the intended reason, either on undefined symbols or with the backoff predicate returning false. The genesis test also failed first: at slot 32 with historical state row needs a rebuild: missing row from table Builders at slot 32, and at slot 0 because the genesis slot had no row (an empty value where a 26-byte row was expected). Some later tests, the append-end check, the hole after a dump, and the single-field changes, guard existing behaviour and passed on first run. Each one was checked to fail when its target branch is disabled.

The snapshot file version is deliberately not bumped, so no Caplin snapshot republish is needed. The raw preamble tells old and new rows apart.

Validation

• go test ./cl/persistence/... ./cl/antiquary/... ./db/snapshotsync/... -count=1, and -race on cl/antiquary
• make lint, two runs, 0 issues
• make erigon integration
• Dual review until converged: two review rounds, a fix-delta review and a fresh adversarial pass. The genesis-row commit had its own dual review, with no blocker.

Follow-ups (not in this PR)

• Gloas required-queue history is unreadable when every slot from the fork through the next dump slot is missed: no Gloas dump is written at that slot. This is pre-existing and affects all three Gloas queues.
• LatestExecutionPayloadBidTable stores a redundant full bid per slot. It can be changed after the Hoodi fork.

[Copilot]

🟡 Changes recommended

Dump-aligned restarts can bypass legacy-row detection and leave unfrozen database history unreadable after upgrading.

1 open issue

• Detect legacy DB history before accepting Gloas resume point · New

What changed in this PR

This PR replaces Caplin’s builders queue history with per-index diffs, reducing storage for in-place registry changes while retaining full dumps and the existing snapshot type.

Changes:

• Adds a versioned builders diff codec with validation before mutation.
• Integrates builders replay, missing-row detection, and rebuild-error backoff.
• Adds codec, reader, collector, snapshot-boundary tests, and a size benchmark.

File	Description
db/​kv/​tables.go	Updates the builders table’s format comment.
cl/​persistence/​state/​historical_states_reader/​historical_states_reader.go	Adds builders-specific replay and shared dump loading.
cl/​persistence/​state/​historical_states_reader/​gloas_roundtrip_test.go	Adapts reconstruction tests to the new format.
cl/​persistence/​state/​historical_states_reader/​builders_reader_test.go	Tests replay, missing rows, and legacy-row rejection.
cl/​persistence/​state/​builders_diff.go	Implements per-index encoding and validated application.
cl/​persistence/​state/​builders_diff_test.go	Tests layout, updates, rejection, and decoder compatibility.
cl/​persistence/​state/​builders_diff_bench_test.go	Compares per-index and queue encoding costs.
cl/​antiquary/​state_prune_reader_test.go	Tests replay across snapshot and database boundaries.
cl/​antiquary/​state_antiquary.go	Adds rebuild-required errors to backoff handling.
cl/​antiquary/​state_antiquary_test.go	Tests the new backoff condition.
cl/​antiquary/​beacon_states_collector.go	Switches builders collection to the new writer.
cl/​antiquary/​beacon_states_collector_test.go	Tests collector round trips and fork-transition handling.

_{Review effort: Balanced}

[domiwei]

Good catch, fixed. ReadBuildersSSZ now starts replay at the dump slot. It checks the format of the kv.Builders row there but does not apply it, because the dump already includes that row. A legacy row at a dump-aligned resume slot now returns ErrHistoryRebuildRequired, so the antiquary backs off further instead of resuming after the dump. A processed dump slot without a row is also rejected, the same as any other slot. A missed dump slot is still accepted. The regression test is TestReadBuildersSSZChecksTheRowAtTheDumpSlot. Commits: 3f47f1b and cfa8786.

[Copilot]

🟡 Changes recommended

Dump-aligned recovery can bypass legacy-format detection and leave earlier unfrozen builders history unreadable after upgrading.

2 open issues

• Prevent dump-aligned resumes from bypassing legacy detection · New
• Detect legacy DB history before accepting Gloas resume point

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The incompatible on-disk format, unsupported downgrades, and unrepaired legacy frozen snapshots require final human compatibility review.

1 open issue

• Detect legacy DB history before accepting Gloas resume point

1 resolved since last review

• Prevent dump-aligned resumes from bypassing legacy detection

_{Review effort: Balanced}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[lystopad]

LGTM. Reviewed head 58ace75f13 (the four commits on top of #25230).

The codec compares builders by value and validates everything (exact length, registry limit, no shrink, strictly increasing indices, contiguous appends) before it changes the list. The dump-aligned resume case Copilot flagged is fixed: ReadBuildersSSZ now format-checks the row at the dump slot, and the resume slot always has a canonical block and therefore a row. readRequiredDump returns the boundary or boundary+1, so no diff row is skipped. cl/persistence/... and cl/antiquary/... pass with -race.

Nits/notes:

• The description still says "review only the last two commits"; there are four now, and the SHAs in the reply to Copilot are pre-rebase (4b8e56f514, 58ace75f13 now).
• Release planning: the stack (#25230, #25231, #25234) should ship together in one release that reaches Hoodi operators before Gloas activates there, so no Hoodi node writes old-format rows.
• The published Sepolia state snapshots for 11250000–11300000 stay unreadable; worth a tracked follow-up to republish that range.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[domiwei]

Correction to the upgrade note: #25234 is closed, so builder pending payments keep the legacy zstd row format. In this release only the builders rows change format. I updated the note in the description.