Remove an idle conversation's landed files, and keep its structure in skeletons

[wu-sheng]

Step 4 of retention. A storage root no longer only grows.

What it does

storage.retention, -1 and off by default, removes the landed files of a conversation that was sent and has had nothing new land for that long, such as 72h or 30d. asz collect and asz server do it at the end of a pass, after the send.

A conversation's files go together, and only when all of these hold:

• It has an owner in push.state: it was sent, or a send was tried.
• Nothing landed for it within the retention.
• Its head round covers its newest file, and its parse did not fail this pass.
• With metrics derived, every file of it is derived.
• It holds no provider bodies. Exact provider cutting after a removal comes in step 5.

Sending is not waited for. Retention never runs over a scenario root.

Skeletons

Each removed file is replaced by <name>.sk: Session Data, compressed with gzip. It keeps every record and part in its place, with every identifier the index reads, and no content.

• A changes record keeps its join, its time, and each file's path, operation, sides and line counts, with no diff.
• An execution record keeps its join, its time and its outcome.
• A provider body keeps its manifest without segments, and every piece's digest.

The header's new removed object records:

• the file's digest, size and record count;
• when, and by which removal, it was removed;
• what the receiver confirmed of it, and the owner.

The index rebuilt from skeletons is the index rebuilt from the landed files, byte for byte, so a conversation that becomes active again continues the same chain. A skeleton is never sent.

Measured on 2026-10-13 on one root: 7,235 landed files of 847.5 MB had skeletons of 29.6 MB, 3.5%.

Nothing lost

1. removing.json records the removal: its generation, the newest sequence it removes, and its phase.
2. Each skeleton is written and synced, and the error of its first sync is checked. It takes its file's modification time and is synced again. A failed sync stops the removal, and the next pass writes every skeleton again.
3. Each landed file is deleted only once its skeleton reads whole and records the file's digest. Then each metrics request the receiver has goes; a request not yet sent goes on the first pass after it is. A receipt stays, so a deriver whose state a power loss took back derives the file as before. Every directory of the removed range is synced.
4. removing.json is deleted.

Every pass finishes an unfinished removal before anything lands. No collector lands into a session while its removal is unfinished: it finds the session busy.

Readers

Reader	What it does with a skeleton
index, parse	reads it as the landed file; binds a round to the digest it recorded
metrics	keys and digests by the landed path; tokens from a skeleton, no MCP call
export	never sends it
LangSmith	continues its next position from it
asz verify	checks lines and bytes through it; reports removed files apart, never as verified
page	marks the file and its steps removed; follows a removal mid-read; keeps what it read for the chain head and the removal generation in removal.state, so a separate asz view sees a removal
asz show	says the record was removed
asz repack	refuses a session with skeletons

Also in this PR: the first commit takes out the owner adoption for push.state files an older build wrote. Upgrades from older builds are not supported.

Deferred to later steps:

• Step 5: provider body sessions.
• Step 6: the page's banner.
• Step 7: local mode, for never-sent sessions.
• Step 8: sent_state_retention and the retired marker.

Tests

Each was proven by reverting its fix.

• tests/chain/retention_test.go
  • All 34 scenarios run twice, once removing everything after each send. They must give the same sent files, rounds and metrics, chains, indexes (also rebuilt from skeletons only), page file lists, changes and executions, and verify cleanly.
  • Each reason a session is kept.
  • Receiver evidence and generations.
  • Metrics cleanup: a request sent later goes, and receipts stay.
  • A crash at every point of a removal.
  • A failed first sync, and a failed sync after the deletions.
  • Nothing lands during a removal.
  • Empty and wrong skeletons.
  • Metrics derived from skeletons.
  • An untimed file stamped after a removal.
  • The page following a removal, from another process too, and keeping nothing while one is in progress.
• Unit tests:
  • pkg/skeleton: the format.
  • internal/storage: listing, digests, the counter, the intent, the removal generation, the fallback open, the busy lock.
  • internal/verify: removed files reported, not verified.
  • internal/view: whole-record steps, removed bodies not served.
  • internal/config: parsing and overflow.
  • internal/adapters/langsmith: the next position.
  • cmd/asz: the pass line, scenario roots left alone, repack refusing, and show and verify.