From e4d0ad20c5c7b9b7f778416c2f3e81fec3017fa6 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 27 Oct 2026 06:22:19 -0700 Subject: [PATCH 1/9] Docs: ADR-0010 one storage format (#198) Commit Keep to one supported storage format. Direct initialization replaces version-1 initialization plus migration, complete stores become portable while live authority checks remain, and future format changes ship a separate upgrade tool. Cross-reference the superseded root identity records and the amended ADR-0009 migration section. --- CHANGELOG.md | 2 + ...retention-roots-release-and-gc-liveness.md | 2 + docs/adr/0010-one-storage-format.md | 260 ++++++++++++++++++ docs/adr/README.md | 1 + docs/adr/retention-reader-root-identity.md | 4 +- docs/formats/segment-store-v2/recovery.md | 4 +- 6 files changed, 271 insertions(+), 2 deletions(-) create mode 100644 docs/adr/0010-one-storage-format.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 08e02f4b..39c16b3e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ after its public API and format compatibility policies are established. ## [Unreleased] +- ADR-0010 commits Keep to one storage format: direct initialization replaces version-1 initialization plus migration, complete stores become portable while live authority checks remain, and future format changes ship a separate upgrade tool. Main's behavior is unchanged until the units tracked in #198 land (#198). + - Update the repository-only serde_json parser to 1.0.154 in both locked workspaces and refresh its dependency admission; the resolved transitive graph, features, license and MSRV are unchanged (#186). - Update the in-process BLAKE3 dependency to 1.8.7 in both locked workspaces, remove the unused arrayref license exception and renew its admission while retaining the external b3sum oracle and version-one identity contract (#94). diff --git a/docs/adr/0009-retention-roots-release-and-gc-liveness.md b/docs/adr/0009-retention-roots-release-and-gc-liveness.md index b8db887d..50e40fa6 100644 --- a/docs/adr/0009-retention-roots-release-and-gc-liveness.md +++ b/docs/adr/0009-retention-roots-release-and-gc-liveness.md @@ -6,6 +6,8 @@ - Related issue: [#18](https://github.com/flyingrobots/keep/issues/18) - Depends on: ADR-0002, ADR-0005, issues #3, #16, and #17 +- Amended by: [ADR-0010](0010-one-storage-format.md), which replaces migration + as the only entry into the format with direct initialization ## Context diff --git a/docs/adr/0010-one-storage-format.md b/docs/adr/0010-one-storage-format.md new file mode 100644 index 00000000..81357f13 --- /dev/null +++ b/docs/adr/0010-one-storage-format.md @@ -0,0 +1,260 @@ +# ADR-0010: One Storage Format + +- Status: Accepted +- Date: 2026-10-27 +- Owners: Keep format, initialization, publication, recovery, and retention + boundaries +- Related issue: + [#198](https://github.com/flyingrobots/keep/issues/198) +- Depends on: ADR-0005, ADR-0009 +- Amends: ADR-0009 §Durable namespace compatibility (migration as the only + entry into version 2) +- Supersedes: the restart root-identity requirement in + [Retention reader root identity](retention-reader-root-identity.md) and + [Root identity across restart](../formats/segment-store-v2/recovery.md#root-identity-across-restart) + +## Context + +Main supports two durable formats. `keep.segment-store/v1` admits the writer +lock, staging, the segment and catalog pools, and `HEAD`. +`keep.segment-store/v2` keeps those bytes unchanged and adds the `FORMAT` +marker, `reader.lock`, retention state, and reserved GC and +recovery-disposition namespaces. A one-way migration connects the two. + +The two formats split Keep's capabilities between them: + +- A version-2 store can be created only by migrating a version-1 store. + Version-2 initialization is undefined. +- The content writer, `FilesystemCatalogPublisher`, and segment and catalog + stage recovery admit only the version-1 namespace. + `FilesystemVersionTwoAdmission` deliberately cannot be passed to them, so a + migrated store can be read and can publish retention roots, but cannot + ingest. General version-2 catalog publication waits on the retained-closure + gate in [#125](https://github.com/flyingrobots/keep/issues/125). +- Every version-2 test, fixture, and crash case first builds a version-1 + store and migrates it. The migration accounts for about 10,500 lines under + `src/adapters/store_migration`, 2,000 lines of integration tests, 1,100 + lines of repository tasks, two golden records and their source table, and +the + `KEEP-CRASH-053..073` process-death matrix. + +Nothing has been released. The crate is `0.0.0` with `publish = false`. On +2026-10-27 the owner stated that no Keep store with data of value exists +anywhere. Both formats can therefore be consolidated now without a +conversion. + +The migration also introduced a persistent binding between a store and its +directory. The migration intent records the root directory's device and +inode, and every later reopen and fenced snapshot compares the live root +against them (`require_root_identity`). The binding proves one thing: the +migration records were not transplanted from another directory. It proves +less than the "is this the store I meant to open" guarantee it has sometimes +been read as: + +- Rename store A away and rename an intact store B into A's path. B's records + match B's directory, so admission passes, even though the caller wanted A. +- Restore an older, internally consistent snapshot into the original + directory. The inode is unchanged, so admission passes. The binding is not + rollback protection. +- Restore a backup, copy a store to a new disk, or let device-mapper renumber + a volume. Every byte is intact, yet admission refuses with + `RootIdentityChanged`, and version 2 defines no re-admission. + +## Decision + +### One supported format + +Keep admits, writes, and recovers exactly one storage format. The codebase +never carries a second runtime format alongside it. The format is the +version-2 record set and namespace without migration artifacts: + +- the segment, catalog, and publication-head grammars, byte-for-byte as + ADR-0005 and the version-1 pages define them; +- `FORMAT`, `reader.lock`, retention records, GC records, and + recovery-disposition records, as the version-2 pages define them; +- no `migration.intent`, `migration.receipt`, their `.next` stages, or + `FORMAT.next`. + +Its documentation is consolidated into one normative set of pages for the +Keep store format. + +### Format identity + +`FORMAT` remains. Its magic, version, and format-definition digest identify +the format contract, not the software release that wrote a store. Normal +opening refuses every other marker, and refuses a root without a marker once +version-1 admission is removed. + +Every grammar change changes the definition digest. Before the first release, +such a change regenerates the affected golden fixtures and needs no +conversion. After the first release, it follows [Future format +changes](#future-format-changes). + +### Direct initialization + +A store is created directly by an initialization protocol under exclusive +writer authority. The protocol is a recoverable state machine: + +- every partial prefix, synchronization failure, interrupted cleanup, and + concurrent initialization attempt has exactly one result: admit, recover, + or refuse; +- a nonempty directory that is not a recoverable initialization prefix is + refused, never adopted; +- neither the existence of `FORMAT` nor the presence of some protocol + directories proves completion. The protocol names one durable commit point + before which the store does not admit; +- missing retention state never means "nothing is retained" when other + artifacts contradict it; and +- an initialized store with no published catalog is a defined state, not an + inferred one. + +While migration code still exists, a migrated store and an initialized store +must be distinguished by an explicit discriminator that is checked on +admission. Treating migration records as optional is not a discriminator. The +unit that adds initialization decides the discriminator and the resulting +definition digest. It does not leave them for the unit that deletes +migration. + +### Portable stores, live authority + +A store is not bound to the directory where it was created. A complete store +copied while nothing was writing to it, or taken from a consistent filesystem +snapshot, opens on any admitted filesystem after normal admission. The +opening process acquires fresh local authority. + +Removed: the requirement that the live root's device and inode match +coordinates persisted in the store. Initialization records no replacement +binding. + +Retained without change: + +- capability-relative, no-follow filesystem access; +- the writer lock and the reader fence, including lock-file identity; +- in-process checks that protocol names still identify the directories + admitted by the same process + ([recovery directory binding](retention-recovery-directory-binding.md)), + and any comparison against an observation the same process made; and +- all content, closure, catalog, retention, and recovery validation. + +This is not permission to copy a live store, to repair a store silently, or +to admit an incomplete store. A copy that is corrupt or incomplete refuses +exactly as the original would. + +Keep does not claim that admission detects an unintended but valid store, or +an older valid state. Detecting either requires an expectation the caller +supplies and keeps outside the state being checked, such as an exact catalog +generation and digest or a retention head digest. A record restored alongside +everything else cannot show that newer history existed. Keep may later offer +such an expected-state check, but only as an explicit caller input. + +The store identifier stays a deterministic, content-derived value. It is not +a unique installation identity, and two stores with identical initial states +may share it. + +### Writer before deletion + +Version 1 and migration are deleted only after the single format can do +everything they do today. The order, tracked in #198, is: + +1. Direct initialization and its crash matrix. +2. The #125 retained-closure gate on every route that can publish a catalog, + including recovery that finalizes a staged catalog. Changing the admission + type that recovery accepts is not sufficient. +3. The segment and catalog writer, with segment and catalog stage recovery, + on the single format. Publication of immutable catalogs and retention roots + keeps running beside readers. Only GC takes the reader fence exclusively, + after writer authority. This unit defines the input envelope it supports + and does not claim the segment rollover in + [#121](https://github.com/flyingrobots/keep/issues/121). +4. Test builders move to direct initialization. Empty stores move after + step 1. Populated fixtures move after step 3 and are built through the real + writer, so that test code is not a second implementation of store + construction. +5. Migration is deleted once one integrated public-API sequence passes: + initialize, ingest, reopen, reconstruct exact bytes, retain, publish a + second catalog, interrupt publication, recover, and read both retained + contents. The sequence also includes a candidate catalog that would lose + retained content, which must refuse with its evidence intact. +6. Version-1 admission is deleted and the format pages are consolidated. + +Retired requirement and crash identifiers are never reused. A retirement +ledger maps each one to its last evidence and states whether its obligation +disappeared with migration or moved to the single format. + +### Future format changes + +After the first release, a release that changes the format ships a separately +invoked upgrade tool. Normal opening never converts a store. + +The tool reads the source store with the previous release's code, pinned to +its tag, and writes a new store with the current code. It never modifies the +source. It checks every content identity and every retention root against +the source. The destination does not admit until the tool's completion +record is durable. The tool's crash protocol is defined with it, and its test +upgrades a fixture store from every prior release. + +## Alternatives considered + +- **Keep both formats and the migration.** Rejected. Every capability would be + built and tested twice. A version-2 store would stay unable to ingest until + that duplicated work landed. +- **Delete version 1 first.** Rejected. Keep would have no durable writer and + no segment or catalog recovery. +- **Define the format in GraphQL and generate codecs with Wesley.** Rejected. + Wesley's breaking-change check follows API rules, which call an added field + safe and do not report field reordering. A stored record breaks under both + changes, so the safety would still have to come from a Keep check. + `definition.tsv` already gives Keep a declarative, hashed format definition. + The approach would add a pre-1.0 build dependency, an IR change, and a + generator for no stronger guarantee. +- **Keep the persistent root binding and add a rebind command.** Rejected. It + adds a durable mutation protocol while answering a narrower question than + the threat it is cited against. A rebind that checks internal consistency + cannot show that the store was intended or current. +- **Rebind automatically on open.** Rejected. It silently rewrites the + evidence the check exists to enforce. +- **Bind the store to the filesystem UUID instead of the device number.** + Rejected. It survives renumbering but still refuses backups and copies, and + it has the same limits for an intended or current store. +- **Upgrade in place, as the migration does.** Rejected for future format + changes. In-place conversion needs a crash matrix over every partial prefix. + Copying into a new store leaves the source intact at every crash point, at + the cost of space for two stores during the upgrade. +- **Self-describing records with tagged, skippable fields.** Rejected. They + give up fixed offsets and bounds checked before allocation, and they + conflict with refusing unknown and noncanonical content. +- **Scrub the whole store on every open, because stores are portable.** + Rejected. Admission, verification of requested reads, and complete backup + verification stay distinct operations. + +## Consequences + +- The definition digest changes at least when initialization adds its + discriminator and again when migration grammar is removed. Golden fixtures + are regenerated each time. +- Until the units in #198 land, main still enforces the persistent root + binding and still creates version-2 stores only by migration. This ADR is a + protocol commitment, not evidence that the change has shipped. +- Removing the binding relaxes `RootIdentityChanged` on reopen and on fenced + snapshots. Live replacement checks keep their existing refusals. +- Portability is accepted through these laws: + - a valid quiescent copy opens; + - a corrupt or incomplete copy refuses as the original would; + - a replaced lock or protocol directory during an operation still refuses; + - a copied store and its original, each written independently, diverge + with independent locks while unchanged content keeps its identity; and + - backup round trip: close, copy, make the original unavailable, reopen and + verify the copy, then interrupt a new write on the copy and recover it. + + Independent copies are not replication and are never described as + coordinated. +- #190 still applies: segment and catalog stage recovery moves to the single + format rather than being deleted, and its recovery outcomes move with it. + The typed migration refusals in #191 are deleted with migration. +- [#108](https://github.com/flyingrobots/keep/issues/108) (migration + recovery) and migration-specific requirement rows are retired through the + ledger, not reused. +- Security: removing a persistent binding does not widen filesystem + authority. All access stays capability-relative and no-follow. The stronger + wrong-store and rollback claims are now stated as unsupported instead of + implied. diff --git a/docs/adr/README.md b/docs/adr/README.md index 1348e0ca..e5effcbb 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -61,3 +61,4 @@ encryption, concurrency, or public-API surface it governs. - [ADR-0007: Terminal signal process-group guard](0007-terminal-signal-process-group-guard.md) - [ADR-0008: Deadline-bounded reader retirement](0008-deadline-bounded-reader-retirement.md) - [ADR-0009: Retention roots, release, and GC liveness](0009-retention-roots-release-and-gc-liveness.md) +- [ADR-0010: One storage format](0010-one-storage-format.md) diff --git a/docs/adr/retention-reader-root-identity.md b/docs/adr/retention-reader-root-identity.md index d3512c51..a6b57df5 100644 --- a/docs/adr/retention-reader-root-identity.md +++ b/docs/adr/retention-reader-root-identity.md @@ -1,6 +1,8 @@ # Retention reader root identity -Status: Accepted +Status: Accepted; the restart comparison is superseded by +[ADR-0010](0010-one-storage-format.md), which makes complete stores portable. +Main enforces this record until that change lands. Joint admission of `FORMAT`, `migration.intent`, and `migration.receipt` establishes that those records agree with each other. It does not establish that the directory receiving the read request is the directory the migration intent names. Reader admission must compare the opened root's physical identity before accepting the version-two read capability. diff --git a/docs/formats/segment-store-v2/recovery.md b/docs/formats/segment-store-v2/recovery.md index d339caae..7868acc2 100644 --- a/docs/formats/segment-store-v2/recovery.md +++ b/docs/formats/segment-store-v2/recovery.md @@ -136,7 +136,9 @@ identity, caller identity, path, and time do not enter the identifier. The migration intent separately binds the physical root coordinates so in-place recovery refuses a substituted store. -### Root identity across restart +(../../adr/0010-one-storage-format.md) supersedes the restart +requirement below and makes complete stores portable. Main enforces this +section until that change lands. The three root coordinates the intent records do not have the same lifetime. `statx.stx_mnt_id` names a mount instance: it can change when a filesystem From 34ef0146eaaaa94f60665e2a453036f1f5701011 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 27 Oct 2026 06:27:49 -0700 Subject: [PATCH 2/9] Docs: order ADR-0010 units so initialization replaces migration in one change (#198) Building the writer on migrated version-2 stores first, then replacing migration with direct initialization in the change that deletes it, means the two store shapes never coexist and needs no transitional discriminator, second corpus or dual admission path. --- docs/adr/0010-one-storage-format.md | 51 ++++++++++++++--------------- 1 file changed, 24 insertions(+), 27 deletions(-) diff --git a/docs/adr/0010-one-storage-format.md b/docs/adr/0010-one-storage-format.md index 81357f13..9140b8a3 100644 --- a/docs/adr/0010-one-storage-format.md +++ b/docs/adr/0010-one-storage-format.md @@ -108,12 +108,11 @@ writer authority. The protocol is a recoverable state machine: - an initialized store with no published catalog is a defined state, not an inferred one. -While migration code still exists, a migrated store and an initialized store -must be distinguished by an explicit discriminator that is checked on -admission. Treating migration records as optional is not a discriminator. The -unit that adds initialization decides the discriminator and the resulting -definition digest. It does not leave them for the unit that deletes -migration. +Initialization replaces migration in the same change that deletes it, so a +migrated store and an initialized store never coexist in the codebase and no +discriminator between them is needed. That change removes the migration +rows from the format definition, which changes the definition digest; the +marker it writes carries the new digest. ### Portable stores, live authority @@ -156,26 +155,25 @@ may share it. Version 1 and migration are deleted only after the single format can do everything they do today. The order, tracked in #198, is: -1. Direct initialization and its crash matrix. -2. The #125 retained-closure gate on every route that can publish a catalog, +1. The #125 retained-closure gate on every route that can publish a catalog, including recovery that finalizes a staged catalog. Changing the admission type that recovery accepts is not sufficient. -3. The segment and catalog writer, with segment and catalog stage recovery, - on the single format. Publication of immutable catalogs and retention roots - keeps running beside readers. Only GC takes the reader fence exclusively, - after writer authority. This unit defines the input envelope it supports - and does not claim the segment rollover in - [#121](https://github.com/flyingrobots/keep/issues/121). -4. Test builders move to direct initialization. Empty stores move after - step 1. Populated fixtures move after step 3 and are built through the real - writer, so that test code is not a second implementation of store - construction. -5. Migration is deleted once one integrated public-API sequence passes: - initialize, ingest, reopen, reconstruct exact bytes, retain, publish a - second catalog, interrupt publication, recover, and read both retained - contents. The sequence also includes a candidate catalog that would lose - retained content, which must refuse with its evidence intact. -6. Version-1 admission is deleted and the format pages are consolidated. +2. The segment and catalog writer, with segment and catalog stage recovery, + on version-2 stores, which until step 4 are still created by migration. + Publication of immutable catalogs and retention roots keeps running beside + readers. Only GC takes the reader fence exclusively, after writer + authority. This unit defines the input envelope it supports and does not + claim the segment rollover in [#121](https://github.com/flyingrobots/keep/issues/121). +3. Populated test fixtures are built through that writer, so that test code is + not a second implementation of store construction. +4. Direct initialization and its crash matrix replace migration, and + migration is deleted in the same change. Before that change merges, one + integrated public-API sequence passes: initialize, ingest, reopen, + reconstruct exact bytes, retain, publish a second catalog, interrupt + publication, recover, and read both retained contents. The sequence also + includes a candidate catalog that would lose retained content, which must + refuse with its evidence intact. +5. Version-1 admission is deleted and the format pages are consolidated. Retired requirement and crash identifiers are never reused. A retirement ledger maps each one to its last evidence and states whether its obligation @@ -229,9 +227,8 @@ upgrades a fixture store from every prior release. ## Consequences -- The definition digest changes at least when initialization adds its - discriminator and again when migration grammar is removed. Golden fixtures - are regenerated each time. +- The definition digest changes when initialization replaces migration and + the migration rows are removed. Golden fixtures are regenerated then. - Until the units in #198 land, main still enforces the persistent root binding and still creates version-2 stores only by migration. This ADR is a protocol commitment, not evidence that the change has shipped. From db81d6a19717e02fe788797382830b68198b6db7 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 27 Oct 2026 06:50:53 -0700 Subject: [PATCH 3/9] Docs: keep ADR-0009 within its 500-line contract (#198) --- docs/adr/0009-retention-roots-release-and-gc-liveness.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/adr/0009-retention-roots-release-and-gc-liveness.md b/docs/adr/0009-retention-roots-release-and-gc-liveness.md index 50e40fa6..d55a096f 100644 --- a/docs/adr/0009-retention-roots-release-and-gc-liveness.md +++ b/docs/adr/0009-retention-roots-release-and-gc-liveness.md @@ -5,9 +5,7 @@ - Owners: Keep retention, catalog, recovery, verification, and GC boundaries - Related issue: [#18](https://github.com/flyingrobots/keep/issues/18) -- Depends on: ADR-0002, ADR-0005, issues #3, #16, and #17 -- Amended by: [ADR-0010](0010-one-storage-format.md), which replaces migration - as the only entry into the format with direct initialization +- Depends on: ADR-0002, ADR-0005, issues #3, #16, and #17; amended by [ADR-0010](0010-one-storage-format.md) ## Context From ed9df3727972bf74b37db87992374197b78aabb4 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 27 Oct 2026 07:55:41 -0700 Subject: [PATCH 4/9] Docs: restore the root-identity heading the ADR links to (#198) --- docs/adr/0010-one-storage-format.md | 3 +-- docs/formats/segment-store-v2/recovery.md | 4 +++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/adr/0010-one-storage-format.md b/docs/adr/0010-one-storage-format.md index 9140b8a3..271853bd 100644 --- a/docs/adr/0010-one-storage-format.md +++ b/docs/adr/0010-one-storage-format.md @@ -34,8 +34,7 @@ The two formats split Keep's capabilities between them: - Every version-2 test, fixture, and crash case first builds a version-1 store and migrates it. The migration accounts for about 10,500 lines under `src/adapters/store_migration`, 2,000 lines of integration tests, 1,100 - lines of repository tasks, two golden records and their source table, and -the + lines of repository tasks, two golden records and their source table, and the `KEEP-CRASH-053..073` process-death matrix. Nothing has been released. The crate is `0.0.0` with `publish = false`. On diff --git a/docs/formats/segment-store-v2/recovery.md b/docs/formats/segment-store-v2/recovery.md index 7868acc2..ebb000a5 100644 --- a/docs/formats/segment-store-v2/recovery.md +++ b/docs/formats/segment-store-v2/recovery.md @@ -136,7 +136,9 @@ identity, caller identity, path, and time do not enter the identifier. The migration intent separately binds the physical root coordinates so in-place recovery refuses a substituted store. -(../../adr/0010-one-storage-format.md) supersedes the restart +### Root identity across restart + +[ADR-0010](../../adr/0010-one-storage-format.md) supersedes the restart requirement below and makes complete stores portable. Main enforces this section until that change lands. From 3cf88d0d25bd9dc313677ff74226b5f907d20638 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 27 Oct 2026 09:47:24 -0700 Subject: [PATCH 5/9] Docs: state that Keep persists no store identifier (#198) --- docs/adr/0010-one-storage-format.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/adr/0010-one-storage-format.md b/docs/adr/0010-one-storage-format.md index 271853bd..aacad220 100644 --- a/docs/adr/0010-one-storage-format.md +++ b/docs/adr/0010-one-storage-format.md @@ -145,9 +145,9 @@ generation and digest or a retention head digest. A record restored alongside everything else cannot show that newer history existed. Keep may later offer such an expected-state check, but only as an explicit caller input. -The store identifier stays a deterministic, content-derived value. It is not -a unique installation identity, and two stores with identical initial states -may share it. +Keep persists no store identifier. The only one, the content-derived +identifier in the migration records, is deleted with migration, and +initialization records no installation identity in its place. ### Writer before deletion From 1c9f28d7cc3e85c94ac3697b0ded850d6a8278c5 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 28 Oct 2026 18:22:32 -0700 Subject: [PATCH 6/9] Docs: define format upgrades as a chain of one-step tools (#198) --- docs/adr/0010-one-storage-format.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/adr/0010-one-storage-format.md b/docs/adr/0010-one-storage-format.md index aacad220..28ae9098 100644 --- a/docs/adr/0010-one-storage-format.md +++ b/docs/adr/0010-one-storage-format.md @@ -183,12 +183,19 @@ disappeared with migration or moved to the single format. After the first release, a release that changes the format ships a separately invoked upgrade tool. Normal opening never converts a store. -The tool reads the source store with the previous release's code, pinned to -its tag, and writes a new store with the current code. It never modifies the -source. It checks every content identity and every retention root against -the source. The destination does not admit until the tool's completion -record is durable. The tool's crash protocol is defined with it, and its test -upgrades a fixture store from every prior release. +Each upgrade tool converts exactly one step: from the format of the previous +format-changing release to the current one. It reads the source store with +that release's code, pinned to its tag, and writes a new store with the +current code. It never modifies the source. It checks every content identity +and every retention root against the source. The destination does not admit +until the tool's completion record is durable. The tool's crash protocol is +defined with it. + +A store several format changes behind upgrades through a chain: the tool of +each intermediate format-changing release, run in order. Each step reads its +source marker and refuses any other, so a step cannot be skipped or run out of +order. Released tools stay available, and every release tests the full chain +from a fixture store of each prior format to the current one. ## Alternatives considered @@ -217,6 +224,10 @@ upgrades a fixture store from every prior release. changes. In-place conversion needs a crash matrix over every partial prefix. Copying into a new store leaves the source intact at every crash point, at the cost of space for two stores during the upgrade. +- **One upgrade tool that embeds a reader for every prior format.** Rejected. + The tool would accumulate every retired format's code, which is the + multi-format codebase this decision forbids. A chain of one-step tools keeps + each retired reader in its own released tag. - **Self-describing records with tagged, skippable fields.** Rejected. They give up fixed offsets and bounds checked before allocation, and they conflict with refusing unknown and noncanonical content. From 4f963a4b94e7e591821fc64ea556872c9500a558 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 3 Nov 2026 08:28:00 -0800 Subject: [PATCH 7/9] Docs: define the upgrade tool's source consistency boundary (#199) ADR-0010 forbids copying a live store but left open how an upgrade tool gets one consistent source view. Hold the source's writer lock for the whole run and refuse otherwise. Addresses the open review finding on #199. --- docs/adr/0010-one-storage-format.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/adr/0010-one-storage-format.md b/docs/adr/0010-one-storage-format.md index 28ae9098..120181d9 100644 --- a/docs/adr/0010-one-storage-format.md +++ b/docs/adr/0010-one-storage-format.md @@ -191,6 +191,15 @@ and every retention root against the source. The destination does not admit until the tool's completion record is durable. The tool's crash protocol is defined with it. +The tool needs one consistent view of its source, and per-item identity checks +do not provide one. It holds the source's writer lock from before its first +read until the destination's completion record is durable, and refuses if the +lock is held or cannot be taken. So no publication can begin during the run, +and a long upgrade blocks writers by design. Taking the lock does not modify +stored content. Existing readers may continue, because they don't change the +source. The tool does not take a filesystem copy of a live store as its source; +a copy qualifies only under the quiescence rule above. + A store several format changes behind upgrades through a chain: the tool of each intermediate format-changing release, run in order. Each step reads its source marker and refuses any other, so a step cannot be skipped or run out of From 9690fb8030f053622e0ba254961a37df6b60eb02 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 4 Nov 2026 16:39:28 -0800 Subject: [PATCH 8/9] Docs: read old segments and write one current format (#199) Owner review of ADR-0010: copying a whole store for every format change costs too much on large stores. After the first release Keep keeps frozen decoders for sealed segments and writes only the current format. Metadata upgrades stay explicit, segment rewriting is optional and resumable, and reads never upgrade. Removing the root binding now comes with a required typed expected-state choice on open. The digest claim is corrected and #205 must close before first release. The torn-copy and rollback laws block migration deletion. --- CHANGELOG.md | 2 +- docs/adr/0010-one-storage-format.md | 151 +++++++++++++++++++--------- docs/adr/README.md | 2 +- 3 files changed, 104 insertions(+), 51 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 907dae04..34492c21 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ after its public API and format compatibility policies are established. - Add field-mutation laws for selected version-2 retention root, manifest, and head fields: rows mutate the frozen corpus record, reseal every digest and checksum they did not target, and assert the named refusal variant. KEEP-RETENTION-003 remains in progress (#212). Salvaged from the closed #107 branch; tests and ledger only. -- ADR-0010 commits Keep to one storage format: direct initialization replaces version-1 initialization plus migration, complete stores become portable while live authority checks remain, and future format changes ship a separate upgrade tool. Main's behavior is unchanged until the units tracked in #198 land (#198). +- ADR-0010 commits Keep to one current storage format: direct initialization replaces version-1 initialization plus migration, complete stores become portable while live authority checks remain, opening requires an explicit expected-state choice before the first release, and after it Keep reads released sealed segments while writing only the current format, with explicit metadata upgrades and optional, resumable segment rewriting. Main's behavior is unchanged until the units tracked in #198 land (#198). - Update the repository-only serde_json parser to 1.0.154 in both locked workspaces and refresh its dependency admission; the resolved transitive graph, features, license and MSRV are unchanged (#186). - Update the in-process BLAKE3 dependency to 1.8.7 in both locked workspaces, remove the unused arrayref license exception and renew its admission while retaining the external b3sum oracle and version-one identity contract (#94). diff --git a/docs/adr/0010-one-storage-format.md b/docs/adr/0010-one-storage-format.md index 120181d9..b2fedc2b 100644 --- a/docs/adr/0010-one-storage-format.md +++ b/docs/adr/0010-one-storage-format.md @@ -1,7 +1,7 @@ -# ADR-0010: One Storage Format +# ADR-0010: One Current Storage Format - Status: Accepted -- Date: 2026-10-27 +- Date: 2026-10-27; amended 2026-11-04 - Owners: Keep format, initialization, publication, recovery, and retention boundaries - Related issue: @@ -63,8 +63,10 @@ been read as: ### One supported format -Keep admits, writes, and recovers exactly one storage format. The codebase -never carries a second runtime format alongside it. The format is the +Keep writes and recovers exactly one storage format, the current format. The +codebase never carries a second writer. Until the first release that is also the +only format Keep reads. After it, Keep also reads released sealed segments, as +[Future format changes](#future-format-changes) defines. The format is the version-2 record set and namespace without migration artifacts: - the segment, catalog, and publication-head grammars, byte-for-byte as @@ -84,9 +86,12 @@ the format contract, not the software release that wrote a store. Normal opening refuses every other marker, and refuses a root without a marker once version-1 admission is removed. -Every grammar change changes the definition digest. Before the first release, -such a change regenerates the affected golden fixtures and needs no -conversion. After the first release, it follows [Future format +Every grammar change must change the definition digest. Today the digest does +not cover the segment, catalog, and head grammars +([#205](https://github.com/flyingrobots/keep/issues/205)), so this is a +requirement rather than a current property, and #205 must close before the first +release. Before the first release, a grammar change regenerates the affected +golden fixtures and needs no conversion. After the first release, it follows [Future format changes](#future-format-changes). ### Direct initialization @@ -138,12 +143,17 @@ This is not permission to copy a live store, to repair a store silently, or to admit an incomplete store. A copy that is corrupt or incomplete refuses exactly as the original would. -Keep does not claim that admission detects an unintended but valid store, or -an older valid state. Detecting either requires an expectation the caller -supplies and keeps outside the state being checked, such as an exact catalog -generation and digest or a retention head digest. A record restored alongside -everything else cannot show that newer history existed. Keep may later offer -such an expected-state check, but only as an explicit caller input. +Admission alone does not detect an unintended but valid store, or an older valid +state. Detecting either requires an expectation the caller supplies and keeps +outside the state being checked, such as an exact catalog generation and digest +or a retention head digest. A record restored alongside everything else cannot +show that newer history existed. + +Before the first release, opening a store therefore requires a typed choice: +the caller passes the state it expects, and a mismatch refuses, or the caller +explicitly states that it has no expectation. Neither is a default, so removing +the binding never silently removes the question. Keep does not store the +expectation. Keep persists no store identifier. The only one, the content-derived identifier in the migration records, is deleted with migration, and @@ -171,7 +181,10 @@ everything they do today. The order, tracked in #198, is: reconstruct exact bytes, retain, publish a second catalog, interrupt publication, recover, and read both retained contents. The sequence also includes a candidate catalog that would lose retained content, which must - refuse with its evidence intact. + refuse with its evidence intact. The change also carries laws for two copies: + a copy torn by a concurrent writer never returns bytes other than those its + identities name, and an older valid snapshot refuses when the caller passes + newer expected state. 5. Version-1 admission is deleted and the format pages are consolidated. Retired requirement and crash identifiers are never reused. A retirement @@ -180,31 +193,61 @@ disappeared with migration or moved to the single format. ### Future format changes -After the first release, a release that changes the format ships a separately -invoked upgrade tool. Normal opening never converts a store. - -Each upgrade tool converts exactly one step: from the format of the previous -format-changing release to the current one. It reads the source store with -that release's code, pinned to its tag, and writes a new store with the -current code. It never modifies the source. It checks every content identity -and every retention root against the source. The destination does not admit -until the tool's completion record is durable. The tool's crash protocol is -defined with it. - -The tool needs one consistent view of its source, and per-item identity checks -do not provide one. It holds the source's writer lock from before its first -read until the destination's completion record is durable, and refuses if the -lock is held or cannot be taken. So no publication can begin during the run, -and a long upgrade blocks writers by design. Taking the lock does not modify -stored content. Existing readers may continue, because they don't change the -source. The tool does not take a filesystem copy of a live store as its source; -a copy qualifies only under the quiescence rule above. - -A store several format changes behind upgrades through a chain: the tool of -each intermediate format-changing release, run in order. Each step reads its -source marker and refuses any other, so a step cannot be skipped or run out of -order. Released tools stay available, and every release tests the full chain -from a fixture store of each prior format to the current one. +After the first release, Keep reads old data and writes the current format. No +format change requires rewriting a store's segments, except a change forced by +a broken integrity primitive, which needs its own decision. + +Sealed segments are immutable, and each records its own magic and +`format_version`. Keep therefore retains a frozen decoder for every released +segment format and selects it per segment. Each frozen decoder keeps its golden +fixtures and fuzz targets in every release. An unknown or unlisted version is +still refused. New data is always written in the current format. Released +segment formats stay readable indefinitely. A support window that retires a +decoder would need a later decision, because it could strand a store that has +not been rewritten. + +Catalog, publication-head, retention, and other mutable metadata has one +current version. A release that changes it, or that changes the definition +digest, ships a separately invoked metadata upgrade, which also rewrites the +`FORMAT` marker. Normal opening never converts a store, and it refuses a marker +other than the current one until that upgrade has run. + +Reads never upgrade. A read holds no writer authority, must behave the same on +a full disk or a read-only mount, and must not change bytes that backups and +synchronization observe. It would also leave unread data behind, so it could +not retire a decoder. + +Rewriting old segments into the current format is optional. It is one explicit +operation under writer authority that is resumable, can be limited by bytes or +time, and verifies every content identity before an old segment becomes +removable. The operator chooses when it runs, including never. Keep stores no +per-store upgrade policy. An application may schedule the operation, for +example as a background job, through a public interface that reports progress +and accepts cancellation. Keep owns no scheduler or user interface. + +Keep reports how many segments and bytes each format holds, so an application +can show that state to an operator. Format is physical layout, so it is not +reported per content identity and is not a warning about content. Content in a +frozen format returns exactly the same bytes as content in the current format. + +These principles bind every future upgrade: + +- content identity never depends on the format that stores it; +- every crash leaves a store that admits or refuses, never one that silently + loses content; +- a store can be read throughout segment rewriting; and +- old segments are removed only through GC's exclusive reader fence. + +Open questions, to be answered by a minimal upgrade tool built after #198 +lands rather than specified here: + +- how long a metadata upgrade takes and how much space it needs for a store + with many files, since catalog size grows with the number of files; +- whether a metadata upgrade writes a new store or converts in place, and what + crash matrix that choice requires; +- what the operation's resume state is, and where it lives; and +- whether operators want a stored schedule or a recorded order, such as most + read segments first. Any such hint must not make a read write to the store. ## Alternatives considered @@ -229,14 +272,18 @@ from a fixture store of each prior format to the current one. - **Bind the store to the filesystem UUID instead of the device number.** Rejected. It survives renumbering but still refuses backups and copies, and it has the same limits for an intended or current store. -- **Upgrade in place, as the migration does.** Rejected for future format - changes. In-place conversion needs a crash matrix over every partial prefix. - Copying into a new store leaves the source intact at every crash point, at - the cost of space for two stores during the upgrade. -- **One upgrade tool that embeds a reader for every prior format.** Rejected. - The tool would accumulate every retired format's code, which is the - multi-format codebase this decision forbids. A chain of one-step tools keeps - each retired reader in its own released tag. +- **Copy the whole store for every format change.** Rejected. Every byte is read, + hashed, and rewritten, the store needs space for two copies, and writers stay + blocked throughout. That cost grows with the store for a change that sealed + segments do not need. +- **Read and write every released format.** Rejected. Every write path and crash + matrix would multiply per format. +- **Upgrade on read.** Rejected. See [Future format + changes](#future-format-changes). +- **A stored per-store upgrade policy.** Rejected for now. It would add states to + test without evidence that running an explicit operation is insufficient. +- **Mark content stored in an older format.** Rejected. It would expose physical + layout as a property of content, and content can share or span segments. - **Self-describing records with tagged, skippable fields.** Rejected. They give up fixed offsets and bounds checked before allocation, and they conflict with refusing unknown and noncanonical content. @@ -264,6 +311,12 @@ from a fixture store of each prior format to the current one. Independent copies are not replication and are never described as coordinated. +- Frozen segment decoders stay in the codebase permanently. This narrows the + owner's original one-format policy of 2026-10-26, in exchange for upgrades that + do not rewrite a whole store. Each released segment format is a permanent + commitment, so a format ships only when Keep will support it indefinitely. +- A broken integrity primitive still forces a rewrite, because a frozen decoder + cannot repair it. - #190 still applies: segment and catalog stage recovery moves to the single format rather than being deleted, and its recovery outcomes move with it. The typed migration refusals in #191 are deleted with migration. @@ -272,5 +325,5 @@ from a fixture store of each prior format to the current one. ledger, not reused. - Security: removing a persistent binding does not widen filesystem authority. All access stays capability-relative and no-follow. The stronger - wrong-store and rollback claims are now stated as unsupported instead of - implied. + wrong-store and rollback claims hold only for a caller that passes expected + state, and every caller must choose whether to. diff --git a/docs/adr/README.md b/docs/adr/README.md index e5effcbb..6f18e1d8 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -61,4 +61,4 @@ encryption, concurrency, or public-API surface it governs. - [ADR-0007: Terminal signal process-group guard](0007-terminal-signal-process-group-guard.md) - [ADR-0008: Deadline-bounded reader retirement](0008-deadline-bounded-reader-retirement.md) - [ADR-0009: Retention roots, release, and GC liveness](0009-retention-roots-release-and-gc-liveness.md) -- [ADR-0010: One storage format](0010-one-storage-format.md) +- [ADR-0010: One current storage format](0010-one-storage-format.md) From 7903f414b5a6ff7503f9878d7b373c95beacba18 Mon Sep 17 00:00:00 2001 From: James Ross Date: Wed, 4 Nov 2026 16:39:53 -0800 Subject: [PATCH 9/9] Docs: keep the metadata upgrade source consistency rule (#199) --- docs/adr/0010-one-storage-format.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/adr/0010-one-storage-format.md b/docs/adr/0010-one-storage-format.md index b2fedc2b..cfa26cbd 100644 --- a/docs/adr/0010-one-storage-format.md +++ b/docs/adr/0010-one-storage-format.md @@ -91,8 +91,8 @@ not cover the segment, catalog, and head grammars ([#205](https://github.com/flyingrobots/keep/issues/205)), so this is a requirement rather than a current property, and #205 must close before the first release. Before the first release, a grammar change regenerates the affected -golden fixtures and needs no conversion. After the first release, it follows [Future format -changes](#future-format-changes). +golden fixtures and needs no conversion. After the first release, it follows +[Future format changes](#future-format-changes). ### Direct initialization @@ -209,7 +209,9 @@ not been rewritten. Catalog, publication-head, retention, and other mutable metadata has one current version. A release that changes it, or that changes the definition digest, ships a separately invoked metadata upgrade, which also rewrites the -`FORMAT` marker. Normal opening never converts a store, and it refuses a marker +`FORMAT` marker. It holds the writer lock from before its first read until its +completion is durable, and refuses if the lock is held, so it sees one consistent +source and blocks writers by design. Normal opening never converts a store, and it refuses a marker other than the current one until that upgrade has run. Reads never upgrade. A read holds no writer authority, must behave the same on