From 1d638e102954182599a1cd59696402fa6e610cc2 Mon Sep 17 00:00:00 2001 From: Brian O'Kelley Date: Fri, 6 Nov 2026 12:17:58 +0000 Subject: [PATCH] docs(reporting-storage): say how to deploy the 0.2.0b1 ledger upgrade without breaking old pods A default rolling update leaves the earlier release's pods running after the first new pod has upgraded the ledger at startup. Those pods can still read, but they can no longer record a generation or close a period, and with status notifications they can no longer record status checkpoints. Each write names a conflict key that migrations 0002 and 0006 widened. If one restarts, it cannot start Reliable Reporting, and its error points at migrate_legacy_reporting, which does not apply here and would quarantine state. Add "Deploying the upgrade" to the ledger migration guide. It lists those symptoms, says to ignore the hint, and says there is no rollback short of restoring a backup. It gives two safe shapes: a Recreate rollout, or two phases (redeploy the earlier release with reporting off, then roll forward). Point to it from MIGRATION_v9_to_v10.md and the storage README. Refs #1831 Co-Authored-By: Claude Opus 5.5 --- MIGRATION_v9_to_v10.md | 8 +++- docs/reporting-ledger-migration.md | 49 +++++++++++++++++++++++ packages/adcp-reporting-storage/README.md | 6 +++ 3 files changed, 61 insertions(+), 2 deletions(-) diff --git a/MIGRATION_v9_to_v10.md b/MIGRATION_v9_to_v10.md index b170b5a8f..b12b0b7ee 100644 --- a/MIGRATION_v9_to_v10.md +++ b/MIGRATION_v9_to_v10.md @@ -76,5 +76,9 @@ adcp 10.0.0b0–b3 are 1): - An existing SDK 9 ledger is upgraded in place by `create_schema()` or `python -m adcp migrate`: `offering_id` is added with `''` for every existing account-level generation, obligation and status, and the applied - schema versions are recorded. Stop every SDK 9 reporting worker first; see - [docs/reporting-ledger-migration.md](docs/reporting-ledger-migration.md). + schema versions are recorded. Stop every SDK 9 reporting worker first. A + default rolling update does not: once the first new pod upgrades the ledger, + old pods can no longer record generations or close periods, and cannot start + reporting if they restart. There is no rollback short of restoring a backup. + See [Deploying the upgrade](docs/reporting-ledger-migration.md#deploying-the-upgrade) + for a Recreate or two-phase rollout. diff --git a/docs/reporting-ledger-migration.md b/docs/reporting-ledger-migration.md index b02a932c8..36578c91a 100644 --- a/docs/reporting-ledger-migration.md +++ b/docs/reporting-ledger-migration.md @@ -314,6 +314,55 @@ one transaction. It is a stopped-worker upgrade: stop every worker of the earlier release first, because their configuration writes name the four-column key. +### Deploying the upgrade + +This applies whenever more than one process runs Reliable Reporting against +the ledger. That includes several replicas that each start +`ReliableReportingService`, and a separate worker deployment. A default +rolling update does **not** stop the earlier release first. The first new +process upgrades the ledger when it starts, while the old processes keep +running against it. From then on, an old process: + +- still reads configurations, obligations, revisions and rows; +- cannot record a new generation (`put_configuration` fails with + `InvalidColumnReference: there is no unique or exclusion constraint matching + the ON CONFLICT specification`); +- cannot close the next period of an existing generation (`commit_obligation` + fails with `LedgerConflictError: the ledger write violated an integrity rule`, + because the period key it names was widened); +- with status notifications, cannot record status checkpoints, whose key + migration `0006` widens; +- cannot start Reliable Reporting again if it restarts. `create_schema()` + raises `Reporting caller ownership requires a stopped-worker maintenance + migration`, with a hint to use `migrate_legacy_reporting`. **Ignore that + hint.** `migrate_legacy_reporting` is the SDK 8 → 9 caller-ownership + migration. It does not apply to this upgrade and would move reporting state + into quarantine. + +Nothing is corrupted: the new release reads every record the old one wrote. +But the old processes' reporting stops for the rest of the rollout. + +**There is no rollback.** Once the ledger is upgraded, no earlier release can +start Reliable Reporting against it. To go back, restore the backup taken +before the upgrade; evidence written after the backup is lost. + +To deploy: + +1. Back up the database. +2. Stop every reporting worker of the earlier release. Two ways: + - **Recreate.** Roll this one deployment out with every old pod stopped + before any new pod starts (Kubernetes `strategy: {type: Recreate}`). + Reporting, and the pods, are down for the length of the swap. + - **Two phases.** First redeploy the *earlier* release with Reliable + Reporting turned off (don't start `ReliableReportingService`), or scale + a separate worker deployment to zero. Then roll out the new release + normally. Only reporting pauses, not the rest of the service. +3. Upgrade the schema: run `python -m adcp migrate` once, or let the first new + process do it at startup. Run `python -m adcp migrate` (or + `ReportingStatusService.migrate()`) if you use `postgres_storage(...)` or + status notifications. `create_schema()` alone applies only `0001`–`0003`. +4. Start the new release everywhere. + `tests/conformance/reporting/test_reporting_storage_published_upgrade_pg.py` seeds ledgers with the actual published wheel pairs (`adcp==9.3.1` with `adcp-reporting-storage==0.1.0b4`, `adcp==10.0.0b2` with `0.2.0b0`) and proves diff --git a/packages/adcp-reporting-storage/README.md b/packages/adcp-reporting-storage/README.md index 93767ff5c..7acce1576 100644 --- a/packages/adcp-reporting-storage/README.md +++ b/packages/adcp-reporting-storage/README.md @@ -94,6 +94,12 @@ rewritten or dropped. The upgrade is transactional, serialized under the reporti advisory lock, and a no-op when re-run. It is a **stopped-worker** upgrade: stop every reporting worker of the earlier release first (its configuration writes name the four-column key), run the upgrade, then start the new release. +A default rolling update does not stop them. Old pods' reporting writes fail +once the first new pod has upgraded the ledger, and an old pod cannot start +reporting again. Downgrading is unsupported: restore the backup taken before +the upgrade. See +[Deploying the upgrade](https://github.com/adcontextprotocol/adcp-client-python/blob/next/docs/reporting-ledger-migration.md#deploying-the-upgrade) +for a Recreate or two-phase rollout. ## Versioning