From 061e1686f98900d1ac9c3bd8da82b618d28fb10c Mon Sep 17 00:00:00 2001 From: Chaffed Date: Sat, 31 Oct 2026 18:31:00 -0700 Subject: [PATCH 1/2] Park the recovery rehearsal for the first real deployment The product owner's decision: the runbook's "Recovering the installation" is rehearsed on the first deployment by a bank or a service provider, not on the sandbox (one SQLite machine whose backup is Fly's volume snapshot). PARKED_PLAN.md carries it with three things the sandbox's look found for that deployment to check; the runbook says to rehearse on your own installation. Co-Authored-By: Claude Opus 5.5 --- RUNBOOK.md | 2 +- plans/CONTINUITY_PLAN.md | 1 + plans/PARKED_PLAN.md | 2 ++ 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/RUNBOOK.md b/RUNBOOK.md index 95240e5a..a1ad6109 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -172,7 +172,7 @@ restore test" on every bank's Controls page, in place of `OPENCLEAR_LAST_RESTORE ## Recovering the installation (for the service provider) In this order, because each step needs the one before it (CONTINUITY_PLAN.md BC-D7; rehearse it -against the sandbox at least yearly, and record the rehearsal): +on your own installation at least yearly, and record the rehearsal): 1. **Tell the banks** from Platform → Incidents (an *Incident* notice: what happened, that decisions already made are safe, when you'll update them), or by the incident contacts' emails diff --git a/plans/CONTINUITY_PLAN.md b/plans/CONTINUITY_PLAN.md index 4d49c7a3..d8a01cb4 100644 --- a/plans/CONTINUITY_PLAN.md +++ b/plans/CONTINUITY_PLAN.md @@ -140,3 +140,4 @@ model. The host's backup job stays the host's; OpenClear only reads what it leav (`d4e8f2a6c1b7`, `e5f9a3b7d2c8`, `f7a1b5c9d3e2`), four controls (OC-CP-02 to OC-CP-04 and the checks under them), one script (`scripts/restore_drill.py`). - 2026-10-31: Pull request #71. CodeQL flagged the open-items download's `?day=` reaching a file name (three path-injection alerts): the route now builds the name from the parsed date, and a local `read_key` goes through `LocalStorage._path`'s resolved-path check like every other local read. +- 2026-11-01: The tabletop of the recovery procedure parked (PARKED_PLAN.md), at the product owner's decision: it is rehearsed on the first deployment by a bank or a service provider, not on the sandbox. The sandbox's look found three things for that deployment to check (the backup-manifest directory off the persistent volume in the image, the keys only on the host, five days of snapshots), recorded with the parked item. diff --git a/plans/PARKED_PLAN.md b/plans/PARKED_PLAN.md index d51ce3df..4845c446 100644 --- a/plans/PARKED_PLAN.md +++ b/plans/PARKED_PLAN.md @@ -45,6 +45,7 @@ The product owner's decision of 2026-10-12: these wait for a first client, who w | **Confirming the edge on a real zone** (Cloudflare, then Akamai); built and tested against a stub only | A real zone | MASTER_PLAN.md M7; WAF_PLAN.md W0 | | **Publishing the SDKs** to PyPI and npm, and the docs site | Approval to publish | MASTER_PLAN.md; SDK_PLAN.md | | **An independent penetration test**; its report joins the third-party risk pack | Funding | OWASP_PLAN.md OW-D2 | +| **A rehearsal of the runbook's "Recovering the installation"**: a snapshot or backup restored into a second instance, the nine steps walked with maintenance mode on, the time against the recovery objectives, recorded with `scripts/restore_drill.py`. On the deployment it protects, not the sandbox: the sandbox is one SQLite machine whose backup is Fly's volume snapshot, so it would rehearse little a bank's installation does. Found while planning it on the sandbox, for that deployment to check: the backup-manifest directory must be on persistent storage (the image leaves `OPENCLEAR_BACKUP_MANIFEST_STORAGE_DIR` at `./data/backups`, off the `/data` volume); the signing keys and `OPENCLEAR_SSO_ENCRYPTION_KEY` must be in a key store as well as on the host; the backups must outlast the time it takes to notice a loss (Fly keeps five days of snapshots by default) | A bank or a service provider deploying | CONTINUITY_PLAN.md BC6 | ## Dates @@ -58,3 +59,4 @@ The product owner's decision of 2026-10-12: these wait for a first client, who w - 2026-10-22: Counsel's review of the check-law page added (REGULATORY_PLAN.md RG-D4). - 2026-10-22: A direct CRI Profile crosswalk added (REGULATORY_PLAN.md RG-D6). - 2026-10-23: Retention periods for a bank's records added (OFFBOARDING_PLAN.md OB-D1). +- 2026-11-01: The rehearsal of the recovery procedure added (CONTINUITY_PLAN.md BC6), at the product owner's decision: on the first real deployment, not the sandbox. From fa39a7731961897ff47f36ea13a1ae6e4c612df6 Mon Sep 17 00:00:00 2001 From: Chaffed Date: Mon, 2 Nov 2026 09:18:06 -0800 Subject: [PATCH 2/2] The recovery procedure rehearsed on Fly against the sandbox's snapshot Steps 2 to 8 of the runbook's "Recovering the installation" on a separate Fly app restored from the sandbox's volume snapshot: verified in under four minutes, serving in under ten, the restore drill passed. The findings and timings are in CONTINUITY_PLAN.md; the rehearsal on a real deployment (Postgres, object storage, open items, stated objectives) stays parked. Co-Authored-By: Claude Opus 5.5 --- plans/CONTINUITY_PLAN.md | 25 ++++++++++++++++++++++++- plans/PARKED_PLAN.md | 4 ++-- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/plans/CONTINUITY_PLAN.md b/plans/CONTINUITY_PLAN.md index d8a01cb4..e099c24e 100644 --- a/plans/CONTINUITY_PLAN.md +++ b/plans/CONTINUITY_PLAN.md @@ -140,4 +140,27 @@ model. The host's backup job stays the host's; OpenClear only reads what it leav (`d4e8f2a6c1b7`, `e5f9a3b7d2c8`, `f7a1b5c9d3e2`), four controls (OC-CP-02 to OC-CP-04 and the checks under them), one script (`scripts/restore_drill.py`). - 2026-10-31: Pull request #71. CodeQL flagged the open-items download's `?day=` reaching a file name (three path-injection alerts): the route now builds the name from the parsed date, and a local `read_key` goes through `LocalStorage._path`'s resolved-path check like every other local read. -- 2026-11-01: The tabletop of the recovery procedure parked (PARKED_PLAN.md), at the product owner's decision: it is rehearsed on the first deployment by a bank or a service provider, not on the sandbox. The sandbox's look found three things for that deployment to check (the backup-manifest directory off the persistent volume in the image, the keys only on the host, five days of snapshots), recorded with the parked item. +- 2026-11-02: **The recovery procedure rehearsed on Fly**, against the sandbox's volume snapshot + of 14:23 UTC, restored into a separate app (`openclear-sandbox-drill`) and destroyed after. + Steps 2 to 8 in order: the signing keys came back with the volume (fingerprints match the + sandbox's); the three Fly secrets were copied from the running sandbox; maintenance mode on from + the first boot (a write answered 503 with the notice); 1.35.0 deployed by image digest; `/ready` + database and storage ok; `scripts/restore_drill.py` passed inside the machine (four banks' chains + intact, no row missing its file, four ML model files without a row) and recorded the restore + test; maintenance off, the first sweep ran and `/ready` was ready. Measured: + + | From the first command | | + |---|---| + | Instance up, in maintenance | 2 min 47 s | + | Verified (drill passed) | 3 min 51 s | + | Serving, sweep run | 9 min 41 s | + | Data lost (snapshot age) | 2 h 44 min; up to 24 h with daily snapshots | + + Found: (1) the SSO encryption key and the attestation texts exist only as Fly secrets, which + can't be read back, so the recovery needed the sandbox's machine alive: with the sandbox lost, + every SAML secret is lost; they belong in a key store too. (2) `scripts/restore_drill.py` isn't + in the image, so step 9 needed it copied in. (3) The image leaves the backup-manifest directory + off the `/data` volume, so the SQLite copy and its manifest go with each deploy. (4) Fly keeps + five days of snapshots. (5) The sandbox states no recovery objectives. (6) No open item had a + deadline in the gap, so the first sweep had nothing to move: correct, but the outage handling + wasn't seen on real items. The rehearsal on a real deployment stays parked (PARKED_PLAN.md). diff --git a/plans/PARKED_PLAN.md b/plans/PARKED_PLAN.md index 4845c446..346dfe58 100644 --- a/plans/PARKED_PLAN.md +++ b/plans/PARKED_PLAN.md @@ -45,7 +45,7 @@ The product owner's decision of 2026-10-12: these wait for a first client, who w | **Confirming the edge on a real zone** (Cloudflare, then Akamai); built and tested against a stub only | A real zone | MASTER_PLAN.md M7; WAF_PLAN.md W0 | | **Publishing the SDKs** to PyPI and npm, and the docs site | Approval to publish | MASTER_PLAN.md; SDK_PLAN.md | | **An independent penetration test**; its report joins the third-party risk pack | Funding | OWASP_PLAN.md OW-D2 | -| **A rehearsal of the runbook's "Recovering the installation"**: a snapshot or backup restored into a second instance, the nine steps walked with maintenance mode on, the time against the recovery objectives, recorded with `scripts/restore_drill.py`. On the deployment it protects, not the sandbox: the sandbox is one SQLite machine whose backup is Fly's volume snapshot, so it would rehearse little a bank's installation does. Found while planning it on the sandbox, for that deployment to check: the backup-manifest directory must be on persistent storage (the image leaves `OPENCLEAR_BACKUP_MANIFEST_STORAGE_DIR` at `./data/backups`, off the `/data` volume); the signing keys and `OPENCLEAR_SSO_ENCRYPTION_KEY` must be in a key store as well as on the host; the backups must outlast the time it takes to notice a loss (Fly keeps five days of snapshots by default) | A bank or a service provider deploying | CONTINUITY_PLAN.md BC6 | +| **A rehearsal of the runbook's "Recovering the installation" on a bank's or provider's own installation**: Postgres and object storage restored to one point, real open items across the gap so the first sweep's outage handling is seen working, and the time against the recovery objectives that installation states. The sandbox was rehearsed on 2026-11-02 (CONTINUITY_PLAN.md progress log); it has one SQLite file, no open items and no stated objectives, so it couldn't show those | A bank or a service provider deploying | CONTINUITY_PLAN.md BC6 | ## Dates @@ -59,4 +59,4 @@ The product owner's decision of 2026-10-12: these wait for a first client, who w - 2026-10-22: Counsel's review of the check-law page added (REGULATORY_PLAN.md RG-D4). - 2026-10-22: A direct CRI Profile crosswalk added (REGULATORY_PLAN.md RG-D6). - 2026-10-23: Retention periods for a bank's records added (OFFBOARDING_PLAN.md OB-D1). -- 2026-11-01: The rehearsal of the recovery procedure added (CONTINUITY_PLAN.md BC6), at the product owner's decision: on the first real deployment, not the sandbox. +- 2026-11-02: The rehearsal of the recovery procedure on a real deployment added (CONTINUITY_PLAN.md BC6); the sandbox's was run the same day.