Continuity: the recovery procedure rehearsed on Fly

RUNBOOK.md

@@ -172,7 +172,7 @@ restore test" on every bank's Controls page, in place of `OPENCLEAR_LAST_RESTORE
 ## Recovering the installation (for the service provider)
 
 In this order, because each step needs the one before it (CONTINUITY_PLAN.md BC-D7; rehearse it
-against the sandbox at least yearly, and record the rehearsal):
+on your own installation at least yearly, and record the rehearsal):
 
 1. **Tell the banks** from Platform → Incidents (an *Incident* notice: what happened, that
    decisions already made are safe, when you'll update them), or by the incident contacts' emails

plans/CONTINUITY_PLAN.md

@@ -140,3 +140,27 @@ model. The host's backup job stays the host's; OpenClear only reads what it leav
   (`d4e8f2a6c1b7`, `e5f9a3b7d2c8`, `f7a1b5c9d3e2`), four controls (OC-CP-02 to OC-CP-04 and the
   checks under them), one script (`scripts/restore_drill.py`).
 - 2026-10-31: Pull request #71. CodeQL flagged the open-items download's `?day=` reaching a file name (three path-injection alerts): the route now builds the name from the parsed date, and a local `read_key` goes through `LocalStorage._path`'s resolved-path check like every other local read.
+- 2026-11-02: **The recovery procedure rehearsed on Fly**, against the sandbox's volume snapshot
+  of 14:23 UTC, restored into a separate app (`openclear-sandbox-drill`) and destroyed after.
+  Steps 2 to 8 in order: the signing keys came back with the volume (fingerprints match the
+  sandbox's); the three Fly secrets were copied from the running sandbox; maintenance mode on from
+  the first boot (a write answered 503 with the notice); 1.35.0 deployed by image digest; `/ready`
+  database and storage ok; `scripts/restore_drill.py` passed inside the machine (four banks' chains
+  intact, no row missing its file, four ML model files without a row) and recorded the restore
+  test; maintenance off, the first sweep ran and `/ready` was ready. Measured:
+
+  | From the first command | |
+  |---|---|
+  | Instance up, in maintenance | 2 min 47 s |
+  | Verified (drill passed) | 3 min 51 s |
+  | Serving, sweep run | 9 min 41 s |
+  | Data lost (snapshot age) | 2 h 44 min; up to 24 h with daily snapshots |
+
+  Found: (1) the SSO encryption key and the attestation texts exist only as Fly secrets, which
+  can't be read back, so the recovery needed the sandbox's machine alive: with the sandbox lost,
+  every SAML secret is lost; they belong in a key store too. (2) `scripts/restore_drill.py` isn't
+  in the image, so step 9 needed it copied in. (3) The image leaves the backup-manifest directory
+  off the `/data` volume, so the SQLite copy and its manifest go with each deploy. (4) Fly keeps
+  five days of snapshots. (5) The sandbox states no recovery objectives. (6) No open item had a
+  deadline in the gap, so the first sweep had nothing to move: correct, but the outage handling
+  wasn't seen on real items. The rehearsal on a real deployment stays parked (PARKED_PLAN.md).

plans/PARKED_PLAN.md

@@ -45,6 +45,7 @@ The product owner's decision of 2026-10-12: these wait for a first client, who w
 | **Confirming the edge on a real zone** (Cloudflare, then Akamai); built and tested against a stub only | A real zone | MASTER_PLAN.md M7; WAF_PLAN.md W0 |
 | **Publishing the SDKs** to PyPI and npm, and the docs site | Approval to publish | MASTER_PLAN.md; SDK_PLAN.md |
 | **An independent penetration test**; its report joins the third-party risk pack | Funding | OWASP_PLAN.md OW-D2 |
+| **A rehearsal of the runbook's "Recovering the installation" on a bank's or provider's own installation**: Postgres and object storage restored to one point, real open items across the gap so the first sweep's outage handling is seen working, and the time against the recovery objectives that installation states. The sandbox was rehearsed on 2026-11-02 (CONTINUITY_PLAN.md progress log); it has one SQLite file, no open items and no stated objectives, so it couldn't show those | A bank or a service provider deploying | CONTINUITY_PLAN.md BC6 |
 
 ## Dates
 
@@ -58,3 +59,4 @@ The product owner's decision of 2026-10-12: these wait for a first client, who w
 - 2026-10-22: Counsel's review of the check-law page added (REGULATORY_PLAN.md RG-D4).
 - 2026-10-22: A direct CRI Profile crosswalk added (REGULATORY_PLAN.md RG-D6).
 - 2026-10-23: Retention periods for a bank's records added (OFFBOARDING_PLAN.md OB-D1).
+- 2026-11-02: The rehearsal of the recovery procedure on a real deployment added (CONTINUITY_PLAN.md BC6); the sandbox's was run the same day.