Continuity plan: BC1–BC6

CHANGELOG.md

@@ -1,5 +1,45 @@
 # Changelog
 
+## Unreleased
+
+- **An outage across a cut-off no longer decides by default** (`plans/CONTINUITY_PLAN.md` BC1).
+  When OpenClear comes back after being down longer than its sweep interval, the items whose
+  cut-off passed in the gap are given the bank's minimum review time again, never past a check's
+  midnight deadline, instead of the customer's default being applied to what nobody could look
+  at. The bank's administrators and the customers' deciders are emailed, the gap is in the action
+  log and a `continuity.gap` security event (high), and the Controls page shows it for a week
+  (new control OC-CP-02, check *No outage across a cut-off*). An item whose midnight deadline
+  passed during the outage is still due at once, flagged as a late decision is.
+- **`GET /ready`** says which parts of an instance work: the database and file storage (a 503
+  without them), Redis when configured and whether the cut-off sweep has run within its interval
+  (degraded without them); `/health` stays the load balancer's one-question check. The metrics
+  gain `openclear_component_up{component=…}`, and the runbook says what to alert on (BC2).
+- **Backups measured.** The host's backup job leaves a manifest (`backup_manifests/latest.json`;
+  the runbook gives its shape) and the Controls page shows the last backup, warning when there's
+  none or it's older than `OPENCLEAR_BACKUP_MAX_AGE_HOURS` (new control OC-CP-03). A SQLite
+  installation makes its own consistent copy nightly and writes the manifest itself (BC3).
+- **Restores proven and reconciled.** `scripts/restore_drill.py` restores a backup into a scratch
+  database, migrates it, verifies every bank's action-log chain and checks the restored rows
+  against file storage, then records the signed result; the newest passed drill is the last
+  restore test on every bank's Controls page. Nightly, every row that names a file and every file
+  are checked against each other (*Files and rows agree*, control OC-CP-04), and Platform →
+  Continuity shows the last backup, the drills and any row whose file is gone (BC4).
+- **Maintenance mode** (`OPENCLEAR_MAINTENANCE_MODE=1`): while an installation is being restored,
+  people can sign in and look, an administrator can take a data export, and every other change
+  answers 503 with the reason; every page carries the notice; the cut-off sweep pauses, and the
+  pause counts as an outage across a cut-off when the mode ends, so nothing is decided by
+  default for the gap. The Controls page shows the mode (BC5).
+- **Your open items, as a file for your continuity plan.** Every hour OpenClear writes each
+  bank's open and pending items (customer, account, item, amount, problems, when it's due, the
+  return deadline, the default at the cut-off) as a signed CSV, with a dated copy at the end of
+  the bank's day; Settings → Continuity downloads them, and can email the daily copy to the
+  incident contacts, so the bank can decide by hand and reach its core before the midnight
+  deadline when OpenClear can't be reached (CUEC B-CP-2-01) (BC5).
+- **The runbook's "Recovering the installation"**: the order of operations after a loss (the
+  banks told; the keys before the database; maintenance mode; the restore and the migrations;
+  storage; `/ready`; the action-log chains and the file reconciliation; maintenance mode off, and
+  what the first sweep then does; closing the incident and recording the drill) (BC6).
+
 ## 1.34.0 — 2026-10-31
 
 - **A check paid by a decision is paid.** Until now only a check that matched its issued item

RUNBOOK.md

@@ -123,13 +123,96 @@ Scrape `GET /api/v1/platform/metrics` every minute with a platform API key that
 | `openclear_decision_files_oldest_unacknowledged_seconds` | above a bank's own file schedule plus an hour | A core isn't taking its decision files. |
 | `openclear_webhook_deliveries{status="failed"}` | rises | A bank's webhook endpoint stopped answering for a day. |
 | `openclear_http_requests_total{status="5xx"}` | rising fast | Server errors; the access log has the request ids. |
+| `openclear_component_up{component=…}` | 0 for 2 minutes | That part of the instance isn't working: `database`, `storage` (a probe byte couldn't be written), `redis` (limits are being held per instance), or `scheduler` (the cut-off sweep hasn't run within its interval; check the scheduler and the leader lock). |
+
+`GET /ready` answers the same per component, as JSON: `ready`, `degraded` (Redis or the sweep;
+the instance still serves) or `unavailable` with a 503 (the database or storage). Point the
+host's monitor at it; `GET /health` stays the load balancer's one-question check (the database).
 
 Request counts are per process since it started; the rest is read from the database.
 
 A single-process SQLite installation uses write-ahead logging and waits up to five seconds for a
 write lock, so the in-process worker importing a file doesn't make web requests fail with
 "database is locked".
 
+## Backups (for the service provider)
+
+Back up the database and every storage area together, as CUEC H-CP-9-01 asks, with the host's own
+tooling (snapshots, `pg_dump`, bucket replication). When the job finishes, have it write a
+manifest to the `backup_manifests` storage area (`OPENCLEAR_BACKUP_MANIFEST_STORAGE_DIR`, or the
+`backup_manifests/` prefix of the bucket) as `latest.json`:
+
+```json
+{"completed_at": "2026-10-31T02:14:09+00:00",
+ "database": {"bytes": 734003200, "sha256": "…"},
+ "storage_areas": {"check_images": {"objects": 18240}, "bulk_uploads": {"objects": 912}, "decision_files": {"objects": 3310}},
+ "kept_at": "s3://acme-backups/openclear/2026-10-31/", "retention_days": 35,
+ "made_by": "backup.sh"}
+```
+
+The Controls page's *Backups fresh* (OC-CP-03) reads it: a warning when there's no manifest or
+the last backup is older than `OPENCLEAR_BACKUP_MAX_AGE_HOURS` (26). A SQLite installation
+needs no job: housekeeping makes a consistent copy nightly (`VACUUM INTO`) into the same area,
+keeps `OPENCLEAR_SQLITE_BACKUP_KEEP_DAYS` (14) of them, and writes the manifest itself. Keep at
+least one copy immutable or offline, behind a sign-in separate from production.
+
+Prove a restore at least yearly with the drill, against a throwaway database (never the live one):
+
+```
+scripts/restore_drill.py /backups/openclear-2026-10-31.dump --scratch postgresql+psycopg://…/openclear_drill --manifest latest.json
+scripts/restore_drill.py data/backups/sqlite/openclear-20261031T020000Z.sqlite --scratch sqlite:////tmp/drill.sqlite
+```
+
+It restores the dump (`pg_restore`, or a copy of the SQLite file), runs the migrations to head,
+verifies every bank's action-log chain, reconciles the restored rows against file storage, checks
+the dump's hash against the manifest, prints each step and writes a signed `restore_test` row to
+the live database. Platform → Continuity lists the drills; the newest passed one is the "last
+restore test" on every bank's Controls page, in place of `OPENCLEAR_LAST_RESTORE_TEST`.
+
+## Recovering the installation (for the service provider)
+
+In this order, because each step needs the one before it (CONTINUITY_PLAN.md BC-D7; rehearse it
+against the sandbox at least yearly, and record the rehearsal):
+
+1. **Tell the banks** from Platform → Incidents (an *Incident* notice: what happened, that
+   decisions already made are safe, when you'll update them), or by the incident contacts' emails
+   if OpenClear itself is down (Settings → Incident contacts lists them; the daily open-items file
+   in each bank's storage, `data_exports/<bank>/continuity/`, is what a bank works from meanwhile).
+2. **Start with the keys, not the database.** Nothing reads without them: the JWT key pair
+   (`OPENCLEAR_JWT_PRIVATE_KEY_PATH`, `_PUBLIC_`), the file-signing pair
+   (`OPENCLEAR_FILE_SIGNING_*`), the action-log signing pair (`OPENCLEAR_AUDIT_LOG_SIGNING_*`)
+   and `OPENCLEAR_SSO_ENCRYPTION_KEY` (and `_PREVIOUS` if a rotation was under way), from the key
+   store, exactly as they were. A different key verifies no signature and decrypts no SAML
+   secret.
+3. **Set `OPENCLEAR_MAINTENANCE_MODE=1`** on every instance before the database is reachable
+   again: people can sign in and look, an administrator can take a data export, and nothing
+   changes; the cut-off sweep pauses and the pause is treated as an outage across a cut-off when
+   the mode ends (step 8), so nothing is decided by default for the gap.
+4. **Restore the database** from the newest backup (`pg_restore --clean --if-exists`, or copy the
+   SQLite file into place), then **`alembic upgrade head`**. If the backup predates the running
+   release, the migrations bring it up; a backup from a newer release than the code is not
+   restorable: deploy that release first.
+5. **Restore file storage** to the same point as the database (the bucket's versioning point, or
+   the directories' copy). The two are backed up apart; step 7 finds what disagrees.
+6. **Start one instance** and check `GET /ready`: `database` and `storage` must be `ok`; `redis`
+   may be `degraded` (limits are then per instance); `scheduler` says when the sweep last ran.
+7. **Verify** before anyone decides anything: every bank's Controls page shows *Action log intact*
+   (the chain verifies against the restored rows; a break means the backup and the log disagree
+   and the bank must be told); Platform → Continuity → *Check now* reconciles rows and files and
+   lists any row whose file is gone (an upload or image written after the backup: its bank should
+   re-upload it, and the row is otherwise harmless); *Backups fresh* shows the backup you restored.
+8. **Turn maintenance mode off** and start the other instances. The first cut-off sweep treats
+   the pause as a gap: items due during it get the minimum review time again (never past a
+   check's midnight deadline), the bank's administrators and the customers' deciders are
+   emailed, and every bank's Controls page shows *No outage across a cut-off* failing for a
+   week. Decisions a core had already taken before the backup are in its files; decisions made
+   after the backup and before the outage are gone with the rows and must be made again, which
+   the gap's notice asks for.
+9. **Close the incident** with an *Update* or *Closed* notice, and record the recovery (the time
+   it took, against the stated recovery objectives) in the restore drill's place: run
+   `scripts/restore_drill.py` against the backup you used so the Controls page's last restore test
+   is this one.
+
 ## Incidents (for the service provider)
 
 The host of an installation is a bank service provider. When something disrupts the service, or

docs/compliance/controls.md

@@ -116,4 +116,7 @@ Inventory version 1.3. The same list, with each check's live status for your ban
 | Id | Control | NIST | CSF 2.0 | Type | Checks | Relies on | Where |
 |---|---|---|---|---|---|---|---|
 | OC-CP-01 | **Recovery objectives stated.** The host's recovery time and recovery point objectives and the date of its last restore test are shown on the Controls page, which warns until they're recorded or once the test is over a year old. | **CP-2** Contingency Plan<br>**CP-4** Contingency Plan Testing | ID.IM-01, ID.IM-02, ID.IM-03, ID.IM-04, PR.IR-02, PR.IR-03, RC.RP-03, RC.CO-04 | detective, built in | `recovery_objectives` (live) | H-CP-2-01, H-CP-9-01, B-CP-2-03 | Controls → Recovery objectives stated |
+| OC-CP-03 | **Backups fresh.** The host's backup job leaves a manifest when it finishes (when, what, where it's kept, for how long); the Controls page shows the last backup and warns when there's no manifest or it's older than the hours allowed. A SQLite installation makes its own nightly copy and manifest. | **CP-9** System Backup<br>**CP-10** System Recovery and Reconstitution | PR.DS-01, PR.DS-10, PR.DS-11, PR.IR-03, RC.RP-01, RC.RP-02, RC.RP-03, RC.RP-05 | detective, built in | `backup_freshness` (live) | H-CP-9-01 | Controls → Backups fresh; RUNBOOK.md, Backups |
+| OC-CP-04 | **A restore is proven and reconciled.** scripts/restore_drill.py restores a backup into a scratch database, migrates it, verifies every bank's action-log chain and reconciles the restored rows against file storage, then signs the result; the Controls page shows the last passed drill as the last restore test. Nightly, the rows that name files and the files are checked against each other, and Platform → Continuity lists any that disagree. | **CP-4** Contingency Plan Testing<br>**CP-9** System Backup<br>**CP-10** System Recovery and Reconstitution | ID.IM-02, PR.DS-01, PR.DS-10, PR.DS-11, PR.IR-03, RC.RP-01, RC.RP-02, RC.RP-03, RC.RP-05 | detective, built in | `storage_consistency` (live)<br>`maintenance_mode` (live) | H-CP-9-01, H-CP-2-01 | scripts/restore_drill.py; Controls → Files and rows agree; Platform → Continuity |
+| OC-CP-02 | **An outage across a cut-off decides nothing by default.** When OpenClear was unavailable across a cut-off, the items due in that time are given the bank's minimum review time again, never past a check's midnight deadline, instead of the customer's default being applied unseen; the bank's administrators and the customers' deciders are told, and the Controls page shows the gap for a week. | **CP-2** Contingency Plan<br>**CP-10** System Recovery and Reconstitution | ID.IM-01, ID.IM-02, ID.IM-03, ID.IM-04, PR.IR-02, PR.IR-03, RC.RP-01, RC.RP-02, RC.RP-03, RC.RP-05, RC.CO-04 | preventive, built in | `continuity_gaps` (live) | B-CP-2-01 | services/continuity_service.py; Controls → No outage across a cut-off |