diff --git a/CHANGELOG.md b/CHANGELOG.md index 65855e18..2fbc42f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,45 @@ # Changelog +## Unreleased + +- **An outage across a cut-off no longer decides by default** (`plans/CONTINUITY_PLAN.md` BC1). + When OpenClear comes back after being down longer than its sweep interval, the items whose + cut-off passed in the gap are given the bank's minimum review time again, never past a check's + midnight deadline, instead of the customer's default being applied to what nobody could look + at. The bank's administrators and the customers' deciders are emailed, the gap is in the action + log and a `continuity.gap` security event (high), and the Controls page shows it for a week + (new control OC-CP-02, check *No outage across a cut-off*). An item whose midnight deadline + passed during the outage is still due at once, flagged as a late decision is. +- **`GET /ready`** says which parts of an instance work: the database and file storage (a 503 + without them), Redis when configured and whether the cut-off sweep has run within its interval + (degraded without them); `/health` stays the load balancer's one-question check. The metrics + gain `openclear_component_up{component=…}`, and the runbook says what to alert on (BC2). +- **Backups measured.** The host's backup job leaves a manifest (`backup_manifests/latest.json`; + the runbook gives its shape) and the Controls page shows the last backup, warning when there's + none or it's older than `OPENCLEAR_BACKUP_MAX_AGE_HOURS` (new control OC-CP-03). A SQLite + installation makes its own consistent copy nightly and writes the manifest itself (BC3). +- **Restores proven and reconciled.** `scripts/restore_drill.py` restores a backup into a scratch + database, migrates it, verifies every bank's action-log chain and checks the restored rows + against file storage, then records the signed result; the newest passed drill is the last + restore test on every bank's Controls page. Nightly, every row that names a file and every file + are checked against each other (*Files and rows agree*, control OC-CP-04), and Platform → + Continuity shows the last backup, the drills and any row whose file is gone (BC4). +- **Maintenance mode** (`OPENCLEAR_MAINTENANCE_MODE=1`): while an installation is being restored, + people can sign in and look, an administrator can take a data export, and every other change + answers 503 with the reason; every page carries the notice; the cut-off sweep pauses, and the + pause counts as an outage across a cut-off when the mode ends, so nothing is decided by + default for the gap. The Controls page shows the mode (BC5). +- **Your open items, as a file for your continuity plan.** Every hour OpenClear writes each + bank's open and pending items (customer, account, item, amount, problems, when it's due, the + return deadline, the default at the cut-off) as a signed CSV, with a dated copy at the end of + the bank's day; Settings → Continuity downloads them, and can email the daily copy to the + incident contacts, so the bank can decide by hand and reach its core before the midnight + deadline when OpenClear can't be reached (CUEC B-CP-2-01) (BC5). +- **The runbook's "Recovering the installation"**: the order of operations after a loss (the + banks told; the keys before the database; maintenance mode; the restore and the migrations; + storage; `/ready`; the action-log chains and the file reconciliation; maintenance mode off, and + what the first sweep then does; closing the incident and recording the drill) (BC6). + ## 1.34.0 — 2026-10-31 - **A check paid by a decision is paid.** Until now only a check that matched its issued item diff --git a/RUNBOOK.md b/RUNBOOK.md index 1b154499..95240e5a 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -123,6 +123,11 @@ Scrape `GET /api/v1/platform/metrics` every minute with a platform API key that | `openclear_decision_files_oldest_unacknowledged_seconds` | above a bank's own file schedule plus an hour | A core isn't taking its decision files. | | `openclear_webhook_deliveries{status="failed"}` | rises | A bank's webhook endpoint stopped answering for a day. | | `openclear_http_requests_total{status="5xx"}` | rising fast | Server errors; the access log has the request ids. | +| `openclear_component_up{component=…}` | 0 for 2 minutes | That part of the instance isn't working: `database`, `storage` (a probe byte couldn't be written), `redis` (limits are being held per instance), or `scheduler` (the cut-off sweep hasn't run within its interval; check the scheduler and the leader lock). | + +`GET /ready` answers the same per component, as JSON: `ready`, `degraded` (Redis or the sweep; +the instance still serves) or `unavailable` with a 503 (the database or storage). Point the +host's monitor at it; `GET /health` stays the load balancer's one-question check (the database). Request counts are per process since it started; the rest is read from the database. @@ -130,6 +135,84 @@ A single-process SQLite installation uses write-ahead logging and waits up to fi write lock, so the in-process worker importing a file doesn't make web requests fail with "database is locked". +## Backups (for the service provider) + +Back up the database and every storage area together, as CUEC H-CP-9-01 asks, with the host's own +tooling (snapshots, `pg_dump`, bucket replication). When the job finishes, have it write a +manifest to the `backup_manifests` storage area (`OPENCLEAR_BACKUP_MANIFEST_STORAGE_DIR`, or the +`backup_manifests/` prefix of the bucket) as `latest.json`: + +```json +{"completed_at": "2026-10-31T02:14:09+00:00", + "database": {"bytes": 734003200, "sha256": "…"}, + "storage_areas": {"check_images": {"objects": 18240}, "bulk_uploads": {"objects": 912}, "decision_files": {"objects": 3310}}, + "kept_at": "s3://acme-backups/openclear/2026-10-31/", "retention_days": 35, + "made_by": "backup.sh"} +``` + +The Controls page's *Backups fresh* (OC-CP-03) reads it: a warning when there's no manifest or +the last backup is older than `OPENCLEAR_BACKUP_MAX_AGE_HOURS` (26). A SQLite installation +needs no job: housekeeping makes a consistent copy nightly (`VACUUM INTO`) into the same area, +keeps `OPENCLEAR_SQLITE_BACKUP_KEEP_DAYS` (14) of them, and writes the manifest itself. Keep at +least one copy immutable or offline, behind a sign-in separate from production. + +Prove a restore at least yearly with the drill, against a throwaway database (never the live one): + +``` +scripts/restore_drill.py /backups/openclear-2026-10-31.dump --scratch postgresql+psycopg://…/openclear_drill --manifest latest.json +scripts/restore_drill.py data/backups/sqlite/openclear-20261031T020000Z.sqlite --scratch sqlite:////tmp/drill.sqlite +``` + +It restores the dump (`pg_restore`, or a copy of the SQLite file), runs the migrations to head, +verifies every bank's action-log chain, reconciles the restored rows against file storage, checks +the dump's hash against the manifest, prints each step and writes a signed `restore_test` row to +the live database. Platform → Continuity lists the drills; the newest passed one is the "last +restore test" on every bank's Controls page, in place of `OPENCLEAR_LAST_RESTORE_TEST`. + +## Recovering the installation (for the service provider) + +In this order, because each step needs the one before it (CONTINUITY_PLAN.md BC-D7; rehearse it +against the sandbox at least yearly, and record the rehearsal): + +1. **Tell the banks** from Platform → Incidents (an *Incident* notice: what happened, that + decisions already made are safe, when you'll update them), or by the incident contacts' emails + if OpenClear itself is down (Settings → Incident contacts lists them; the daily open-items file + in each bank's storage, `data_exports//continuity/`, is what a bank works from meanwhile). +2. **Start with the keys, not the database.** Nothing reads without them: the JWT key pair + (`OPENCLEAR_JWT_PRIVATE_KEY_PATH`, `_PUBLIC_`), the file-signing pair + (`OPENCLEAR_FILE_SIGNING_*`), the action-log signing pair (`OPENCLEAR_AUDIT_LOG_SIGNING_*`) + and `OPENCLEAR_SSO_ENCRYPTION_KEY` (and `_PREVIOUS` if a rotation was under way), from the key + store, exactly as they were. A different key verifies no signature and decrypts no SAML + secret. +3. **Set `OPENCLEAR_MAINTENANCE_MODE=1`** on every instance before the database is reachable + again: people can sign in and look, an administrator can take a data export, and nothing + changes; the cut-off sweep pauses and the pause is treated as an outage across a cut-off when + the mode ends (step 8), so nothing is decided by default for the gap. +4. **Restore the database** from the newest backup (`pg_restore --clean --if-exists`, or copy the + SQLite file into place), then **`alembic upgrade head`**. If the backup predates the running + release, the migrations bring it up; a backup from a newer release than the code is not + restorable: deploy that release first. +5. **Restore file storage** to the same point as the database (the bucket's versioning point, or + the directories' copy). The two are backed up apart; step 7 finds what disagrees. +6. **Start one instance** and check `GET /ready`: `database` and `storage` must be `ok`; `redis` + may be `degraded` (limits are then per instance); `scheduler` says when the sweep last ran. +7. **Verify** before anyone decides anything: every bank's Controls page shows *Action log intact* + (the chain verifies against the restored rows; a break means the backup and the log disagree + and the bank must be told); Platform → Continuity → *Check now* reconciles rows and files and + lists any row whose file is gone (an upload or image written after the backup: its bank should + re-upload it, and the row is otherwise harmless); *Backups fresh* shows the backup you restored. +8. **Turn maintenance mode off** and start the other instances. The first cut-off sweep treats + the pause as a gap: items due during it get the minimum review time again (never past a + check's midnight deadline), the bank's administrators and the customers' deciders are + emailed, and every bank's Controls page shows *No outage across a cut-off* failing for a + week. Decisions a core had already taken before the backup are in its files; decisions made + after the backup and before the outage are gone with the rows and must be made again, which + the gap's notice asks for. +9. **Close the incident** with an *Update* or *Closed* notice, and record the recovery (the time + it took, against the stated recovery objectives) in the restore drill's place: run + `scripts/restore_drill.py` against the backup you used so the Controls page's last restore test + is this one. + ## Incidents (for the service provider) The host of an installation is a bank service provider. When something disrupts the service, or diff --git a/docs/compliance/controls.md b/docs/compliance/controls.md index f509529c..0bb740e3 100644 --- a/docs/compliance/controls.md +++ b/docs/compliance/controls.md @@ -116,4 +116,7 @@ Inventory version 1.3. The same list, with each check's live status for your ban | Id | Control | NIST | CSF 2.0 | Type | Checks | Relies on | Where | |---|---|---|---|---|---|---|---| | OC-CP-01 | **Recovery objectives stated.** The host's recovery time and recovery point objectives and the date of its last restore test are shown on the Controls page, which warns until they're recorded or once the test is over a year old. | **CP-2** Contingency Plan
**CP-4** Contingency Plan Testing | ID.IM-01, ID.IM-02, ID.IM-03, ID.IM-04, PR.IR-02, PR.IR-03, RC.RP-03, RC.CO-04 | detective, built in | `recovery_objectives` (live) | H-CP-2-01, H-CP-9-01, B-CP-2-03 | Controls → Recovery objectives stated | +| OC-CP-03 | **Backups fresh.** The host's backup job leaves a manifest when it finishes (when, what, where it's kept, for how long); the Controls page shows the last backup and warns when there's no manifest or it's older than the hours allowed. A SQLite installation makes its own nightly copy and manifest. | **CP-9** System Backup
**CP-10** System Recovery and Reconstitution | PR.DS-01, PR.DS-10, PR.DS-11, PR.IR-03, RC.RP-01, RC.RP-02, RC.RP-03, RC.RP-05 | detective, built in | `backup_freshness` (live) | H-CP-9-01 | Controls → Backups fresh; RUNBOOK.md, Backups | +| OC-CP-04 | **A restore is proven and reconciled.** scripts/restore_drill.py restores a backup into a scratch database, migrates it, verifies every bank's action-log chain and reconciles the restored rows against file storage, then signs the result; the Controls page shows the last passed drill as the last restore test. Nightly, the rows that name files and the files are checked against each other, and Platform → Continuity lists any that disagree. | **CP-4** Contingency Plan Testing
**CP-9** System Backup
**CP-10** System Recovery and Reconstitution | ID.IM-02, PR.DS-01, PR.DS-10, PR.DS-11, PR.IR-03, RC.RP-01, RC.RP-02, RC.RP-03, RC.RP-05 | detective, built in | `storage_consistency` (live)
`maintenance_mode` (live) | H-CP-9-01, H-CP-2-01 | scripts/restore_drill.py; Controls → Files and rows agree; Platform → Continuity | +| OC-CP-02 | **An outage across a cut-off decides nothing by default.** When OpenClear was unavailable across a cut-off, the items due in that time are given the bank's minimum review time again, never past a check's midnight deadline, instead of the customer's default being applied unseen; the bank's administrators and the customers' deciders are told, and the Controls page shows the gap for a week. | **CP-2** Contingency Plan
**CP-10** System Recovery and Reconstitution | ID.IM-01, ID.IM-02, ID.IM-03, ID.IM-04, PR.IR-02, PR.IR-03, RC.RP-01, RC.RP-02, RC.RP-03, RC.RP-05, RC.CO-04 | preventive, built in | `continuity_gaps` (live) | B-CP-2-01 | services/continuity_service.py; Controls → No outage across a cut-off | diff --git a/docs/compliance/csf.md b/docs/compliance/csf.md index 81319212..d53014d6 100644 --- a/docs/compliance/csf.md +++ b/docs/compliance/csf.md @@ -71,10 +71,10 @@ NIST's mapping links no subcategory to `MP-6`, `AU-10`, `MA-4`, `SC-18`, `SC-23` | **ID.RA-08** Processes for receiving, analyzing, and responding to vulnerability disclosures are established | — | H-RA-5-01 | | **ID.RA-09** The authenticity and integrity of hardware and software are assessed prior to acquisition and use | OC-AU-02, OC-SC-03, OC-SI-02 | B-SI-7-01, B-SA-4-01, B-SA-11-01, C-SI-7-01 | | **ID.RA-10** Critical suppliers are assessed prior to acquisition | — | — | -| **ID.IM-01** Improvements are identified from evaluations | OC-AU-04, OC-SI-03, OC-SI-04, OC-SI-05, OC-SI-09, OC-SI-10, OC-SI-11, OC-CM-02, OC-CP-01, OC-IR-01, OC-CA-01, OC-CA-02, OC-CA-03 | B-PL-1-01, B-CA-2-01, B-RA-3-02, B-RA-3-01, B-RA-3-03, B-RA-3-04, B-RA-3-05, B-CA-7-01, B-CA-7-02, B-SI-4-01, B-SI-4-02, B-SI-4-03, B-CP-2-02, B-CP-2-01, B-CP-2-03, B-SA-11-01, C-SI-4-02, C-SI-4-01, H-CP-2-01, H-SI-2-01, H-RA-5-01, H-SI-4-01, H-IR-8-01, H-CA-8-01 | -| **ID.IM-02** Improvements are identified from security tests and exercises, including those done in coordination with suppliers and relevant third parties | OC-AU-04, OC-SI-03, OC-SI-04, OC-SI-05, OC-SI-09, OC-SI-10, OC-SI-11, OC-CM-02, OC-CP-01, OC-IR-01, OC-CA-01, OC-CA-02, OC-CA-03 | B-PL-1-01, B-CA-2-01, B-RA-3-02, B-RA-3-01, B-RA-3-03, B-RA-3-04, B-RA-3-05, B-CA-7-01, B-CA-7-02, B-SI-4-01, B-SI-4-02, B-SI-4-03, B-CP-2-02, B-CP-2-01, B-CP-2-03, B-SA-11-01, C-SI-4-02, C-SI-4-01, H-CP-2-01, H-SI-2-01, H-RA-5-01, H-SI-4-01, H-IR-8-01, H-CA-8-01 | -| **ID.IM-03** Improvements are identified from execution of operational processes, procedures, and activities | OC-AU-04, OC-SI-03, OC-SI-04, OC-SI-05, OC-SI-09, OC-SI-10, OC-SI-11, OC-CM-02, OC-CP-01, OC-IR-01, OC-CA-01, OC-CA-02, OC-CA-03 | B-PL-1-01, B-CA-2-01, B-RA-3-02, B-RA-3-01, B-RA-3-03, B-RA-3-04, B-RA-3-05, B-CA-7-01, B-CA-7-02, B-SI-4-01, B-SI-4-02, B-SI-4-03, B-CP-2-02, B-CP-2-01, B-CP-2-03, B-SA-4-01, B-SA-11-01, C-SI-4-02, C-SI-4-01, H-CP-2-01, H-SI-2-01, H-RA-5-01, H-SI-4-01, H-IR-8-01, H-CA-8-01 | -| **ID.IM-04** Incident response plans and other cybersecurity plans that affect operations are established, communicated, maintained, and improved | OC-SI-05, OC-CP-01 | B-CP-2-02, B-CP-2-01, B-CP-2-03, H-CP-2-01, H-IR-8-01 | +| **ID.IM-01** Improvements are identified from evaluations | OC-AU-04, OC-SI-03, OC-SI-04, OC-SI-05, OC-SI-09, OC-SI-10, OC-SI-11, OC-CM-02, OC-CP-01, OC-CP-02, OC-IR-01, OC-CA-01, OC-CA-02, OC-CA-03 | B-PL-1-01, B-CA-2-01, B-RA-3-02, B-RA-3-01, B-RA-3-03, B-RA-3-04, B-RA-3-05, B-CA-7-01, B-CA-7-02, B-SI-4-01, B-SI-4-02, B-SI-4-03, B-CP-2-02, B-CP-2-01, B-CP-2-03, B-SA-11-01, C-SI-4-02, C-SI-4-01, H-CP-2-01, H-SI-2-01, H-RA-5-01, H-SI-4-01, H-IR-8-01, H-CA-8-01 | +| **ID.IM-02** Improvements are identified from security tests and exercises, including those done in coordination with suppliers and relevant third parties | OC-AU-04, OC-SI-03, OC-SI-04, OC-SI-05, OC-SI-09, OC-SI-10, OC-SI-11, OC-CM-02, OC-CP-01, OC-CP-04, OC-CP-02, OC-IR-01, OC-CA-01, OC-CA-02, OC-CA-03 | B-PL-1-01, B-CA-2-01, B-RA-3-02, B-RA-3-01, B-RA-3-03, B-RA-3-04, B-RA-3-05, B-CA-7-01, B-CA-7-02, B-SI-4-01, B-SI-4-02, B-SI-4-03, B-CP-2-02, B-CP-2-01, B-CP-2-03, B-SA-11-01, C-SI-4-02, C-SI-4-01, H-CP-2-01, H-SI-2-01, H-RA-5-01, H-SI-4-01, H-IR-8-01, H-CA-8-01 | +| **ID.IM-03** Improvements are identified from execution of operational processes, procedures, and activities | OC-AU-04, OC-SI-03, OC-SI-04, OC-SI-05, OC-SI-09, OC-SI-10, OC-SI-11, OC-CM-02, OC-CP-01, OC-CP-02, OC-IR-01, OC-CA-01, OC-CA-02, OC-CA-03 | B-PL-1-01, B-CA-2-01, B-RA-3-02, B-RA-3-01, B-RA-3-03, B-RA-3-04, B-RA-3-05, B-CA-7-01, B-CA-7-02, B-SI-4-01, B-SI-4-02, B-SI-4-03, B-CP-2-02, B-CP-2-01, B-CP-2-03, B-SA-4-01, B-SA-11-01, C-SI-4-02, C-SI-4-01, H-CP-2-01, H-SI-2-01, H-RA-5-01, H-SI-4-01, H-IR-8-01, H-CA-8-01 | +| **ID.IM-04** Incident response plans and other cybersecurity plans that affect operations are established, communicated, maintained, and improved | OC-SI-05, OC-CP-01, OC-CP-02 | B-CP-2-02, B-CP-2-01, B-CP-2-03, H-CP-2-01, H-IR-8-01 | ## Protect (PR) @@ -88,10 +88,10 @@ NIST's mapping links no subcategory to `MP-6`, `AU-10`, `MA-4`, `SC-18`, `SC-23` | **PR.AA-06** Physical access to assets is managed, monitored, and enforced commensurate with risk | — | — | | **PR.AT-01** Personnel are provided with awareness and training so that they possess the knowledge and skills to perform general tasks with cybersecurity risks in mind | — | B-AT-3-01, C-AT-2(3)-01, H-AT-2-01 | | **PR.AT-02** Individuals in specialized roles are provided with awareness and training so that they possess the knowledge and skills to perform relevant tasks with cybersecurity risks in mind | — | B-AT-3-01 | -| **PR.DS-01** The confidentiality, integrity, and availability of data-at-rest are protected | OC-AU-02, OC-AU-04, OC-SC-01, OC-SC-02, OC-SC-03, OC-SC-05, OC-SC-07, OC-SI-02, OC-SI-03, OC-SI-05, OC-SI-06, OC-SI-08, OC-SI-09, OC-SI-10, OC-SI-11, OC-CA-02 | B-SC-28-01, B-SI-4-01, B-SI-4-02, B-SI-3-01, B-CA-3-03, B-CA-3-04, B-SI-4-03, B-CA-3-01, B-SI-7-01, B-CA-3-02, C-SI-7-01, C-SI-4-02, C-SI-4-01, C-SC-28-01, H-SC-28(1)-01, H-SC-12-01, H-SC-7-01, H-CP-9-01, H-SI-4-01 | +| **PR.DS-01** The confidentiality, integrity, and availability of data-at-rest are protected | OC-AU-02, OC-AU-04, OC-SC-01, OC-SC-02, OC-SC-03, OC-SC-05, OC-SC-07, OC-SI-02, OC-SI-03, OC-SI-05, OC-SI-06, OC-SI-08, OC-SI-09, OC-SI-10, OC-SI-11, OC-CP-03, OC-CP-04, OC-CA-02 | B-SC-28-01, B-SI-4-01, B-SI-4-02, B-SI-3-01, B-CA-3-03, B-CA-3-04, B-SI-4-03, B-CA-3-01, B-SI-7-01, B-CA-3-02, C-SI-7-01, C-SI-4-02, C-SI-4-01, C-SC-28-01, H-SC-28(1)-01, H-SC-12-01, H-SC-7-01, H-CP-9-01, H-SI-4-01 | | **PR.DS-02** The confidentiality, integrity, and availability of data-in-transit are protected | OC-AU-02, OC-AU-04, OC-SC-01, OC-SC-03, OC-SC-04, OC-SC-05, OC-SC-07, OC-SI-02, OC-SI-03, OC-SI-05, OC-SI-06, OC-SI-08, OC-SI-09, OC-SI-10, OC-SI-11, OC-CA-02 | B-SI-4-01, B-SI-4-02, B-SI-3-01, B-CA-3-03, B-CA-3-04, B-SI-4-03, B-CA-3-01, B-SI-7-01, B-CA-3-02, B-SC-8-01, C-SI-7-01, C-SI-4-02, C-SI-4-01, H-SC-12-01, H-SC-8(1)-01, H-SC-7-01, H-SI-4-01 | -| **PR.DS-10** The confidentiality, integrity, and availability of data-in-use are protected | OC-AC-01, OC-AC-02, OC-AC-03, OC-AC-04, OC-AC-06, OC-AC-07, OC-AC-08, OC-AC-09, OC-AC-12, OC-AC-13, OC-IA-03, OC-AU-01, OC-AU-02, OC-AU-03, OC-AU-04, OC-SC-01, OC-SC-03, OC-SC-05, OC-SC-07, OC-SI-01, OC-SI-02, OC-SI-03, OC-SI-05, OC-SI-06, OC-SI-08, OC-SI-09, OC-SI-10, OC-SI-11, OC-SI-13, OC-SI-14, OC-CA-02 | B-AC-2-01, B-AC-2(1)-01, B-AC-3(2)-01, B-AC-2-02, B-SI-4-01, B-SI-4-02, B-SI-3-01, B-CA-3-03, B-CA-3-04, B-SI-4-03, B-CA-3-01, B-SI-7-01, B-CA-3-02, C-AC-2-01, C-AC-2-02, C-AC-2(1)-01, C-SI-7-01, C-SI-10-01, C-SI-4-02, C-SI-4-01, C-AC-3-01, H-SC-7-01, H-CP-9-01, H-AU-9(2)-01, H-SI-4-01 | -| **PR.DS-11** Backups of data are created, protected, maintained, and tested | — | H-CP-9-01 | +| **PR.DS-10** The confidentiality, integrity, and availability of data-in-use are protected | OC-AC-01, OC-AC-02, OC-AC-03, OC-AC-04, OC-AC-06, OC-AC-07, OC-AC-08, OC-AC-09, OC-AC-12, OC-AC-13, OC-IA-03, OC-AU-01, OC-AU-02, OC-AU-03, OC-AU-04, OC-SC-01, OC-SC-03, OC-SC-05, OC-SC-07, OC-SI-01, OC-SI-02, OC-SI-03, OC-SI-05, OC-SI-06, OC-SI-08, OC-SI-09, OC-SI-10, OC-SI-11, OC-SI-13, OC-SI-14, OC-CP-03, OC-CP-04, OC-CA-02 | B-AC-2-01, B-AC-2(1)-01, B-AC-3(2)-01, B-AC-2-02, B-SI-4-01, B-SI-4-02, B-SI-3-01, B-CA-3-03, B-CA-3-04, B-SI-4-03, B-CA-3-01, B-SI-7-01, B-CA-3-02, C-AC-2-01, C-AC-2-02, C-AC-2(1)-01, C-SI-7-01, C-SI-10-01, C-SI-4-02, C-SI-4-01, C-AC-3-01, H-SC-7-01, H-CP-9-01, H-AU-9(2)-01, H-SI-4-01 | +| **PR.DS-11** Backups of data are created, protected, maintained, and tested | OC-CP-03, OC-CP-04 | H-CP-9-01 | | **PR.PS-01** Configuration management practices are established and applied | OC-AC-07, OC-AC-13, OC-SI-04, OC-CM-01, OC-CM-02, OC-CM-03 | B-CM-3-01, B-CM-4-01, B-CM-8-01, C-CM-4-01, H-CM-6-01, H-CM-4-01 | | **PR.PS-02** Software is maintained, replaced, and removed commensurate with risk | OC-AU-02, OC-SC-03, OC-SI-02, OC-CM-02 | B-SI-7-01, C-SI-7-01, H-SI-2-01 | | **PR.PS-03** Hardware is maintained, replaced, and removed commensurate with risk | — | — | @@ -99,8 +99,8 @@ NIST's mapping links no subcategory to `MP-6`, `AU-10`, `MA-4`, `SC-18`, `SC-23` | **PR.PS-05** Installation and execution of unauthorized software are prevented | — | — | | **PR.PS-06** Secure software development practices are integrated, and their performance is monitored throughout the software development life cycle | — | B-SA-11-01 | | **PR.IR-01** Networks and environments are protected from unauthorized logical access and usage | OC-AC-01, OC-AC-03, OC-AC-04, OC-AC-06, OC-AC-07, OC-AC-08, OC-AC-09, OC-AC-12, OC-AC-13, OC-SC-08, OC-SC-05, OC-SC-06, OC-SC-07, OC-SI-08 | B-AC-3(2)-01, C-AC-3-01, H-SC-7-01 | -| **PR.IR-02** The organization's technology assets are protected from environmental threats | OC-SI-05, OC-CP-01 | B-CP-2-02, B-CP-2-01, B-CP-2-03, H-CP-2-01 | -| **PR.IR-03** Mechanisms are implemented to achieve resilience requirements in normal and adverse situations | OC-SI-05, OC-SI-07, OC-CM-02, OC-CP-01, OC-IR-01, OC-IR-02 | B-IR-6-03, B-IR-6-04, B-IR-6-05, B-IR-6-06, B-IR-6-07, B-IR-6-08, B-IR-6-02, B-CP-2-02, B-CP-2-01, B-CP-2-03, B-IR-6-01, C-IR-6-01, H-CP-9-01, H-CP-2-01, H-IR-6-01, H-IR-6-02, H-IR-8-01 | +| **PR.IR-02** The organization's technology assets are protected from environmental threats | OC-SI-05, OC-CP-01, OC-CP-02 | B-CP-2-02, B-CP-2-01, B-CP-2-03, H-CP-2-01 | +| **PR.IR-03** Mechanisms are implemented to achieve resilience requirements in normal and adverse situations | OC-SI-05, OC-SI-07, OC-CM-02, OC-CP-01, OC-CP-03, OC-CP-04, OC-CP-02, OC-IR-01, OC-IR-02 | B-IR-6-03, B-IR-6-04, B-IR-6-05, B-IR-6-06, B-IR-6-07, B-IR-6-08, B-IR-6-02, B-CP-2-02, B-CP-2-01, B-CP-2-03, B-IR-6-01, C-IR-6-01, H-CP-9-01, H-CP-2-01, H-IR-6-01, H-IR-6-02, H-IR-8-01 | | **PR.IR-04** Adequate resource capacity to ensure availability is maintained | — | — | ## Detect (DE) @@ -141,11 +141,11 @@ NIST's mapping links no subcategory to `MP-6`, `AU-10`, `MA-4`, `SC-18`, `SC-23` | Subcategory | OpenClear controls | CUECs | |---|---|---| -| **RC.RP-01** The recovery portion of the incident response plan is executed once initiated from the incident response process | OC-CM-02, OC-IR-01 | H-IR-8-01 | -| **RC.RP-02** Recovery actions are selected, scoped, prioritized, and performed | OC-CM-02, OC-IR-01 | H-IR-8-01 | -| **RC.RP-03** The integrity of backups and other restoration assets is verified before using them for restoration | OC-SI-05, OC-CP-01 | B-CP-2-02, B-CP-2-01, B-CP-2-03, H-CP-9-01, H-CP-2-01 | +| **RC.RP-01** The recovery portion of the incident response plan is executed once initiated from the incident response process | OC-CM-02, OC-CP-03, OC-CP-04, OC-CP-02, OC-IR-01 | H-IR-8-01 | +| **RC.RP-02** Recovery actions are selected, scoped, prioritized, and performed | OC-CM-02, OC-CP-03, OC-CP-04, OC-CP-02, OC-IR-01 | H-IR-8-01 | +| **RC.RP-03** The integrity of backups and other restoration assets is verified before using them for restoration | OC-SI-05, OC-CP-01, OC-CP-03, OC-CP-04, OC-CP-02 | B-CP-2-02, B-CP-2-01, B-CP-2-03, H-CP-9-01, H-CP-2-01 | | **RC.RP-04** Critical mission functions and cybersecurity risk management are considered to establish post-incident operational norms | — | H-IR-8-01 | -| **RC.RP-05** The integrity of restored assets is verified, systems and services are restored, and normal operating status is confirmed | OC-CM-02 | — | +| **RC.RP-05** The integrity of restored assets is verified, systems and services are restored, and normal operating status is confirmed | OC-CM-02, OC-CP-03, OC-CP-04, OC-CP-02 | — | | **RC.RP-06** The end of incident recovery is declared based on criteria, and incident-related documentation is completed | OC-IR-01 | H-IR-8-01 | | **RC.CO-03** Recovery activities and progress in restoring operational capabilities are communicated to designated internal and external stakeholders | OC-SI-07, OC-IR-01, OC-IR-02 | B-IR-6-03, B-IR-6-04, B-IR-6-05, B-IR-6-06, B-IR-6-07, B-IR-6-08, B-IR-6-02, B-IR-6-01, C-IR-6-01, H-IR-6-01, H-IR-6-02 | -| **RC.CO-04** Public updates on incident recovery are shared using approved methods and messaging | OC-SI-05, OC-CP-01, OC-IR-01 | B-CP-2-02, B-CP-2-01, B-CP-2-03, H-CP-2-01 | +| **RC.CO-04** Public updates on incident recovery are shared using approved methods and messaging | OC-SI-05, OC-CP-01, OC-CP-02, OC-IR-01 | B-CP-2-02, B-CP-2-01, B-CP-2-03, H-CP-2-01 | diff --git a/docs/compliance/cuecs.csv b/docs/compliance/cuecs.csv index 089f2031..75ce79ae 100644 --- a/docs/compliance/cuecs.csv +++ b/docs/compliance/cuecs.csv @@ -82,7 +82,7 @@ H-SC-12-01,host,SC-12,Cryptographic Key Establishment and Management,sc-12,SC-13 H-SC-8(1)-01,host,SC-8(1),Cryptographic Protection,sc-8.1,SC-8; SC-23,Serve only over HTTPS,"Terminate HTTPS with a current certificate and modern TLS in front of OpenClear, and never serve it over plain HTTP.","OpenClear sets secure, SameSite cookies and a strict Content Security Policy.",automated,transport_security,live,Continuous,PR.DS-02,scripts/edge_check.py,, H-SC-7-01,host,SC-7,Boundary Protection,sc-7,SC-5,Lock down the origin,"Let only the edge network reach OpenClear, keep the database and storage off the internet, and record how the origin is locked down.","OpenClear checks edge headers come from trusted ranges, and accepts an edge secret.",automated,edge,live,Continuous,PR.DS-01; PR.DS-02; PR.DS-10; PR.IR-01; DE.CM-01,Controls → Edge network set up,, H-CM-6-01,host,CM-6,Configuration Settings,cm-6,CM-2,Run with production settings,"Run production with production settings: real keys, no development defaults, and the scheduled control tests on.",OpenClear checks its own settings at startup and on the Controls page.,automated,production_safety,live,Continuous,PR.PS-01; DE.CM-09,Controls → Production settings,, -H-CP-9-01,host,CP-9,System Backup,cp-9,CP-4; CP-10,Back up and test restores,"Back up the database and file storage, encrypted and kept apart from the primary, with at least one copy immutable or offline behind sign-in separate from production; state how long backups are kept; and test a clean restore at least yearly.","OpenClear's database migrations upgrade and downgrade, and its action log verifies after a restore.",attested,,,Daily; restore tested yearly,PR.DS-01; PR.DS-10; PR.DS-11; PR.IR-03; RC.RP-03,The host's backup records,, +H-CP-9-01,host,CP-9,System Backup,cp-9,CP-4; CP-10,Back up and test restores,"Back up the database and file storage, encrypted and kept apart from the primary, with at least one copy immutable or offline behind sign-in separate from production; state how long backups are kept; and test a clean restore at least yearly.","OpenClear's database migrations upgrade and downgrade, its action log verifies after a restore, and scripts/restore_drill.py does both against a scratch database and signs the result; the Controls page reads the host's backup manifest (Backups fresh) and the last passed drill (Recovery objectives stated).",attested,,,Daily; restore tested yearly,PR.DS-01; PR.DS-10; PR.DS-11; PR.IR-03; RC.RP-03,The host's backup records; Controls → Backups fresh; the restore drill,, H-CP-2-01,host,CP-2,Contingency Plan,cp-2,CP-4,State recovery objectives,"State the installation's recovery time and recovery point objectives, record them and each restore test's date in OpenClear's settings, and give the banks the restore tests' results.","OpenClear shows them on every bank's Controls page, and warns when they're missing or the test is over a year old.",automated,recovery_objectives,live,Yearly,ID.IM-01; ID.IM-02; ID.IM-03; ID.IM-04; PR.IR-02; PR.IR-03; RC.RP-03; RC.CO-04,Controls → Recovery objectives stated,, H-SI-2-01,host,SI-2,Flaw Remediation,si-2,RA-5,Apply updates,"Apply OpenClear releases, and operating system, database and dependency patches, within the host's patching deadlines.","OpenClear publishes versioned releases with a changelog, and shows the running version.",assisted,software_currency,live,Monthly,ID.IM-01; ID.IM-02; ID.IM-03; PR.PS-02,/openapi.json version; CHANGELOG.md,, H-RA-5-01,host,RA-5,Vulnerability Monitoring and Scanning,ra-5,SI-2,Scan for vulnerabilities,"Scan the servers, container images and database for vulnerabilities, and fix what's found within the host's deadlines.",OpenClear's source and dependencies are public for anyone to scan.,attested,,,Monthly,GV.SC-10; ID.RA-01; ID.RA-08; ID.IM-01; ID.IM-02; ID.IM-03,The host's scan records,, diff --git a/docs/compliance/cuecs.json b/docs/compliance/cuecs.json index 40924d84..36e4691e 100644 --- a/docs/compliance/cuecs.json +++ b/docs/compliance/cuecs.json @@ -3641,7 +3641,7 @@ ], "title": "Back up and test restores", "statement": "Back up the database and file storage, encrypted and kept apart from the primary, with at least one copy immutable or offline behind sign-in separate from production; state how long backups are kept; and test a clean restore at least yearly.", - "complements": "OpenClear's database migrations upgrade and downgrade, and its action log verifies after a restore.", + "complements": "OpenClear's database migrations upgrade and downgrade, its action log verifies after a restore, and scripts/restore_drill.py does both against a scratch database and signs the result; the Controls page reads the host's backup manifest (Backups fresh) and the last passed drill (Recovery objectives stated).", "evidence": "attested", "check": null, "check_status": null, @@ -3668,7 +3668,7 @@ "title": "The integrity of backups and other restoration assets is verified before using them for restoration" } ], - "where": "The host's backup records", + "where": "The host's backup records; Controls \u2192 Backups fresh; the restore drill", "withdrawn": false }, { diff --git a/docs/compliance/cuecs.md b/docs/compliance/cuecs.md index c9b12107..00ea160f 100644 --- a/docs/compliance/cuecs.md +++ b/docs/compliance/cuecs.md @@ -123,7 +123,7 @@ and owner. | H-SC-8(1)-01 | **SC-8(1)** Cryptographic Protection
Related: SC-8, SC-23 | PR.DS-02 | **Serve only over HTTPS.** Terminate HTTPS with a current certificate and modern TLS in front of OpenClear, and never serve it over plain HTTP. | OpenClear sets secure, SameSite cookies and a strict Content Security Policy. | automated | `transport_security` (live) | Continuous | scripts/edge_check.py | | H-SC-7-01 | **SC-7** Boundary Protection
Related: SC-5 | PR.DS-01, PR.DS-02, PR.DS-10, PR.IR-01, DE.CM-01 | **Lock down the origin.** Let only the edge network reach OpenClear, keep the database and storage off the internet, and record how the origin is locked down. | OpenClear checks edge headers come from trusted ranges, and accepts an edge secret. | automated | `edge` (live) | Continuous | Controls → Edge network set up | | H-CM-6-01 | **CM-6** Configuration Settings
Related: CM-2 | PR.PS-01, DE.CM-09 | **Run with production settings.** Run production with production settings: real keys, no development defaults, and the scheduled control tests on. | OpenClear checks its own settings at startup and on the Controls page. | automated | `production_safety` (live) | Continuous | Controls → Production settings | -| H-CP-9-01 | **CP-9** System Backup
Related: CP-4, CP-10 | PR.DS-01, PR.DS-10, PR.DS-11, PR.IR-03, RC.RP-03 | **Back up and test restores.** Back up the database and file storage, encrypted and kept apart from the primary, with at least one copy immutable or offline behind sign-in separate from production; state how long backups are kept; and test a clean restore at least yearly. | OpenClear's database migrations upgrade and downgrade, and its action log verifies after a restore. | attested | — | Daily; restore tested yearly | The host's backup records | +| H-CP-9-01 | **CP-9** System Backup
Related: CP-4, CP-10 | PR.DS-01, PR.DS-10, PR.DS-11, PR.IR-03, RC.RP-03 | **Back up and test restores.** Back up the database and file storage, encrypted and kept apart from the primary, with at least one copy immutable or offline behind sign-in separate from production; state how long backups are kept; and test a clean restore at least yearly. | OpenClear's database migrations upgrade and downgrade, its action log verifies after a restore, and scripts/restore_drill.py does both against a scratch database and signs the result; the Controls page reads the host's backup manifest (Backups fresh) and the last passed drill (Recovery objectives stated). | attested | — | Daily; restore tested yearly | The host's backup records; Controls → Backups fresh; the restore drill | | H-CP-2-01 | **CP-2** Contingency Plan
Related: CP-4 | ID.IM-01, ID.IM-02, ID.IM-03, ID.IM-04, PR.IR-02, PR.IR-03, RC.RP-03, RC.CO-04 | **State recovery objectives.** State the installation's recovery time and recovery point objectives, record them and each restore test's date in OpenClear's settings, and give the banks the restore tests' results. | OpenClear shows them on every bank's Controls page, and warns when they're missing or the test is over a year old. | automated | `recovery_objectives` (live) | Yearly | Controls → Recovery objectives stated | | H-SI-2-01 | **SI-2** Flaw Remediation
Related: RA-5 | ID.IM-01, ID.IM-02, ID.IM-03, PR.PS-02 | **Apply updates.** Apply OpenClear releases, and operating system, database and dependency patches, within the host's patching deadlines. | OpenClear publishes versioned releases with a changelog, and shows the running version. | assisted | `software_currency` (live) | Monthly | /openapi.json version; CHANGELOG.md | | H-RA-5-01 | **RA-5** Vulnerability Monitoring and Scanning
Related: SI-2 | GV.SC-10, ID.RA-01, ID.RA-08, ID.IM-01, ID.IM-02, ID.IM-03 | **Scan for vulnerabilities.** Scan the servers, container images and database for vulnerabilities, and fix what's found within the host's deadlines. | OpenClear's source and dependencies are public for anyone to scan. | attested | — | Monthly | The host's scan records | diff --git a/migrations/versions/d4e8f2a6c1b7_deadline_moved_reason.py b/migrations/versions/d4e8f2a6c1b7_deadline_moved_reason.py new file mode 100644 index 00000000..48683c9b --- /dev/null +++ b/migrations/versions/d4e8f2a6c1b7_deadline_moved_reason.py @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""exception_item.deadline_moved_reason (CONTINUITY_PLAN.md BC-D1) + +Why an item's decision deadline was moved after it was set: "outage", when OpenClear was down +across its cut-off and the sweep gave it the bank's minimum review time again. + +Revision ID: d4e8f2a6c1b7 +Revises: c9d5e3f7a2b4 +Create Date: 2026-10-31 +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +revision: str = 'd4e8f2a6c1b7' +down_revision: Union[str, None] = 'c9d5e3f7a2b4' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + with op.batch_alter_table('exception_item') as batch: + batch.add_column(sa.Column('deadline_moved_reason', sa.String(length=30), nullable=True)) + + +def downgrade() -> None: + with op.batch_alter_table('exception_item') as batch: + batch.drop_column('deadline_moved_reason') diff --git a/migrations/versions/e5f9a3b7d2c8_restore_test.py b/migrations/versions/e5f9a3b7d2c8_restore_test.py new file mode 100644 index 00000000..4473c963 --- /dev/null +++ b/migrations/versions/e5f9a3b7d2c8_restore_test.py @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""restore_test: a restore drill's signed result (CONTINUITY_PLAN.md BC-D2) + +No tenant_id: one row per drill, for the whole installation, so no row-level security policy. + +Revision ID: e5f9a3b7d2c8 +Revises: d4e8f2a6c1b7 +Create Date: 2026-10-31 +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +revision: str = 'e5f9a3b7d2c8' +down_revision: Union[str, None] = 'd4e8f2a6c1b7' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.create_table( + 'restore_test', + sa.Column('id', sa.Uuid(), nullable=False), + sa.Column('run_at', sa.DateTime(timezone=True), server_default=sa.text('(CURRENT_TIMESTAMP)'), nullable=False), + sa.Column('ok', sa.Boolean(), nullable=False), + sa.Column('backup_completed_at', sa.DateTime(timezone=True), nullable=True), + sa.Column('source', sa.String(length=1000), nullable=False), + sa.Column('detail', sa.JSON(), nullable=False), + sa.Column('certificate', sa.JSON(), nullable=True), + sa.Column('certificate_sha256', sa.String(length=64), nullable=True), + sa.PrimaryKeyConstraint('id'), + ) + op.create_index('ix_restore_test_run_at', 'restore_test', ['run_at']) + + +def downgrade() -> None: + op.drop_index('ix_restore_test_run_at', table_name='restore_test') + op.drop_table('restore_test') diff --git a/migrations/versions/f7a1b5c9d3e2_continuity_snapshot_emailed.py b/migrations/versions/f7a1b5c9d3e2_continuity_snapshot_emailed.py new file mode 100644 index 00000000..75f2779c --- /dev/null +++ b/migrations/versions/f7a1b5c9d3e2_continuity_snapshot_emailed.py @@ -0,0 +1,31 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""tenant.continuity_snapshot_emailed (CONTINUITY_PLAN.md BC-D5) + +Whether the bank's daily open-items snapshot is emailed to its incident contacts. + +Revision ID: f7a1b5c9d3e2 +Revises: e5f9a3b7d2c8 +Create Date: 2026-10-31 +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +revision: str = 'f7a1b5c9d3e2' +down_revision: Union[str, None] = 'e5f9a3b7d2c8' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + with op.batch_alter_table('tenant') as batch: + batch.add_column(sa.Column('continuity_snapshot_emailed', sa.Boolean(), nullable=False, server_default='0')) + + +def downgrade() -> None: + with op.batch_alter_table('tenant') as batch: + batch.drop_column('continuity_snapshot_emailed') diff --git a/plans/CONTINUITY_PLAN.md b/plans/CONTINUITY_PLAN.md index a9a234f3..4d49c7a3 100644 --- a/plans/CONTINUITY_PLAN.md +++ b/plans/CONTINUITY_PLAN.md @@ -103,3 +103,40 @@ model. The host's backup job stays the host's; OpenClear only reads what it leav ## Progress log - 2026-10-31: Plan written. Waiting for the product owner's decisions BC-D1 to BC-D8. +- 2026-10-31: BC1 built on `bc1-gap`: `services/continuity_service.py` (`recover_from_gap`, run at the + start of the sweep, compares the sweep's own `ScheduledJobRun` with its interval plus five + minutes; moves the deadlines that fell in the gap to `min(midnight, now + min review)`; + `exception_item.deadline_moved_reason = "outage"`, migration `d4e8f2a6c1b7`; the action-log + entry and `continuity.gap` (integrity, high); `notify_continuity_gap` to administrators and to + each customer's deciders); control check `continuity_gaps` and control OC-CP-02. **Deviation + from BC-D1:** nothing is skipped on the recovering sweep: moving the deadlines is enough, since + the sweep decides only items whose deadline has passed, and items that were due *before* the + gap (missed by the previous sweep for a reason of their own) are swept as usual. +- 2026-10-31: BC2 built on `bc1-gap`: `services/readiness.py` (database, a probe byte written and + deleted in storage, Redis when configured, the sweep's last run against its interval), `GET + /ready` (503 only for the database and storage; `degraded` for the rest), the + `openclear_component_up` metric, the runbook's row and paragraph. +- 2026-10-31: BC3 built on `bc1-gap`: the `backup_manifests` area, `storage.read_key`, + `services/backup_service.py` (the manifest, `freshness`, `sqlite_copy` with `VACUUM INTO` from + housekeeping), check `backup_freshness` and control OC-CP-03, the runbook's "Backups". +- 2026-10-31: BC4 built on `bc1-gap`: `storage.exists/ref/list_keys` on every backend; + `services/storage_reconcile.py` (`reconcile`, the `storage_consistency` nightly check, control + OC-CP-04); `scripts/restore_drill.py` with `restore_test` rows (migration `e5f9a3b7d2c8`), read by + the recovery-objectives check as the last restore test; Platform → Continuity (the manifest, the + drills, "Check now"); the runbook's drill commands; H-CP-9-01 cites them. +- 2026-10-31: BC5 built on `bc1-gap`: `maintenance_mode` (the write guard in `main.py` with the + sign-in, session and data-export paths allowed; `ctx.maintenance` and the banner; the sweep's + pause marker, read by `gap_before`; the `maintenance_mode` check); `services/continuity_snapshot.py` + (hourly `continuity_snapshot_job`; the signed latest file and the dated copy at the bank's 23:00; + emailed to the incident contacts when `tenant.continuity_snapshot_emailed`, migration + `f7a1b5c9d3e2`); Settings → Continuity with the download and the switch. **Deviation from + BC-D4:** the mode is a setting, not a platform switch: a switch kept in the database is exactly + what's unavailable during a restore. +- 2026-10-31: BC6 written: RUNBOOK.md "Recovering the installation", nine steps in the order each + needs the one before. The tabletop against the sandbox is for the product owner to schedule; + it needs a restore of the sandbox's database, which is the host's operation. +- 2026-10-31: **BC1–BC6 built** on `bc1-gap`, each stage's SQLite suite green at its commit; the + Postgres and Linux suites on the branch head before the pull request. Three migrations + (`d4e8f2a6c1b7`, `e5f9a3b7d2c8`, `f7a1b5c9d3e2`), four controls (OC-CP-02 to OC-CP-04 and the + checks under them), one script (`scripts/restore_drill.py`). +- 2026-10-31: Pull request #71. CodeQL flagged the open-items download's `?day=` reaching a file name (three path-injection alerts): the route now builds the name from the parsed date, and a local `read_key` goes through `LocalStorage._path`'s resolved-path check like every other local read. diff --git a/plans/MASTER_PLAN.md b/plans/MASTER_PLAN.md index 737e38ca..d023a867 100644 --- a/plans/MASTER_PLAN.md +++ b/plans/MASTER_PLAN.md @@ -177,3 +177,4 @@ zone, M7 is still built and tested, against a stub edge; only the final confirma - 2026-10-31 REVIEW_1_33_PLAN.md RV1–RV8 built on `review-1-33`: a check paid by a decision is paid (so reverse positive pay can't pay it twice); the teller answer gives nothing the caller didn't send; a late check gets a look before a default; bank deletion clears every storage area, freezes the bank and reports failure; a customer's service ends at midnight everywhere and a sync can't bring it back; custom fields by the letter of their messages; the Redis limiter holds per instance; incident contacts' fallback; refusals as sentences for people. Waiting: the suites on the branch head, then a pull request. - 2026-10-31 Released 1.34.0: REVIEW_1_33_PLAN.md RV1–RV8 (#69): a check paid by a decision is paid, so reverse positive pay can't pay it twice; the teller answer gives nothing the caller didn't send; a late check gets a look before a default; bank deletion clears every storage area, freezes the bank and reports failure; a customer's service ends at midnight everywhere and a sync can't bring it back; custom fields by the letter of their messages; the Redis limiter holds per instance; incident contacts' fallback; refusals as sentences for people. pyasn1 0.6.5 for two CVEs. No security advisories of OpenClear's own. - 2026-10-31 CONTINUITY_PLAN.md written: what OpenClear does across an outage at the cut-off (today: defaults everything when it returns), backups measured instead of attested (a manifest, a freshness check, a restore drill), storage reconciled after a restore, a maintenance mode, a daily open-items snapshot for the bank's own plan, `/ready`, the runbook's recovery procedure, a SQLite copy (BC1–BC6). Waiting for decisions BC-D1 to BC-D8. +- 2026-10-31 CONTINUITY_PLAN.md BC1–BC6 built on `bc1-gap`: an outage across a cut-off decides nothing by default; `/ready`; backups measured from the host's manifest and a SQLite copy; the restore drill and the nightly files-and-rows reconciliation; maintenance mode and the bank's daily open-items file; the runbook's recovery procedure. Waiting: the suites on the branch head, then a pull request. diff --git a/scripts/restore_drill.py b/scripts/restore_drill.py new file mode 100755 index 00000000..8a39ab18 --- /dev/null +++ b/scripts/restore_drill.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""The restore drill (CONTINUITY_PLAN.md BC-D2; CUEC H-CP-9-01): proves a backup restores. + + scripts/restore_drill.py --scratch [--manifest latest.json] + +`dump` is a SQLite copy (backup_manifests/sqlite/…, or any .sqlite file) or a pg_dump custom-format +archive; `--scratch` is a throwaway database to restore into (for SQLite, a path the copy is put +at; for Postgres, an empty database `pg_restore` can write to). The drill: restores the dump; +runs the migrations to head; verifies every bank's action-log chain; reconciles the restored rows +against file storage; checks the dump's hash against the manifest when one is given; then writes +a signed RestoreTest row into the live database (OPENCLEAR_DATABASE_URL), which the Controls page +shows as the last restore test. Exit status 1 when any step fails. Never touches the live +database's own tables except restore_test.""" + +import argparse +import hashlib +import json +import shutil +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / "src")) + + +def restore(dump: Path, scratch_url: str) -> dict: + from sqlalchemy.engine import make_url + + url = make_url(scratch_url) + if url.get_backend_name() == "sqlite": + target = Path(url.database) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(dump, target) + return {"method": "copied the SQLite file", "target": str(target)} + completed = subprocess.run(["pg_restore", "--clean", "--if-exists", "--no-owner", "--no-privileges", "-d", scratch_url, str(dump)], + capture_output=True, text=True) + if completed.returncode != 0: + raise RuntimeError(f"pg_restore failed: {completed.stderr.strip()[-2000:]}") + return {"method": "pg_restore --clean --if-exists", "target": url.render_as_string(hide_password=True)} + + +def migrate(scratch_url: str) -> dict: + from alembic import command + from alembic.config import Config + from alembic.runtime.migration import MigrationContext + from alembic.script import ScriptDirectory + from sqlalchemy import create_engine + + cfg = Config(str(ROOT / "alembic.ini")) + cfg.set_main_option("script_location", str(ROOT / "migrations")) + cfg.set_main_option("sqlalchemy.url", scratch_url.replace("%", "%%")) + engine = create_engine(scratch_url) + with engine.connect() as conn: + before = MigrationContext.configure(conn).get_current_revision() + command.upgrade(cfg, "head") + head = ScriptDirectory.from_config(cfg).get_current_head() + engine.dispose() + return {"from": before, "to": head} + + +def verify_chains(scratch_url: str) -> dict: + from sqlalchemy import create_engine, select + from sqlalchemy.orm import Session + + from openclear.db.tenancy import bind_session_to_tenant + from openclear.domain.tenant import Tenant + from openclear.services import audit_log_service + + engine = create_engine(scratch_url) + banks, broken = 0, [] + with Session(engine) as session: + for tenant_id, slug in session.execute(select(Tenant.id, Tenant.slug)): + bind_session_to_tenant(session, tenant_id) + outcome = audit_log_service.verify_chain(session, tenant_id) + banks += 1 + if not outcome.valid: + broken.append({"bank": slug, "checked": outcome.checked_count, "broken_at": str(outcome.broken_at_entry_id)}) + engine.dispose() + return {"banks": banks, "broken": broken} + + +def reconcile(scratch_url: str) -> dict: + from sqlalchemy import create_engine + from sqlalchemy.orm import Session + + from openclear.services import storage_reconcile + + engine = create_engine(scratch_url) + with Session(engine) as session: + found = storage_reconcile.reconcile(session) + engine.dispose() + return {"ok": found.ok, "summary": found.summary(), **found.as_dict()} + + +def run_drill(dump: Path, scratch_url: str, manifest: dict | None = None, *, record_in=None) -> tuple[bool, dict]: + """Every step, each recorded; `record_in` is a Session on the live database for the RestoreTest row.""" + from openclear.domain.restore_test import RestoreTest + from openclear.services import evidence_service + + steps: dict = {"dump": str(dump), "dump_bytes": dump.stat().st_size, "dump_sha256": hashlib.sha256(dump.read_bytes()).hexdigest()} + ok = True + if manifest is not None: + stated = (manifest.get("database") or {}).get("sha256") + steps["manifest"] = {"completed_at": manifest.get("completed_at"), "sha256_matches": stated is None or stated == steps["dump_sha256"]} + ok &= steps["manifest"]["sha256_matches"] + for name, step in (("restore", lambda: restore(dump, scratch_url)), ("migrate", lambda: migrate(scratch_url)), + ("chains", lambda: verify_chains(scratch_url)), ("files", lambda: reconcile(scratch_url))): + try: + steps[name] = step() + except Exception as exc: # noqa: BLE001 -- the step's failure is the drill's finding + steps[name] = {"error": f"{type(exc).__name__}: {exc}"[:2000]} + ok = False + break + if "chains" in steps and steps["chains"].get("broken"): + ok = False + if "files" in steps and steps["files"].get("ok") is False: + ok = False + steps["ok"], steps["run_at"] = ok, datetime.now(timezone.utc).isoformat() + if record_in is not None: + completed = manifest.get("completed_at") if manifest else None + signed = evidence_service.sign({"type": "OpenClear restore drill", **steps}) + record_in.add(RestoreTest(ok=ok, source=str(dump)[:1000], detail=steps, certificate=signed, certificate_sha256=signed["signature"]["sha256"], + backup_completed_at=datetime.fromisoformat(completed) if completed else None)) + record_in.commit() + return ok, steps + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("dump", type=Path) + parser.add_argument("--scratch", required=True, help="a throwaway database URL to restore into") + parser.add_argument("--manifest", type=Path, help="the backup's latest.json, to check the dump's hash") + parser.add_argument("--no-record", action="store_true", help="don't write the result to the live database") + args = parser.parse_args() + manifest = json.loads(args.manifest.read_text()) if args.manifest else None + session = None + if not args.no_record: + from sqlalchemy.orm import Session + + from openclear.db.session import get_engine + + session = Session(get_engine()) + ok, steps = run_drill(args.dump, args.scratch, manifest, record_in=session) + print(json.dumps(steps, indent=2, default=str)) + print("RESTORE DRILL", "PASSED" if ok else "FAILED") + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/openclear/compliance/checks.py b/src/openclear/compliance/checks.py index 05c66a6e..a2175c5b 100644 --- a/src/openclear/compliance/checks.py +++ b/src/openclear/compliance/checks.py @@ -49,10 +49,14 @@ def _registry() -> tuple[Check, ...]: # The thirteen from COMPLIANCE_PLAN.md C4, in the Controls page's order. Check("field_encryption", "Secrets encrypted in the database", s._field_encryption), Check("storage_encryption", "Disk and database encryption", lambda session, tenant_id: s._storage_encryption(), scope="installation"), - Check("recovery_objectives", "Recovery objectives stated", lambda session, tenant_id: s._recovery_objectives(), + Check("recovery_objectives", "Recovery objectives stated", lambda session, tenant_id: s._recovery_objectives(session), scope="installation"), Check("rate_limit_backend", "Rate limits shared across instances", lambda session, tenant_id: s._rate_limit_backend(), scope="installation"), + Check("backup_freshness", "Backups fresh", lambda session, tenant_id: s._backup_freshness(), scope="installation"), + Check("storage_consistency", "Files and rows agree", lambda session, tenant_id: s._storage_consistency(session), + scope="installation", schedule="nightly"), + Check("maintenance_mode", "Not in maintenance mode", lambda session, tenant_id: s._maintenance_mode(), scope="installation"), Check("edge", "Edge network set up", lambda session, tenant_id: s._edge(), scope="installation", needs_request=True), Check("audit_chain", "Action log intact", s._audit_chain), Check("segregation_of_duties", "Segregation of duties", s._sod), @@ -86,6 +90,7 @@ def _registry() -> tuple[Check, ...]: Check("return_deadline", "Checks decided within their return deadline", c.return_deadline), Check("incident_contacts", "Incident contacts named", c.incident_contacts, schedule="nightly"), Check("custom_fields", "Required custom fields filled in", c.custom_fields, schedule="nightly"), + Check("continuity_gaps", "No outage across a cut-off", c.continuity_gaps), Check("evidence_export", "Evidence exported this quarter", c.evidence_export, schedule="nightly"), # Population checks, nightly over the previous day. Check("dual_control", "Dual control on every decision", c.dual_control, kind="population", schedule="nightly"), @@ -136,11 +141,11 @@ def by_key() -> dict[str, Check]: # The keys, without importing the services (compliance/controls.py and the CUEC inventory read # them at import time); tests/test_compliance checks this matches registry(). KEYS = ( - "field_encryption", "storage_encryption", "recovery_objectives", "rate_limit_backend", "edge", "audit_chain", "segregation_of_duties", "admin_mfa", "anomaly_capture", + "field_encryption", "storage_encryption", "recovery_objectives", "rate_limit_backend", "backup_freshness", "storage_consistency", "maintenance_mode", "edge", "audit_chain", "segregation_of_duties", "admin_mfa", "anomaly_capture", "parser_hardening", "sla", "api_client_secrets", "customer_self_service", "event_delivery", "production_safety", "dormant_accounts", "customer_dormant_users", "privileged_accounts", "staff_mfa", "customer_mfa", "customer_sod", "customer_api_client_secrets", "customer_sso", "scim_sync", "integration_tls", - "default_dispositions", "rejected_rows", "stale_ach_authorizations", "wsud_form", "reverse_accounts_agreed", "return_deadline", "incident_contacts", "custom_fields", + "default_dispositions", "rejected_rows", "stale_ach_authorizations", "wsud_form", "reverse_accounts_agreed", "return_deadline", "incident_contacts", "custom_fields", "continuity_gaps", "evidence_export", "dual_control", "two_person_changes", "change_impact_replay", "support_access_review", "customer_decision_timeliness", "scheduled_jobs", "transport_security", "platform_staff", "software_currency", "regulatory_currency", "installation_settings", "provider_log", "second_approver", "sensitive_settings_protected", "access_review", "customer_access_review", "file_reconciliation", diff --git a/src/openclear/compliance/controls.py b/src/openclear/compliance/controls.py index 5fb2ceb8..d570ec80 100644 --- a/src/openclear/compliance/controls.py +++ b/src/openclear/compliance/controls.py @@ -303,6 +303,24 @@ def as_dict(self) -> dict: "page, which warns until they're recorded or once the test is over a year old.", ("CP-2", "CP-4"), "detective", "built_in", ("recovery_objectives",), ("H-CP-2-01", "H-CP-9-01", "B-CP-2-03"), "Controls → Recovery objectives stated"), + C("OC-CP-03", "Backups fresh", + "The host's backup job leaves a manifest when it finishes (when, what, where it's kept, for how long); the Controls page " + "shows the last backup and warns when there's no manifest or it's older than the hours allowed. A SQLite installation " + "makes its own nightly copy and manifest.", + ("CP-9", "CP-10"), "detective", "built_in", ("backup_freshness",), ("H-CP-9-01",), "Controls → Backups fresh; RUNBOOK.md, Backups"), + C("OC-CP-04", "A restore is proven and reconciled", + "scripts/restore_drill.py restores a backup into a scratch database, migrates it, verifies every bank's action-log chain " + "and reconciles the restored rows against file storage, then signs the result; the Controls page shows the last passed " + "drill as the last restore test. Nightly, the rows that name files and the files are checked against each other, and " + "Platform → Continuity lists any that disagree.", + ("CP-4", "CP-9", "CP-10"), "detective", "built_in", ("storage_consistency", "maintenance_mode"), ("H-CP-9-01", "H-CP-2-01"), + "scripts/restore_drill.py; Controls → Files and rows agree; Platform → Continuity"), + C("OC-CP-02", "An outage across a cut-off decides nothing by default", + "When OpenClear was unavailable across a cut-off, the items due in that time are given the bank's minimum review time " + "again, never past a check's midnight deadline, instead of the customer's default being applied unseen; the bank's " + "administrators and the customers' deciders are told, and the Controls page shows the gap for a week.", + ("CP-2", "CP-10"), "preventive", "built_in", ("continuity_gaps",), ("B-CP-2-01",), + "services/continuity_service.py; Controls → No outage across a cut-off"), # --- Incident response ------------------------------------------------------------------- C("OC-IR-01", "Incident notices to each bank", "The installation's operator sends incident and maintenance notices to the banks affected: each is emailed to the bank's " diff --git a/src/openclear/compliance/cuecs.py b/src/openclear/compliance/cuecs.py index d8abc177..8d5fe648 100644 --- a/src/openclear/compliance/cuecs.py +++ b/src/openclear/compliance/cuecs.py @@ -666,8 +666,11 @@ def oscal_id(nist: str) -> str: "Back up the database and file storage, encrypted and kept apart from the primary, with at least one copy immutable or " "offline behind sign-in separate from production; state how long backups are kept; and test a clean restore at least " "yearly.", - "OpenClear's database migrations upgrade and downgrade, and its action log verifies after a restore.", - "attested", None, "Daily; restore tested yearly", "The host's backup records", related=("CP-4", "CP-10")), + "OpenClear's database migrations upgrade and downgrade, its action log verifies after a restore, and " + "scripts/restore_drill.py does both against a scratch database and signs the result; the Controls page reads the " + "host's backup manifest (Backups fresh) and the last passed drill (Recovery objectives stated).", + "attested", None, "Daily; restore tested yearly", "The host's backup records; Controls → Backups fresh; the restore drill", + related=("CP-4", "CP-10")), Cuec("H-CP-2-01", "host", "CP-2", "State recovery objectives", "State the installation's recovery time and recovery point objectives, record them and each restore test's date in " "OpenClear's settings, and give the banks the restore tests' results.", diff --git a/src/openclear/config.py b/src/openclear/config.py index fcdd2a55..352f5ec5 100644 --- a/src/openclear/config.py +++ b/src/openclear/config.py @@ -147,6 +147,14 @@ def settings_customise_sources( data_export_storage_dir: str = "./data/exports" audit_archive_storage_dir: str = "./data/audit_archives" decision_file_storage_dir: str = "./data/decision_files" + # Where the host's backup job leaves its manifest (latest.json) and a SQLite installation its nightly + # copy (CONTINUITY_PLAN.md BC-D2, BC-D8); RUNBOOK.md "Backups" has the manifest's shape. + backup_manifest_storage_dir: str = "./data/backups" + backup_max_age_hours: int = 26 # the Controls page warns when the last backup is older + # Maintenance mode (CONTINUITY_PLAN.md BC-D4): sign-in and reading work, every write answers 503 except the data + # export, the cut-off sweep pauses (and the pause counts as a gap when it ends), and every page carries the banner. + maintenance_mode: bool = False + sqlite_backup_keep_days: int = 14 # Where new files are written (storage.py): "local" (the directories above) or "s3", # an S3-compatible bucket shared by every instance (pip install 'openclear[s3]'). # s3_endpoint_url is for MinIO, Cloudflare R2 and the like; leave it empty for AWS. diff --git a/src/openclear/db/tenancy.py b/src/openclear/db/tenancy.py index dd6e3725..2cb1a72a 100644 --- a/src/openclear/db/tenancy.py +++ b/src/openclear/db/tenancy.py @@ -133,6 +133,8 @@ class TenantContext: # (services/customer_offboarding_service.py): shown to its people on every page until then. service_ends_at: datetime | None = None service_last_day: date | None = None # the same, as the day it is at the bank (REVIEW_1_33_PLAN.md O10) + # The installation is being restored (config.maintenance_mode): every page says so (CONTINUITY_PLAN.md BC-D4). + maintenance: bool = False # The soonest decision deadline among them (base.html's "decide now" banner). nearest_deadline: datetime | None = None # For base.html's "New customer users" nav entry: people a customer's side added who wait diff --git a/src/openclear/domain/__init__.py b/src/openclear/domain/__init__.py index fbd3813e..3b759f24 100644 --- a/src/openclear/domain/__init__.py +++ b/src/openclear/domain/__init__.py @@ -58,6 +58,7 @@ platform_log_verification, platform_webauthn, revoked_session, + restore_test, scheduled_job_run, security_event, security_event_destination, diff --git a/src/openclear/domain/exception_item.py b/src/openclear/domain/exception_item.py index dee3fc18..36537c9e 100644 --- a/src/openclear/domain/exception_item.py +++ b/src/openclear/domain/exception_item.py @@ -95,6 +95,8 @@ class ExceptionItem(Base): # When the exception.deadline_approaching webhook went out (services/webhook_service.py # warn_deadlines_approaching), so each exception is announced once. deadline_warned_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + # Why the deadline was moved after it was set: "outage" (services/continuity_service.py, CONTINUITY_PLAN.md BC-D1). + deadline_moved_reason: Mapped[str | None] = mapped_column(String(30), nullable=True) created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) # When the row last changed: what list endpoints' updated_since filters on, so a bank's # data pipeline fetches only what changed (docs/guides/bi.md). diff --git a/src/openclear/domain/notification.py b/src/openclear/domain/notification.py index 27b448d4..88a634ce 100644 --- a/src/openclear/domain/notification.py +++ b/src/openclear/domain/notification.py @@ -42,6 +42,12 @@ class NotificationType(str, enum.Enum): EMAIL_CHANGED = "email_changed" # services/malware_service.py: an upload was malware (OW-D34). To the bank's administrators; always email. MALWARE_DETECTED = "malware_detected" + # services/continuity_service.py: OpenClear was down across a cut-off and the items due then are due + # again (CONTINUITY_PLAN.md BC-D1). To the bank's administrators and the customers' deciders; always email. + CONTINUITY_GAP = "continuity_gap" + # services/continuity_snapshot.py: the bank's open items as of its end of day, to its incident contacts when it asks + # (CONTINUITY_PLAN.md BC-D5). To addresses, always email. + CONTINUITY_SNAPSHOT = "continuity_snapshot" # services/sign_in_links.py: someone asked for their organizations' sign-in links (FR8). To # the address asked about, so always email. SIGN_IN_LINKS = "sign_in_links" diff --git a/src/openclear/domain/restore_test.py b/src/openclear/domain/restore_test.py new file mode 100644 index 00000000..ab5a22f2 --- /dev/null +++ b/src/openclear/domain/restore_test.py @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +import uuid +from datetime import datetime + +from sqlalchemy import JSON, Boolean, DateTime, String, func +from sqlalchemy.orm import Mapped, mapped_column + +from openclear.db.base import Base, new_uuid + + +class RestoreTest(Base): + """A restore drill's result (scripts/restore_drill.py; CONTINUITY_PLAN.md BC-D2): a backup + restored into a scratch database, migrated, its action logs verified, its files reconciled, + and the outcome signed, so "the last restore test" is evidence rather than a date typed in. + One row per drill, for the whole installation: no bank's data.""" + + __tablename__ = "restore_test" + + id: Mapped[uuid.UUID] = mapped_column(primary_key=True, default=new_uuid) + run_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now(), nullable=False, index=True) + ok: Mapped[bool] = mapped_column(Boolean, nullable=False) + backup_completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) # from the manifest + source: Mapped[str] = mapped_column(String(1000), nullable=False) # the dump or copy restored + detail: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict) # steps and what each found + certificate: Mapped[dict | None] = mapped_column(JSON, nullable=True) # evidence_service.sign of the detail + certificate_sha256: Mapped[str | None] = mapped_column(String(64), nullable=True) diff --git a/src/openclear/domain/tenant.py b/src/openclear/domain/tenant.py index ad9b8ef1..48af54b0 100644 --- a/src/openclear/domain/tenant.py +++ b/src/openclear/domain/tenant.py @@ -178,6 +178,8 @@ class Tenant(Base): # When an administrator last confirmed the bank's incident contacts (domain/incident.py; # REGULATORY_PLAN.md RG3): the control check incident_contacts asks for it yearly. incident_contacts_confirmed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + # Email the daily open-items snapshot to the incident contacts (services/continuity_snapshot.py; CONTINUITY_PLAN.md BC-D5). + continuity_snapshot_emailed: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="0") # Offboarding (plans/OFFBOARDING_PLAN.md): when the service provider started it, so the bank's # data can be deleted once the wait is over; and when it was deleted (the row stays, inactive, # so the slug is never reused). diff --git a/src/openclear/main.py b/src/openclear/main.py index 191f5e2d..a53510b3 100644 --- a/src/openclear/main.py +++ b/src/openclear/main.py @@ -51,6 +51,12 @@ } +_WRITES = frozenset({"POST", "PUT", "PATCH", "DELETE"}) +# What still works in maintenance mode: signing in and out, and taking a data export. +_MAINTENANCE_ALLOWED = ("/ui/login", "/ui/logout", "/ui/webauthn", "/ui/session", "/api/v1/auth/", "/api/v1/oauth/", "/api/v1/webauthn", + "/ui/settings/data-export", "/api/v1/data-exports", "/ui/platform/login") + + def _wants_json(request: Request) -> bool: return request.url.path.startswith(_API_PREFIXES) @@ -217,6 +223,19 @@ def create_app() -> FastAPI: idempotency.install(app) + @app.middleware("http") + async def _maintenance_mode(request: Request, call_next): + # Being restored (CONTINUITY_PLAN.md BC-D4): people can sign in and look, and an administrator can take a + # data export; nothing else changes until the mode is off. Decisions already made are safe. + if get_settings().maintenance_mode and request.method in _WRITES and not request.url.path.startswith(_MAINTENANCE_ALLOWED): + if request.url.path.startswith(_API_PREFIXES): + return JSONResponse({"detail": "OpenClear is being restored; nothing can be changed until it's done. Decisions already made are safe."}, + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, headers={"Retry-After": "600"}) + return _web_error(request, 503, title="OpenClear is being restored", + message="Nothing can be changed until the restore is done; decisions already made are safe. You can still look, " + "and an administrator can take a data export.", headers={"Retry-After": "600"}) + return await call_next(request) + @app.middleware("http") async def _limit_request_body_size(request: Request, call_next): # Fast, uniform first line of defense against oversized uploads — reads @@ -262,7 +281,7 @@ async def _rate_limit_global(request: Request, call_next): if request.url.path.startswith(_API_PREFIXES): return JSONResponse({"detail": "Too many requests; try again in a minute"}, status_code=status.HTTP_429_TOO_MANY_REQUESTS, headers={"Retry-After": "60"}) - if request.url.path.startswith("/static/") or request.url.path == "/health": + if request.url.path.startswith("/static/") or request.url.path in ("/health", "/ready"): return PlainTextResponse("Too many requests -- please slow down.", status_code=status.HTTP_429_TOO_MANY_REQUESTS, headers={"Retry-After": "60"}) return _web_error(request, 429, headers={"Retry-After": "60"}) @@ -535,6 +554,22 @@ def health(db: Session = Depends(get_db)): headers={"Retry-After": "30", "Cache-Control": "no-store"}) return {"status": "ok"} + @app.get("/ready") + def ready(db: Session = Depends(get_db)): + """Which parts of this instance work (CONTINUITY_PLAN.md BC-D6): the database and file + storage (required: 503 without them), Redis when configured and the cut-off sweep's last run + (degraded without them: 200, with the component named). For the host's monitors; `/health` + stays the load balancer's one-question check.""" + from openclear.services import readiness + + found = readiness.components(db) + state = readiness.status(found) + body = {"status": state, "components": {c.name: {"ok": c.ok, "detail": c.detail} for c in found}} + headers = {"Cache-Control": "no-store"} + if state == "unavailable": + return JSONResponse(body, status_code=503, headers={**headers, "Retry-After": "30"}) + return JSONResponse(body, headers=headers) + @app.get("/.well-known/security.txt", include_in_schema=False) def security_txt(): """RFC 9116: where to report a vulnerability in this installation (Settings.security_txt_*). diff --git a/src/openclear/metrics.py b/src/openclear/metrics.py index 19737027..bd00066c 100644 --- a/src/openclear/metrics.py +++ b/src/openclear/metrics.py @@ -90,4 +90,13 @@ def render(session: Session) -> str: out += ["# HELP openclear_http_request_seconds_total Time this process spent serving them.", "# TYPE openclear_http_request_seconds_total counter"] out += [_line("openclear_http_request_seconds_total", round(t, 3), method=m, status=s) for (m, s), t in sorted(seconds.items())] + # Each component this instance needs, 1 or 0 (CONTINUITY_PLAN.md BC-D6; services/readiness.py). + from openclear.services import readiness + + bind_session_to_tenant(session, None) + out += ["# HELP openclear_component_up Whether the database, file storage, Redis (when configured) and the cut-off sweep are working.", + "# TYPE openclear_component_up gauge"] + for component in readiness.components(session): + out.append(_line("openclear_component_up", 1 if component.ok else 0, component=component.name)) + return "\n".join(out) + "\n" diff --git a/src/openclear/services/backup_service.py b/src/openclear/services/backup_service.py new file mode 100644 index 00000000..79f9de1b --- /dev/null +++ b/src/openclear/services/backup_service.py @@ -0,0 +1,123 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""Backups measured, not attested (CONTINUITY_PLAN.md BC-D2, BC-D8). + +The host's backup job writes `latest.json` into the `backup_manifests` storage area when it +finishes (RUNBOOK.md "Backups" gives the shape); the `backup_freshness` check reads it and warns +when it's missing or older than `backup_max_age_hours`. A SQLite installation has no backup job +of its own, so housekeeping makes one: `VACUUM INTO` a dated copy (consistent without stopping +writers), puts it in the same area, and writes the manifest itself.""" + +import hashlib +import json +import logging +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +from pathlib import Path + +from sqlalchemy.engine import Engine, make_url + +from openclear import storage +from openclear.config import get_settings + +logger = logging.getLogger(__name__) + +AREA = "backup_manifests" +MANIFEST_KEY = "latest.json" + + +@dataclass(frozen=True) +class Manifest: + completed_at: datetime + database_bytes: int | None + database_sha256: str | None + storage_areas: dict + kept_at: str | None + retention_days: int | None + made_by: str | None + + @property + def age(self) -> timedelta: + return datetime.now(timezone.utc) - self.completed_at + + +def latest() -> Manifest | None: + """The newest manifest, or None when the host's job hasn't left one (or left one that doesn't read).""" + raw = storage.read_key(AREA, MANIFEST_KEY) + if raw is None: + return None + try: + data = json.loads(raw) + completed = datetime.fromisoformat(str(data["completed_at"])) + if completed.tzinfo is None: + completed = completed.replace(tzinfo=timezone.utc) + database = data.get("database") or {} + return Manifest(completed, database.get("bytes"), database.get("sha256"), data.get("storage_areas") or {}, + data.get("kept_at"), data.get("retention_days"), data.get("made_by")) + except (ValueError, KeyError, TypeError) as exc: + logger.warning("The backup manifest at %s/%s doesn't read: %s", AREA, MANIFEST_KEY, exc) + return None + + +def freshness(): + """The control check (compliance/checks.py `backup_freshness`, installation scope).""" + from openclear.services.controls_service import Control, PASS, WARN + + title = "Backups fresh" + manifest = latest() + limit = timedelta(hours=get_settings().backup_max_age_hours) + if manifest is None: + return Control("backup_freshness", title, WARN, + f"No backup manifest at {AREA}/{MANIFEST_KEY}. The host's backup job should write one when it finishes " + "(RUNBOOK.md, Backups); until it does, backups are known only from the host's own records.", + {"manifest": None, "verified_by_openclear": False}) + evidence = {"completed_at": manifest.completed_at.isoformat(), "database_bytes": manifest.database_bytes, + "storage_areas": sorted(manifest.storage_areas), "kept_at": manifest.kept_at, "retention_days": manifest.retention_days, + "made_by": manifest.made_by, "max_age_hours": get_settings().backup_max_age_hours} + when = manifest.completed_at.strftime("%Y-%m-%d %H:%M UTC") + hours = int(manifest.age.total_seconds() // 3600) + if manifest.age > limit: + return Control("backup_freshness", title, WARN, f"The last backup finished {when}, {hours} hours ago: more than the " + f"{get_settings().backup_max_age_hours} hours allowed (OPENCLEAR_BACKUP_MAX_AGE_HOURS).", evidence) + size = f"{manifest.database_bytes / 1_000_000:,.0f} MB" if manifest.database_bytes else "size not stated" + kept = f"; kept at {manifest.kept_at}" if manifest.kept_at else "" + days = f" for {manifest.retention_days} days" if manifest.retention_days else "" + return Control("backup_freshness", title, PASS, f"Last backup {when}: the database ({size}) and {len(manifest.storage_areas)} storage " + f"area{'s' if len(manifest.storage_areas) != 1 else ''}{kept}{days}.", evidence) + + +def write_manifest(*, database_bytes: int | None, database_sha256: str | None, storage_areas: dict, kept_at: str | None, + retention_days: int | None, made_by: str, completed_at: datetime | None = None) -> str: + """Writes latest.json; the host's own job writes the same shape (RUNBOOK.md, Backups).""" + body = {"completed_at": (completed_at or datetime.now(timezone.utc)).isoformat(), "database": {"bytes": database_bytes, "sha256": database_sha256}, + "storage_areas": storage_areas, "kept_at": kept_at, "retention_days": retention_days, "made_by": made_by} + return storage.put(AREA, MANIFEST_KEY, json.dumps(body, indent=2).encode()) + + +def sqlite_copy(engine: Engine) -> Path | None: + """A SQLite installation's nightly backup (BC-D8): `VACUUM INTO` a dated copy in the backup + area (uploaded when storage is a bucket), the manifest written, copies older than + `sqlite_backup_keep_days` removed. None for any other database: the host's job backs it up.""" + settings = get_settings() + url = make_url(settings.database_url) + if url.get_backend_name() != "sqlite" or not url.database or url.database == ":memory:": + return None + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S.%fZ") # to the microsecond: two copies a second apart are two files + local_dir = Path(settings.backup_manifest_storage_dir) / "sqlite" + local_dir.mkdir(parents=True, exist_ok=True) + target = local_dir / f"openclear-{stamp}.sqlite" + with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn: + conn.exec_driver_sql("VACUUM INTO ?", (str(target),)) + digest, size = hashlib.sha256(target.read_bytes()).hexdigest(), target.stat().st_size + kept_at = str(target) + if storage.is_object_ref(storage.put(AREA, f"sqlite/{target.name}.probe", b"")): + kept_at = storage.put_file(AREA, f"sqlite/{target.name}", target) # moved into the bucket + write_manifest(database_bytes=size, database_sha256=digest, storage_areas={a: {} for a in storage.AREAS if a != AREA}, + kept_at=kept_at, retention_days=settings.sqlite_backup_keep_days, made_by="openclear housekeeping (VACUUM INTO)") + cutoff = datetime.now(timezone.utc) - timedelta(days=settings.sqlite_backup_keep_days) + for old in local_dir.glob("openclear-*.sqlite"): + if datetime.fromtimestamp(old.stat().st_mtime, tz=timezone.utc) < cutoff: + old.unlink(missing_ok=True) + logger.info("SQLite backup written: %s (%d bytes)", kept_at, size) + return target diff --git a/src/openclear/services/continuity_service.py b/src/openclear/services/continuity_service.py new file mode 100644 index 00000000..912031bb --- /dev/null +++ b/src/openclear/services/continuity_service.py @@ -0,0 +1,147 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""What the cut-off sweep does after a gap (CONTINUITY_PLAN.md BC-D1). + +The disposition sweep decides every open item whose deadline has passed. When OpenClear was down +across a cut-off, that means the first sweep after it comes back applies each customer's default +to every item nobody could look at. Here, a gap longer than the sweep's interval is treated as an +outage: every open item whose deadline fell inside it gets the bank's minimum review time again, +never past a check's midnight deadline, and the bank's administrators and the customers' deciders +are told. Items whose midnight deadline already passed stay due at once, flagged as RG1 flags them.""" + +import logging +import uuid +from dataclasses import dataclass, field +from datetime import datetime, timedelta, timezone +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from openclear.config import get_settings +from openclear.db.tenancy import bind_session_to_tenant +from openclear.domain.exception_item import ExceptionItem, ExceptionStatus +from openclear.domain.scheduled_job_run import ScheduledJobRun +from openclear.domain.tenant import Tenant + +logger = logging.getLogger(__name__) + +SWEEP_JOB_ID = "sweep_expired_dispositions_job" +PAUSED_MARKER = SWEEP_JOB_ID + ":paused" # a ScheduledJobRun row: when maintenance mode first stopped the sweep (BC-D4) +GRACE = timedelta(minutes=5) # a slow sweep isn't an outage +OUTAGE = "outage" # exception_item.deadline_moved_reason + + +@dataclass +class Gap: + tenant_id: uuid.UUID + unavailable_from: datetime + unavailable_to: datetime + moved: int = 0 + due_again_at: datetime | None = None # the earliest new deadline + by_customer: dict[uuid.UUID | None, int] = field(default_factory=dict) + + +def _now() -> datetime: + return datetime.now(timezone.utc) + + +def _aware(value: datetime | None) -> datetime | None: + return value.replace(tzinfo=timezone.utc) if value is not None and value.tzinfo is None else value + + +def note_paused(session: Session, now: datetime | None = None) -> None: + """Maintenance mode stopped the sweep: remember when it began, so the pause counts as a gap + once it ends (the sweep's own run row keeps being written by the scheduler meanwhile).""" + now = now or _now() + marker = session.get(ScheduledJobRun, PAUSED_MARKER) + if marker is None: + session.add(ScheduledJobRun(job_id=PAUSED_MARKER, last_started_at=now, last_finished_at=None, last_ok=None)) + session.flush() + + +def gap_before(session: Session, now: datetime) -> tuple[datetime, datetime] | None: + """(from, to) when the sweep's previous run finished longer ago than its interval allows, or + when maintenance mode paused it: from the pause's start (the marker is then cleared).""" + marker = session.get(ScheduledJobRun, PAUSED_MARKER) + if marker is not None: + since = _aware(marker.last_started_at) + session.delete(marker) + session.flush() + return (since, now) + run = session.get(ScheduledJobRun, SWEEP_JOB_ID) + last = _aware(run.last_finished_at) if run else None + if last is None: + return None + interval = timedelta(seconds=get_settings().disposition_sweep_interval_seconds) + return (last, now) if now - last > interval + GRACE else None + + +def recover_from_gap(session: Session, now: datetime | None = None) -> list[Gap]: + """Before the sweep decides anything: moves the deadlines that fell in the gap, records it + in each bank's action log (continuity.gap, a high security event) and tells people. Commits + per bank; leaves the session unbound. Returns what was done, per bank.""" + from openclear.services import audit_log_service, notification_service + + now = now or _now() + gap = gap_before(session, now) + if gap is None: + return [] + since, until = gap + bind_session_to_tenant(session, None) + rows = session.execute(select(ExceptionItem.id, ExceptionItem.tenant_id).where( + ExceptionItem.status.in_([ExceptionStatus.OPEN, ExceptionStatus.PENDING_APPROVAL]), + ExceptionItem.decision_deadline > since, ExceptionItem.decision_deadline <= until)).all() + by_tenant: dict[uuid.UUID, list[uuid.UUID]] = {} + for item_id, tenant_id in rows: + by_tenant.setdefault(tenant_id, []).append(item_id) + done: list[Gap] = [] + for tenant_id, item_ids in by_tenant.items(): + bind_session_to_tenant(session, tenant_id) + try: + tenant = session.get(Tenant, tenant_id) + summary = Gap(tenant_id, since, until) + window = timedelta(minutes=tenant.cutoff_min_review_minutes) + for item in session.execute(select(ExceptionItem).where(ExceptionItem.id.in_(item_ids))).scalars(): + due = now + window + returns_by = _aware(item.return_deadline) + if returns_by is not None: + due = min(due, returns_by) + if due <= now: + continue # its midnight deadline has passed: due at once, and flagged as late when decided + item.decision_deadline, item.deadline_moved_reason, item.deadline_warned_at = due, OUTAGE, None + summary.moved += 1 + summary.due_again_at = due if summary.due_again_at is None else min(summary.due_again_at, due) + summary.by_customer[item.customer_id] = summary.by_customer.get(item.customer_id, 0) + 1 + session.flush() + audit_log_service.record_action( + session, tenant_id, actor_user_id=None, channel="system", action="continuity.gap", + summary=(f"OpenClear was unavailable from {since:%Y-%m-%d %H:%M} to {until:%H:%M} UTC, across a cut-off; " + f"{summary.moved} items due in that time are due again from {summary.due_again_at:%H:%M} UTC" + if summary.moved else + f"OpenClear was unavailable from {since:%Y-%m-%d %H:%M} to {until:%H:%M} UTC; {len(item_ids)} items were due in " + "that time and are past their midnight deadline, so they are due at once")[:500], + resource_type="tenant", resource_id=tenant_id, + metadata={"unavailable_from": since.isoformat(), "unavailable_to": until.isoformat(), "items_due_in_gap": len(item_ids), + "moved": summary.moved, "due_again_at": summary.due_again_at.isoformat() if summary.due_again_at else None}) + if summary.moved: + notification_service.notify_continuity_gap(session, tenant, summary, zone=_zone(tenant)) + session.commit() + done.append(summary) + logger.warning("Continuity gap for bank %s: %d of %d items due again from %s", tenant_id, summary.moved, len(item_ids), + summary.due_again_at) + except Exception: # noqa: BLE001 -- one bank's failure doesn't stop the others, nor the sweep + session.rollback() + logger.exception("Recovering from the gap failed for bank %s", tenant_id) + bind_session_to_tenant(session, None) + return done + + +def _zone(tenant: Tenant) -> ZoneInfo: + from openclear.services.cutoff_service import time_zone_for + + try: + return ZoneInfo(time_zone_for(tenant)) + except ZoneInfoNotFoundError: + return ZoneInfo("UTC") diff --git a/src/openclear/services/continuity_snapshot.py b/src/openclear/services/continuity_snapshot.py new file mode 100644 index 00000000..aad3e637 --- /dev/null +++ b/src/openclear/services/continuity_snapshot.py @@ -0,0 +1,201 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""The bank's open items, as a file it can work from without OpenClear (CONTINUITY_PLAN.md BC-D5). + +Every hour, each bank's open and pending items are written as a CSV into its storage +(`data_exports//continuity/open-items-latest.csv`) with a signed sidecar; once a day, at the +bank's end of day, a dated copy is kept, and emailed to the incident contacts when the bank asks. +It's what CUEC B-CP-2-01's plan works from when OpenClear is unreachable: the bank can decide by +hand and send returns to its core before the midnight deadline.""" + +import csv +import hashlib +import io +import json +import logging +import uuid +from dataclasses import dataclass +from datetime import datetime, timezone +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from openclear import storage +from openclear.db.tenancy import bind_session_to_tenant +from openclear.domain.account import Account +from openclear.domain.customer import Customer +from openclear.domain.tenant import Tenant + +logger = logging.getLogger(__name__) + +AREA = "data_exports" +LATEST = "open-items-latest.csv" +COLUMNS = ("customer_number", "customer_name", "account_number", "network", "item", "amount", "presented", "problems", "status", + "recommended", "decide_by", "return_by", "default_at_cutoff") +EMAIL_ROWS = 200 + + +@dataclass(frozen=True) +class Snapshot: + as_of: datetime + rows: int + sha256: str + ref: str + dated_ref: str | None + emailed: list[str] + + +def _key(tenant_id: uuid.UUID, name: str) -> str: + return f"{tenant_id}/continuity/{name}" + + +def _zone(tenant: Tenant) -> ZoneInfo: + from openclear.services.cutoff_service import time_zone_for + + try: + return ZoneInfo(time_zone_for(tenant)) + except ZoneInfoNotFoundError: + return ZoneInfo("UTC") + + +def _local(value: datetime | None, zone: ZoneInfo) -> str: + if value is None: + return "" + value = value.replace(tzinfo=timezone.utc) if value.tzinfo is None else value + return value.astimezone(zone).strftime("%Y-%m-%d %H:%M") + + +def rows_for(session: Session, tenant: Tenant) -> list[dict]: + """Every open or pending item, with what a person deciding by hand needs. The bank is bound.""" + from openclear.networks.registry import get_adapter + from openclear.services import exception_service + from openclear.services.auto_disposition_service import get_disposition_mode + + zone = _zone(tenant) + items = exception_service.list_exceptions(session, tenant.id, statuses=exception_service.NEEDS_ATTENTION) + by_network: dict[str, list[uuid.UUID]] = {} + for item in items: + by_network.setdefault(item.network_code, []).append(item.source_item_id) + sources = {} + for code, ids in by_network.items(): + sources.update(get_adapter(code).load_source_items(session, ids)) + accounts = {a.id: a for a in session.execute(select(Account).where( + Account.id.in_({getattr(s, "account_id", None) for s in sources.values()} - {None}))).scalars()} if sources else {} + customers = {c.id: c for c in session.execute(select(Customer).where( + Customer.id.in_({i.customer_id for i in items} - {None}))).scalars()} if items else {} + modes: dict[tuple, str] = {} + out = [] + for item in items: + source = sources.get(item.source_item_id) + account = accounts.get(getattr(source, "account_id", None)) + customer = customers.get(item.customer_id) + if item.customer_id is not None: + modes.setdefault((item.customer_id, item.network_code), + get_disposition_mode(session, tenant.id, item.customer_id, item.network_code).value) + if item.network_code == "check": + label, amount, presented = getattr(source, "check_number", ""), getattr(source, "presented_amount", ""), getattr(source, "presented_date", "") + else: + label, amount, presented = getattr(source, "trace_number", "") or getattr(source, "originator_name", ""), getattr(source, "amount", ""), \ + getattr(source, "effective_date", "") + out.append({ + "customer_number": customer.customer_number if customer else "", "customer_name": customer.name if customer else "(the bank's own)", + "account_number": account.account_number if account else "", "network": item.network_code, "item": str(label), + "amount": str(amount), "presented": str(presented or ""), "problems": item.exception_types, "status": item.status.value, + "recommended": item.recommended_outcome or "", "decide_by": _local(item.decision_deadline, zone), + "return_by": _local(item.return_deadline, zone), "default_at_cutoff": modes.get((item.customer_id, item.network_code), ""), + }) + return out + + +def _csv(rows: list[dict]) -> bytes: + out = io.StringIO() + writer = csv.DictWriter(out, fieldnames=COLUMNS, extrasaction="ignore") + writer.writeheader() + for row in rows: + writer.writerow({k: ("'" + v if isinstance(v, str) and v[:1] in ("=", "+", "@", "\t", "\r") else v) for k, v in row.items()}) + return out.getvalue().encode() + + +def write(session: Session, tenant: Tenant, now: datetime | None = None) -> Snapshot: + """The latest file and its signature; the dated copy at the bank's end of day. The bank is bound; commits nothing.""" + from openclear.services import evidence_service + + now = now or datetime.now(timezone.utc) + zone = _zone(tenant) + rows = rows_for(session, tenant) + data = _csv(rows) + digest = hashlib.sha256(data).hexdigest() + signed = evidence_service.sign({"type": "OpenClear open-items snapshot", "bank": {"id": str(tenant.id), "slug": tenant.slug}, + "as_of": now.isoformat(), "rows": len(rows), "sha256": digest, "columns": list(COLUMNS)}) + ref = storage.put(AREA, _key(tenant.id, LATEST), data) + storage.put(AREA, _key(tenant.id, "open-items-latest.sig.json"), json.dumps(signed, indent=2).encode()) + dated_ref, emailed = None, [] + local = now.astimezone(zone) + dated_name = f"open-items-{local:%Y-%m-%d}.csv" + if local.hour >= 23 and not storage.exists(storage.ref(AREA, _key(tenant.id, dated_name))): + dated_ref = storage.put(AREA, _key(tenant.id, dated_name), data) + storage.put(AREA, _key(tenant.id, dated_name.replace(".csv", ".sig.json")), json.dumps(signed, indent=2).encode()) + if tenant.continuity_snapshot_emailed: + emailed = _email(session, tenant, rows, local) + return Snapshot(now, len(rows), digest, ref, dated_ref, emailed) + + +def _email(session: Session, tenant: Tenant, rows: list[dict], local: datetime) -> list[str]: + from openclear.domain.notification import Notification, NotificationChannel, NotificationType + from openclear.services import incident_service + from openclear.services.notification_service import _render_body + + to = incident_service.recipients(session, tenant.id) + context = {"bank_name": tenant.name, "as_of": local.strftime("%Y-%m-%d %H:%M"), "zone": local.tzinfo.key, "count": len(rows), + "shown": rows[:EMAIL_ROWS], "more": max(0, len(rows) - EMAIL_ROWS), "columns": COLUMNS} + for address in to: + session.add(Notification(tenant_id=tenant.id, recipient_user_id=None, notification_type=NotificationType.CONTINUITY_SNAPSHOT, + channel=NotificationChannel.EMAIL, destination=address, + subject=f"{tenant.name}: {len(rows)} open positive pay items as of {local:%Y-%m-%d}"[:255], + body=_render_body(NotificationType.CONTINUITY_SNAPSHOT, NotificationChannel.EMAIL, context), + resource_type="tenant", resource_id=tenant.id)) + session.flush() + return to + + +def latest(tenant_id: uuid.UUID) -> dict | None: + """The latest snapshot's signed description (as_of, rows, sha256), or None.""" + raw = storage.read_key(AREA, _key(tenant_id, "open-items-latest.sig.json")) + if raw is None: + return None + try: + return json.loads(raw) + except ValueError: + return None + + +def read(tenant_id: uuid.UUID, name: str = LATEST) -> bytes | None: + if not (name == LATEST or (name.startswith("open-items-") and name.endswith(".csv") and len(name) == len("open-items-2026-10-31.csv"))): + return None + return storage.read_key(AREA, _key(tenant_id, name)) + + +def dated(tenant_id: uuid.UUID) -> list[str]: + """The dated copies' file names, newest first.""" + prefix = _key(tenant_id, "open-items-20") + return sorted((k.rsplit("/", 1)[-1] for k in storage.list_keys(AREA, prefix) if k.endswith(".csv")), reverse=True) + + +def run_all(session: Session, now: datetime | None = None) -> int: + """Every active bank, each bound in turn and committed on its own.""" + bind_session_to_tenant(session, None) + banks = session.execute(select(Tenant.id).where(Tenant.is_active.is_(True), Tenant.data_deleted_at.is_(None))).scalars().all() + written = 0 + for tenant_id in banks: + bind_session_to_tenant(session, tenant_id) + try: + write(session, session.get(Tenant, tenant_id), now) + session.commit() + written += 1 + except Exception: # noqa: BLE001 -- one bank's failure doesn't stop the others + session.rollback() + logger.exception("Continuity snapshot failed for bank %s", tenant_id) + bind_session_to_tenant(session, None) + return written diff --git a/src/openclear/services/control_checks.py b/src/openclear/services/control_checks.py index b736f1d5..a53caf5f 100644 --- a/src/openclear/services/control_checks.py +++ b/src/openclear/services/control_checks.py @@ -442,6 +442,23 @@ def incident_contacts(session: Session, tenant_id: uuid.UUID) -> Control: return Control("incident_contacts", title, PASS, f"{who}, confirmed {confirmed:%Y-%m-%d}.", evidence) +def continuity_gaps(session: Session, tenant_id: uuid.UUID) -> Control: + """OpenClear wasn't down across a cut-off in the last seven days (CONTINUITY_PLAN.md BC-D1). + When it was, the items due then were given the minimum review time instead of a default, and + the bank should know: FAIL for a week, with the gap.""" + from openclear.domain.audit_log_entry import AuditLogEntry + + title = "No outage across a cut-off" + since = _now() - timedelta(days=7) + gaps = session.execute(select(AuditLogEntry).where(AuditLogEntry.tenant_id == tenant_id, AuditLogEntry.action == "continuity.gap", + AuditLogEntry.occurred_at >= since).order_by(AuditLogEntry.occurred_at.desc())).scalars().all() + evidence = {"gaps_last_7_days": [{"at": _aware(g.occurred_at).isoformat(), "summary": g.summary} for g in gaps]} + if gaps: + return Control("continuity_gaps", title, FAIL, f"{_plural(len(gaps), 'outage')} across a cut-off in the last 7 days; the latest: " + f"{gaps[0].summary}", evidence) + return Control("continuity_gaps", title, PASS, "No outage across a cut-off in the last 7 days.", evidence) + + def custom_fields(session: Session, tenant_id: uuid.UUID) -> Control: """Required custom fields are filled in (plans/CUSTOM_FIELDS_PLAN.md CF-D5). Required holds in people's forms only, so a record made by a file, the API or the core's sync can lack one; the diff --git a/src/openclear/services/controls_service.py b/src/openclear/services/controls_service.py index 5ba32433..3be83e41 100644 --- a/src/openclear/services/controls_service.py +++ b/src/openclear/services/controls_service.py @@ -88,11 +88,48 @@ def _rate_limit_backend() -> Control: return Control("rate_limit_backend", title, PASS, "Rate limits are shared through Redis, which answers.", evidence) -def _recovery_objectives() -> Control: +def _storage_consistency(session) -> Control: + """Rows whose files are gone (services/storage_reconcile.py; CONTINUITY_PLAN.md BC-D3).""" + from openclear.services import storage_reconcile + + return storage_reconcile.consistency_check(session) + + +def _maintenance_mode() -> Control: + """Whether the installation is in maintenance mode (config.maintenance_mode; CONTINUITY_PLAN.md BC-D4).""" + title = "Not in maintenance mode" + if get_settings().maintenance_mode: + return Control("maintenance_mode", title, WARN, "OpenClear is in maintenance mode: sign-in and reading work, writes are refused, " + "the cut-off sweep is paused and will treat the pause as a gap when it ends.", {"maintenance_mode": True}) + return Control("maintenance_mode", title, PASS, "Not in maintenance mode.", {"maintenance_mode": False}) + + +def _backup_freshness() -> Control: + """The last backup, from the manifest the host's job leaves (services/backup_service.py; CONTINUITY_PLAN.md BC-D2).""" + from openclear.services import backup_service + + return backup_service.freshness() + + +def _last_restore_drill(session): + """The newest passed restore drill (scripts/restore_drill.py), if any: measured, where the setting is typed.""" + from sqlalchemy import select + + from openclear.domain.restore_test import RestoreTest + + return session.execute(select(RestoreTest).where(RestoreTest.ok.is_(True)).order_by(RestoreTest.run_at.desc()).limit(1)).scalar_one_or_none() + + +def _recovery_objectives(session=None) -> Control: settings = get_settings() statement, tested = settings.recovery_objectives_attestation, settings.last_restore_test + drill = _last_restore_drill(session) if session is not None else None + if drill is not None and (tested is None or _aware(drill.run_at).date() > tested): + tested = _aware(drill.run_at).date() # the drill's own record wins over a date typed in (CONTINUITY_PLAN.md BC-D2) title = "Recovery objectives stated" - evidence = {"attestation": statement, "last_restore_test": tested.isoformat() if tested else None, "verified_by_openclear": False} + evidence = {"attestation": statement, "last_restore_test": tested.isoformat() if tested else None, + "verified_by_openclear": drill is not None and _aware(drill.run_at).date() == tested, + "last_restore_drill": {"run_at": _aware(drill.run_at).isoformat(), "certificate_sha256": drill.certificate_sha256} if drill else None} if not statement: return Control("recovery_objectives", title, WARN, "The host's recovery time and recovery point objectives aren't recorded. Agree them with your provider, then " diff --git a/src/openclear/services/notification_service.py b/src/openclear/services/notification_service.py index 12da329a..f09b707b 100644 --- a/src/openclear/services/notification_service.py +++ b/src/openclear/services/notification_service.py @@ -21,6 +21,7 @@ from sqlalchemy.orm import Session from openclear.config import get_settings +from openclear.domain.customer import Customer from openclear.domain.decision import Decision from openclear.domain.exception_item import ExceptionItem from openclear.domain.notification import Notification, NotificationChannel, NotificationPreference, NotificationType @@ -35,7 +36,8 @@ # SMS for every type, including these) stays fully preference-controlled. _ALWAYS_EMAIL_TYPES = frozenset({NotificationType.ACCOUNT_LOCKED, NotificationType.PASSWORD_CHANGED, NotificationType.INACTIVITY_WARNING, NotificationType.PASSWORD_RESET_LINK, NotificationType.SECURITY_KEY_CHANGED, - NotificationType.NEW_DEVICE_SIGN_IN, NotificationType.MALWARE_DETECTED, NotificationType.SIGN_IN_LINKS}) + NotificationType.NEW_DEVICE_SIGN_IN, NotificationType.MALWARE_DETECTED, NotificationType.SIGN_IN_LINKS, + NotificationType.CONTINUITY_GAP, NotificationType.CONTINUITY_SNAPSHOT}) _EMAIL_SUBJECTS: dict[NotificationType, str] = { NotificationType.EXCEPTION_CREATED: "New exception ready for review", @@ -52,6 +54,8 @@ NotificationType.EMAIL_CHANGE_CONFIRM: "Confirm your new OpenClear email address", NotificationType.EMAIL_CHANGED: "The email address on your OpenClear sign-in changed", NotificationType.MALWARE_DETECTED: "Malware was found in a file uploaded to OpenClear", + NotificationType.CONTINUITY_GAP: "OpenClear was unavailable across a cut-off: items are due again", + NotificationType.CONTINUITY_SNAPSHOT: "Your open positive pay items, for your continuity plan", NotificationType.CUSTOMER_USER_DECIDED: "Your bank decided on a new person", NotificationType.ATTESTATION_REQUESTED: "Your bank asks you to attest your responsibilities", NotificationType.INACTIVITY_WARNING: "Sign in to keep your OpenClear access", @@ -464,6 +468,29 @@ def notify_malware_detected(session: Session, tenant_id: uuid.UUID, *, filename: "later": later}, resource_type="file") +def notify_continuity_gap(session: Session, tenant, gap, *, zone) -> None: + """OpenClear was down across a cut-off (services/continuity_service.py): the bank's + administrators hear of every item moved; each customer's deciders of their own.""" + def context(count: int, customer_name: str | None) -> dict[str, Any]: + return {"unavailable_from": gap.unavailable_from.astimezone(zone).strftime("%Y-%m-%d %H:%M"), + "unavailable_to": gap.unavailable_to.astimezone(zone).strftime("%H:%M"), + "due_again_at": gap.due_again_at.astimezone(zone).strftime("%Y-%m-%d %H:%M"), "zone": zone.key, "count": count, + "customer_name": customer_name, "exceptions_url": _absolute_url("/ui/exceptions")} + + for admin in _recipients_for_permission(session, tenant.id, "admin:manage", customer_id=None): + _queue(session, tenant_id=tenant.id, recipient=admin, notification_type=NotificationType.CONTINUITY_GAP, + template_context=context(gap.moved, None), resource_type="tenant", resource_id=tenant.id) + for customer_id, count in gap.by_customer.items(): + if customer_id is None: + continue + customer = session.get(Customer, customer_id) + for person in _recipients_for_permission(session, tenant.id, "exception:decide", customer_id=customer_id): + if person.id in {a.id for a in _recipients_for_permission(session, tenant.id, "admin:manage", customer_id=None)}: + continue # told already, as an administrator + _queue(session, tenant_id=tenant.id, recipient=person, notification_type=NotificationType.CONTINUITY_GAP, + template_context=context(count, customer.name if customer else None), resource_type="customer", resource_id=customer_id) + + def notify_email_address(session: Session, user: User, *, destination: str, notification_type: NotificationType, context: dict[str, Any], tenant_id: uuid.UUID | None = None) -> None: """An email about a person's address, sent to an address rather than to the person's diff --git a/src/openclear/services/readiness.py b/src/openclear/services/readiness.py new file mode 100644 index 00000000..1a35003f --- /dev/null +++ b/src/openclear/services/readiness.py @@ -0,0 +1,95 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""Whether this instance can serve (CONTINUITY_PLAN.md BC-D6): the database, file storage, +Redis when it's configured, and whether the scheduler has run lately. `/health` keeps checking +the database alone, for load balancers; `/ready` and the `openclear_component_up` metric say +which part is failing, so a partial failure (files can't be written; the sweep hasn't run for an +hour) is seen before a bank notices.""" + +import logging +import uuid +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone + +from sqlalchemy import text +from sqlalchemy.orm import Session + +logger = logging.getLogger(__name__) + +PROBE_AREA = "data_exports" # every installation has it; the probe is one byte, written and deleted +REQUIRED = ("database", "storage") # without these the instance can't serve; the rest degrade + + +@dataclass(frozen=True) +class Component: + name: str + ok: bool + detail: str + + +def database(session: Session) -> Component: + try: + session.execute(text("SELECT 1")) + return Component("database", True, "answers") + except Exception as exc: # noqa: BLE001 -- the answer is no + logger.exception("Readiness: the database didn't answer") + return Component("database", False, f"unreachable ({type(exc).__name__})") + + +def storage() -> Component: + from openclear import storage as files + + key = f"_ready/{uuid.uuid4().hex}" + try: + ref = files.put(PROBE_AREA, key, b"ready") + files.delete(ref) + return Component("storage", True, f"{files.backend().__class__.__name__} writes") + except Exception as exc: # noqa: BLE001 -- the answer is no + logger.exception("Readiness: storage can't be written") + return Component("storage", False, f"can't write ({type(exc).__name__})") + + +def redis() -> Component | None: + """None when limits are kept in the process: nothing to check.""" + from openclear.web.rate_limit import backend_status + + backend, reachable = backend_status() + if backend != "redis": + return None + return Component("redis", reachable, "answers" if reachable else "unreachable: limits held per instance") + + +def scheduler(session: Session, now: datetime | None = None) -> Component: + """The cut-off sweep's last run against its interval (the same measure continuity_service + uses for a gap). Not yet run is fine on a fresh installation.""" + from openclear.services.continuity_service import GRACE, SWEEP_JOB_ID, _aware + from openclear.config import get_settings + from openclear.domain.scheduled_job_run import ScheduledJobRun + + now = now or datetime.now(timezone.utc) + run = session.get(ScheduledJobRun, SWEEP_JOB_ID) + last = _aware(run.last_finished_at) if run else None + if last is None: + return Component("scheduler", True, "no run recorded yet") + interval = timedelta(seconds=get_settings().disposition_sweep_interval_seconds) + age = now - last + if age > interval + GRACE: + return Component("scheduler", False, f"the cut-off sweep last ran {int(age.total_seconds() // 60)} minutes ago") + return Component("scheduler", True, f"the cut-off sweep ran {int(age.total_seconds() // 60)} minutes ago") + + +def components(session: Session) -> list[Component]: + found = [database(session), storage()] + cache = redis() + if cache is not None: + found.append(cache) + found.append(scheduler(session) if found[0].ok else Component("scheduler", False, "unknown: the database didn't answer")) + return found + + +def status(found: list[Component]) -> str: + """ready, degraded (something optional is down) or unavailable (something required is).""" + if any(not c.ok for c in found if c.name in REQUIRED): + return "unavailable" + return "degraded" if any(not c.ok for c in found) else "ready" diff --git a/src/openclear/services/security_events.py b/src/openclear/services/security_events.py index 8e9721f9..2a1536c6 100644 --- a/src/openclear/services/security_events.py +++ b/src/openclear/services/security_events.py @@ -152,6 +152,8 @@ class EventType: "custom_field.changed": EventType("configuration", "medium"), # A check decided after its midnight deadline: the bank may be accountable for it (REGULATORY_PLAN.md RG1). "decision.after_return_deadline": EventType("integrity", "high"), + # OpenClear was down across a cut-off; the items due then were given time instead of a default (CONTINUITY_PLAN.md BC-D1). + "continuity.gap": EventType("integrity", "high"), # Teller positive pay (TELLER_PLAN.md TL-D6): each check verified at a branch, declined, recorded cashed, and # what looks like someone trying check numbers. "teller.verify": EventType("data", "low"), @@ -211,6 +213,7 @@ class EventType: "file.malware_detected": "file.malware_detected", "bulk_upload.refused": "bulk_upload.refused", "incident.notice.sent": "incident.notice.sent", + "continuity.gap": "continuity.gap", "incident.notice.acknowledged": "incident.notice.acknowledged", "incident_contact.changed": "incident_contact.changed", "incident_contact.confirmed": "incident_contact.confirmed", diff --git a/src/openclear/services/storage_reconcile.py b/src/openclear/services/storage_reconcile.py new file mode 100644 index 00000000..1f99a092 --- /dev/null +++ b/src/openclear/services/storage_reconcile.py @@ -0,0 +1,127 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""Rows and files, checked against each other (CONTINUITY_PLAN.md BC-D3). + +The database and file storage are backed up separately, so a restore can leave rows naming files +that aren't there (uploads, check images, exports, decision files written after the snapshot) and +files no row names. `reconcile` walks every table that keeps a storage reference and every storage +area, and says which is which: run by the restore drill, by an operator after a restore, and +nightly as a warning-only check so drift is seen before a disaster.""" + +import logging +import uuid +from dataclasses import dataclass, field + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from openclear import storage +from openclear.db.base import Base +from openclear.db.tenancy import acting_as_tenant + +logger = logging.getLogger(__name__) + +# Every column that holds a storage reference, by table, with the area it's kept in. +REFERENCES: dict[str, tuple[tuple[str, str], ...]] = { + "check_image": (("front_image_path", "check_images"), ("back_image_path", "check_images")), + "bulk_upload_file": (("storage_path", "bulk_uploads"),), + "data_export_job": (("archive_path", "data_exports"),), + "decision_batch": (("storage_ref", "decision_files"),), + "ml_model": (("artifact_path", "ml_artifacts"),), + "tenant": (("logo_path", "tenant_assets"), ("favicon_path", "tenant_assets")), +} +# Areas nothing references by row: OpenClear's own working files, not reconciled. +UNREFERENCED_AREAS = frozenset({"audit_archives", "backup_manifests"}) + + +@dataclass +class Reconciliation: + rows_checked: int = 0 + files_checked: int = 0 + missing: list[dict] = field(default_factory=list) # a row whose file isn't there: {table, id, tenant_id, column, ref} + orphans: dict[str, list[str]] = field(default_factory=dict) # files no row names, by area: keys + unreachable: list[str] = field(default_factory=list) # areas storage couldn't list + + @property + def ok(self) -> bool: + return not self.missing and not self.unreachable + + @property + def orphan_count(self) -> int: + return sum(len(v) for v in self.orphans.values()) + + def summary(self) -> str: + parts = [f"{self.rows_checked} references checked"] + parts.append(f"{len(self.missing)} missing" if self.missing else "none missing") + parts.append(f"{self.orphan_count} files with no row" if self.orphan_count else "no files without a row") + if self.unreachable: + parts.append(f"couldn't list: {', '.join(self.unreachable)}") + return "; ".join(parts) + "." + + def as_dict(self) -> dict: + return {"rows_checked": self.rows_checked, "files_checked": self.files_checked, "missing": self.missing[:500], + "missing_count": len(self.missing), "orphans": {a: k[:200] for a, k in self.orphans.items()}, + "orphan_count": self.orphan_count, "unreachable": self.unreachable} + + +def reconcile(session: Session, *, list_files: bool = True) -> Reconciliation: + """Every bank's rows (bound to each in turn, for row-level security) against the files. + Leaves the session bound as it found it. With `list_files` off, only rows are checked.""" + from openclear.domain.tenant import Tenant + + result = Reconciliation() + referenced: set[str] = set() + # Each bank in turn, and back to whatever the caller was bound to (a check runner records its + # results for the bank it started with): acting_as_tenant restores the binding on exit. + with acting_as_tenant(session, None): + tenant_ids = list(session.execute(select(Tenant.id)).scalars()) + for name, columns in REFERENCES.items(): + table = Base.metadata.tables[name] + scopes = tenant_ids if "tenant_id" in table.columns and name != "tenant" else [None] + for tenant_id in scopes: + stmt = select(table.c.id, *(table.c[c] for c, _area in columns), *([table.c.tenant_id] if tenant_id else [])) + if tenant_id: + stmt = stmt.where(table.c.tenant_id == tenant_id) + with acting_as_tenant(session, tenant_id): + rows = session.execute(stmt).all() + for row in rows: + for index, (column, _area) in enumerate(columns): + ref = row[1 + index] + if not ref: + continue + result.rows_checked += 1 + referenced.add(str(ref)) + if not storage.exists(str(ref)): + result.missing.append({"table": name, "id": str(row[0]), "tenant_id": str(tenant_id or row[0]) if name == "tenant" else str(tenant_id), + "column": column, "ref": str(ref)}) + if list_files: + for area in storage.AREAS: + if area in UNREFERENCED_AREAS: + continue + try: + keys = storage.list_keys(area) + except Exception: # noqa: BLE001 -- the store couldn't be listed: said, not raised + logger.exception("Reconciliation: storage area %s couldn't be listed", area) + result.unreachable.append(area) + continue + result.files_checked += len(keys) + lost = [k for k in keys if storage.ref(area, k) not in referenced and not k.startswith("_ready/")] + if lost: + result.orphans[area] = sorted(lost) + return result + + +def consistency_check(session: Session): + """The nightly control check (installation scope, WARN only): rows whose files are gone, and + areas that couldn't be listed. Files without rows are reported, not warned about: a newer + restore's, or an upload still being written.""" + from openclear.services.controls_service import Control, PASS, WARN + + title = "Files and rows agree" + found = reconcile(session) + evidence = found.as_dict() + if found.missing or found.unreachable: + return Control("storage_consistency", title, WARN, found.summary() + " A row whose file is gone means a restore or a storage change " + "left the two apart; Platform → Continuity lists them.", evidence) + return Control("storage_consistency", title, PASS, found.summary(), evidence) diff --git a/src/openclear/storage.py b/src/openclear/storage.py index 2b09a83a..3d1478f5 100644 --- a/src/openclear/storage.py +++ b/src/openclear/storage.py @@ -49,6 +49,7 @@ "ml_artifacts": "ml_artifact_dir", "audit_archives": "audit_archive_storage_dir", "decision_files": "decision_file_storage_dir", + "backup_manifests": "backup_manifest_storage_dir", } @@ -138,6 +139,18 @@ def delete(self, ref: str) -> None: def delete_tenant(self, area: str, tenant_id) -> None: shutil.rmtree(Path(getattr(get_settings(), AREAS[area])) / str(tenant_id), ignore_errors=True) + def exists(self, ref: str) -> bool: + return Path(ref).is_file() + + def ref(self, area: str, key: str) -> str: + return str(Path(getattr(get_settings(), AREAS[area])) / safe_key(key)) + + def list_keys(self, area: str, prefix: str = "") -> list[str]: + root = Path(getattr(get_settings(), AREAS[area])) + if not root.is_dir(): + return [] + return sorted(p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_file() and p.relative_to(root).as_posix().startswith(prefix)) + class S3Storage: """An S3-compatible bucket. `s3_endpoint_url` points it at MinIO, R2 or another @@ -199,6 +212,24 @@ def open(self, ref: str) -> BinaryIO: bucket, key = self._split(ref) return self.client.get_object(Bucket=bucket, Key=key)["Body"] + def exists(self, ref: str) -> bool: + bucket, key = self._split(ref) + try: + self.client.head_object(Bucket=bucket, Key=key) + return True + except Exception: # noqa: BLE001 -- NoSuchKey, 404 and 403 alike read as "not there" + return False + + def ref(self, area: str, key: str) -> str: + return self._ref(self._key(area, key)) + + def list_keys(self, area: str, prefix: str = "") -> list[str]: + base = self._key(area, "") .rstrip("/") + "/" + keys = [] + for page in self.client.get_paginator("list_objects_v2").paginate(Bucket=self.bucket, Prefix=base + prefix): + keys.extend(obj["Key"].removeprefix(base) for obj in page.get("Contents", [])) + return sorted(keys) + @contextlib.contextmanager def local_path(self, ref: str) -> Iterator[Path]: """Downloads the object to a temporary file, for what needs a real file (an mmap @@ -298,6 +329,21 @@ def open(self, ref: str) -> BinaryIO: container, name = self._split(ref) return _ChunkReader(self._blob(container, name).download_blob().chunks()) + + def exists(self, ref: str) -> bool: + container, name = self._split(ref) + try: + return bool(self._blob(container, name).exists()) + except Exception: # noqa: BLE001 -- unreachable reads as "not there" + return False + + def ref(self, area: str, key: str) -> str: + return self._ref(self._name(area, key)) + + def list_keys(self, area: str, prefix: str = "") -> list[str]: + base = self._name(area, "").rstrip("/") + "/" + container = self.service.get_container_client(self.container) + return sorted(b.name.removeprefix(base) for b in container.list_blobs(name_starts_with=base + prefix)) @contextlib.contextmanager def local_path(self, ref: str) -> Iterator[Path]: container, name = self._split(ref) @@ -456,6 +502,37 @@ def delete(ref: str) -> None: for_ref(ref).delete(ref) +def exists(ref: str) -> bool: + """Whether the file a stored reference names is there (services/storage_reconcile.py).""" + return for_ref(ref).exists(ref) + + +def ref(area: str, key: str) -> str: + """The reference `put(area, key, …)` would return, without writing.""" + return backend().ref(area, key) + + +def list_keys(area: str, prefix: str = "") -> list[str]: + """Every key in an area (relative to it), from the configured backend.""" + return backend().list_keys(area, prefix) + + +def read_key(area: str, key: str) -> bytes | None: + """A file by its area and key, or None when there isn't one or the store can't be reached: + for the backup manifest the host's job leaves (CONTINUITY_PLAN.md BC-D2), which nothing in + OpenClear wrote, so there's no stored reference to it.""" + store = backend() + try: + if isinstance(store, LocalStorage): + path = store._path(area, key) # safe_key, and the resolved path stays in the area + return path.read_bytes() if path.is_file() else None + if isinstance(store, S3Storage): + return store.open(store._ref(store._key(area, key))).read() + return store.open(store._ref(store._name(area, key))).read() + except Exception: # noqa: BLE001 -- not there, or unreachable: either way there's nothing to read + return None + + def delete_tenant(area: str, tenant_id) -> None: """Removes a bank's files in one area, wherever they are.""" _local.delete_tenant(area, tenant_id) diff --git a/src/openclear/templates/base.html b/src/openclear/templates/base.html index 39dd0b3c..d58a9171 100644 --- a/src/openclear/templates/base.html +++ b/src/openclear/templates/base.html @@ -186,6 +186,10 @@ visitors out are turned off.

{% endif %} {% include "partials/_flash.html" %} + {% if ctx and ctx.maintenance %} +

OpenClear is being restored. You can look, and an + administrator can take a data export, but nothing can be changed until it's done. Decisions already made are safe.

+ {% endif %} {% for delivery_id, kind, title in (ctx.incident_notices if ctx else ()) %} {# The service provider's incident notice (services/incident_service.py): until an administrator acknowledges it. #} +
+

Files and rows

+
+ + +
+ {% if reconciliation %} +

{{ "agree" if reconciliation.ok else "disagree" }} {{ reconciliation.summary() }}

+ {% if reconciliation.missing %} +

Rows whose file is gone

+ + + {% for m in reconciliation.missing[:200] %}{% endfor %} +
TableRowBankColumnReference
{{ m.table }}{{ m.id }}{{ m.tenant_id }}{{ m.column }}{{ m.ref }}
+ {% endif %} + {% if reconciliation.orphans %} +

Files no row names

+
    {% for area, keys in reconciliation.orphans.items() %}
  • {{ area }}: {{ keys | length }}{% if keys | length <= 20 %} — {{ keys | join(", ") | safe }}{% endif %}
    • {% endfor %}
+ {% endif %} + {% else %} +

The nightly check Files and rows agree runs this; run it by hand after a restore.

+ {% endif %} +
+{% endblock %} diff --git a/src/openclear/templates/platform/home.html b/src/openclear/templates/platform/home.html index 9164a7c2..606fe18e 100644 --- a/src/openclear/templates/platform/home.html +++ b/src/openclear/templates/platform/home.html @@ -47,6 +47,7 @@

Banks

{{ icon("add") }}Set up a bank {{ icon("fact_check") }}Host attestations {{ icon("notifications_active") }}Incidents + {{ icon("backup") }}Continuity {{ icon("list_alt") }}Custom fields {{ icon("fact_check") }}Changes diff --git a/src/openclear/templates/settings/sections/continuity.html b/src/openclear/templates/settings/sections/continuity.html new file mode 100644 index 00000000..9d4ab323 --- /dev/null +++ b/src/openclear/templates/settings/sections/continuity.html @@ -0,0 +1,21 @@ +{% from "_macros/icon.html" import icon %} +

For your continuity plan (CUEC B-CP-2-01): every hour OpenClear writes your open items, with when each is due + and what happens by default at the cut-off, as a signed file you can download here and work from if OpenClear can't be + reached. A dated copy is kept each day at the end of your day{% if emailed %} and emailed to your incident contacts{% endif %}.

+{% if snapshot %} +

{{ snapshot.bundle.rows }} open {{ "item" if snapshot.bundle.rows == 1 else "items" }} as of + + · {{ icon("download") }}Download the latest + · SHA-256 {{ snapshot.bundle.sha256[:16] }}…, signed with the installation's file-signing key.

+{% else %} +

No snapshot yet: the first is written within the hour.

+{% endif %} +{% if dated %} +

Daily copies: {% for name in dated %}{{ name[11:21] }}{{ ", " if not loop.last }}{% endfor %}

+{% endif %} +
+ + +
+
diff --git a/src/openclear/web/deps.py b/src/openclear/web/deps.py index 40759b1d..11e5dbc6 100644 --- a/src/openclear/web/deps.py +++ b/src/openclear/web/deps.py @@ -11,6 +11,7 @@ from openclear.auth.deps import AccessRevoked, WrongTokenType, decode_and_build_context from openclear.db.session import get_db from openclear.web.demo_guard import DEMO_LOCKED_MESSAGE, is_locked_in_demo +from openclear.config import get_settings from openclear.db.tenancy import TenantContext from openclear.web.security import ( read_cookie, @@ -100,6 +101,10 @@ def get_web_context(request: Request, db: Session = Depends(get_db)) -> TenantCo count, nearest = exception_service.attention_summary(db, ctx.tenant_id, customer_id=ctx.customer_id) ctx = dataclasses.replace(ctx, needs_attention_count=count, nearest_deadline=nearest) + if get_settings().maintenance_mode: + import dataclasses + + ctx = dataclasses.replace(ctx, maintenance=True) if ctx.customer_id is None and ({"admin:manage", "tenant:manage"} & set(ctx.permissions)): import dataclasses diff --git a/src/openclear/web/routers/platform.py b/src/openclear/web/routers/platform.py index 081539e3..19a6cb0f 100644 --- a/src/openclear/web/routers/platform.py +++ b/src/openclear/web/routers/platform.py @@ -536,6 +536,37 @@ def when(name: str): return see_other("/ui/platform/incidents", flash=f"Notice {notice.title!r} sent.") +@router.get("/continuity") +def continuity(request: Request, db: Session = Depends(get_db), user: User = Depends(platform_session.require_operator)) -> HTMLResponse: + """Backups, restore drills and files against rows (CONTINUITY_PLAN.md BC-D2, BC-D3).""" + from sqlalchemy import select + + from openclear.domain.restore_test import RestoreTest + from openclear.services import backup_service + + drills = db.execute(select(RestoreTest).order_by(RestoreTest.run_at.desc()).limit(20)).scalars().all() + return render_template(request, "platform/continuity.html", user=user, manifest=backup_service.latest(), + freshness=backup_service.freshness(), drills=drills, reconciliation=None) + + +@router.post("/continuity/reconcile") +def reconcile_storage(request: Request, db: Session = Depends(get_db), user: User = Depends(platform_session.require_operator), + _csrf: None = Depends(verify_csrf)) -> HTMLResponse: + """Rows against files, now: after a restore, or when the nightly check warned.""" + from sqlalchemy import select + + from openclear.domain.restore_test import RestoreTest + from openclear.services import backup_service, platform_log_service, storage_reconcile + + found = storage_reconcile.reconcile(db) + platform_log_service.record(db, actor_user_id=user.id, action="platform.continuity.reconcile", summary=found.summary()[:500], + resource_type="installation", resource_id=None) + db.commit() + drills = db.execute(select(RestoreTest).order_by(RestoreTest.run_at.desc()).limit(20)).scalars().all() + return render_template(request, "platform/continuity.html", user=user, manifest=backup_service.latest(), + freshness=backup_service.freshness(), drills=drills, reconciliation=found) + + @router.get("/banks/new") def new_bank(request: Request, _user: User = Depends(platform_session.require_operator)) -> HTMLResponse: return render_template(request, "platform/provision.html") diff --git a/src/openclear/web/routers/security_settings.py b/src/openclear/web/routers/security_settings.py index 89161177..c34e2e7a 100644 --- a/src/openclear/web/routers/security_settings.py +++ b/src/openclear/web/routers/security_settings.py @@ -41,7 +41,8 @@ NotificationType.NEW_DEVICE_SIGN_IN, NotificationType.EMAIL_CHANGE_CONFIRM, NotificationType.EMAIL_CHANGED, NotificationType.MALWARE_DETECTED, NotificationType.INACTIVITY_WARNING, NotificationType.SIGN_IN_LINKS, - NotificationType.INCIDENT_NOTICE, NotificationType.SERVICE_ENDING) + NotificationType.INCIDENT_NOTICE, NotificationType.SERVICE_ENDING, + NotificationType.CONTINUITY_GAP, NotificationType.CONTINUITY_SNAPSHOT) ] _NOTIFICATION_TYPE_LABELS = { diff --git a/src/openclear/web/routers/tenant_settings.py b/src/openclear/web/routers/tenant_settings.py index 2b6b0702..381ae86d 100644 --- a/src/openclear/web/routers/tenant_settings.py +++ b/src/openclear/web/routers/tenant_settings.py @@ -9,7 +9,7 @@ import uuid -from fastapi import APIRouter, Depends, File, Form, Request, UploadFile +from fastapi import APIRouter, Depends, File, Form, Request, Response, UploadFile from fastapi.responses import HTMLResponse, JSONResponse from sqlalchemy.orm import Session @@ -64,6 +64,7 @@ "auto-import": "Auto-import", "wsud-form": "Unauthorized debit statement", "incident-contacts": "Incident contacts", + "continuity": "Continuity", "custom-fields": "Custom fields", "provider-fields": "From your service provider", } @@ -193,6 +194,11 @@ def _section_context(db: Session, ctx: TenantContext, key: str) -> dict: "cio": next((c for c in rows if c.kind == "cio"), None), "confirmed_at": tenant.incident_contacts_confirmed_at, "notices": incident_service.history(db, ctx.tenant_id, limit=20), "kinds": incident_service.KINDS, "max_contacts": incident_service.MAX_CONTACTS} + if key == "continuity": + from openclear.services import continuity_snapshot + + return {"snapshot": continuity_snapshot.latest(ctx.tenant_id), "dated": continuity_snapshot.dated(ctx.tenant_id)[:14], + "emailed": tenant.continuity_snapshot_emailed} if key == "provider-fields": from openclear.services import custom_field_service @@ -792,6 +798,46 @@ def archive_custom_field(request: Request, field_id: uuid.UUID, archived: str = return _change(request, db, ctx, "custom_field.archive", {"field_id": str(field_id), "archived": archived == "1"}, key="custom-fields") +@router.post("/continuity") +async def update_continuity(request: Request, db: Session = Depends(get_db), + ctx: TenantContext = Depends(require_web_permission("tenant:manage")), _csrf: None = Depends(verify_csrf)): + """Whether the daily open-items snapshot is emailed to the incident contacts (CONTINUITY_PLAN.md BC-D5).""" + form = await request.form() + tenant = db.get(Tenant, ctx.tenant_id) + wanted = form.get("emailed") == "true" + if wanted != tenant.continuity_snapshot_emailed: + tenant.continuity_snapshot_emailed = wanted + audit_log_service.record_action(db, ctx.tenant_id, actor_user_id=ctx.user_id, channel="web", action="continuity.snapshot_email.changed", + summary=f"The daily open-items snapshot is {'now' if wanted else 'no longer'} emailed to the incident contacts", + resource_type="tenant", resource_id=ctx.tenant_id) + db.commit() + return _saved(request, "Continuity settings saved.", key="continuity") + + +@router.get("/continuity/open-items.csv") +def download_open_items(day: str = "", db: Session = Depends(get_db), + ctx: TenantContext = Depends(require_web_permission("exception:read"))) -> Response: + """The latest open-items snapshot, or a dated copy (`?day=YYYY-MM-DD`): the bank's own, signed.""" + from openclear.services import continuity_snapshot + + if ctx.customer_id is not None: + raise WebNotFound() # the bank's whole list; a customer's people see their items on the Exceptions page + if day: + # The name is built from the parsed date, never from the string the browser sent. + from datetime import date + + try: + name = f"open-items-{date.fromisoformat(day).isoformat()}.csv" + except ValueError: + raise WebNotFound() from None + else: + name = continuity_snapshot.LATEST + data = continuity_snapshot.read(ctx.tenant_id, name) + if data is None: + raise WebNotFound() + return Response(data, media_type="text/csv", headers={"Content-Disposition": f'attachment; filename="{name}"', "Cache-Control": "no-store"}) + + @router.post("/incident-contacts") async def update_incident_contacts( request: Request, diff --git a/src/openclear/workers/scheduler.py b/src/openclear/workers/scheduler.py index 2a4b7c2b..9f9dcbb1 100644 --- a/src/openclear/workers/scheduler.py +++ b/src/openclear/workers/scheduler.py @@ -23,6 +23,7 @@ notification_dispatch_job, retrain_job, security_event_delivery_job, + continuity_snapshot_job, sweep_expired_dispositions_job, decision_outbox_job, decision_lock_job, @@ -48,6 +49,8 @@ def _jobs(settings) -> list[tuple[str, Callable[[], None], object, timedelta]]: IntervalTrigger(seconds=settings.disposition_sweep_interval_seconds), timedelta(seconds=settings.disposition_sweep_interval_seconds))) jobs.append(("decision_lock_job", decision_lock_job, IntervalTrigger(seconds=settings.decision_lock_interval_seconds), timedelta(seconds=settings.decision_lock_interval_seconds))) + # Each bank's open items as a signed file it can work from without OpenClear (CONTINUITY_PLAN.md BC-D5). + jobs.append(("continuity_snapshot_job", continuity_snapshot_job, IntervalTrigger(hours=1), hour)) if settings.enable_decision_outbox_scheduler: jobs.append(("decision_outbox_job", decision_outbox_job, IntervalTrigger(seconds=settings.decision_outbox_tick_seconds), timedelta(seconds=settings.decision_outbox_tick_seconds))) diff --git a/src/openclear/workers/tasks.py b/src/openclear/workers/tasks.py index 90015ea0..4040e7ff 100644 --- a/src/openclear/workers/tasks.py +++ b/src/openclear/workers/tasks.py @@ -137,6 +137,13 @@ def sweep_expired_dispositions_job() -> None: session = get_session_factory()() try: + if get_settings().maintenance_mode: + # Being restored (CONTINUITY_PLAN.md BC-D4): nothing is decided; the pause is a gap when it ends. + from openclear.services import continuity_service + + continuity_service.note_paused(session) + session.commit() + return # First, warn about deadlines that are near (exception.deadline_approaching webhook). try: if warned := webhook_service.warn_deadlines_approaching(session): @@ -174,6 +181,14 @@ def sweep_expired_dispositions_job() -> None: session.rollback() logger.exception("Deadline reminders failed") now = datetime.now(timezone.utc) + # Down across a cut-off? The items due in the gap get a look before any default (CONTINUITY_PLAN.md BC-D1). + try: + from openclear.services import continuity_service + + continuity_service.recover_from_gap(session, now) + except Exception: # noqa: BLE001 -- the sweep itself matters more + session.rollback() + logger.exception("Recovering from a gap failed") rows = session.execute( select(ExceptionItem.id, ExceptionItem.tenant_id).where( ExceptionItem.status.in_([ExceptionStatus.OPEN, ExceptionStatus.PENDING_APPROVAL]), @@ -394,6 +409,19 @@ def malware_recheck_job() -> None: session.close() +def continuity_snapshot_job() -> None: + """Hourly: each bank's open items as a signed CSV in its storage (services/continuity_snapshot.py).""" + from openclear.services import continuity_snapshot + + session = get_session_factory()() + try: + written = continuity_snapshot.run_all(session) + if written: + logger.info("Continuity snapshots written for %d banks", written) + finally: + session.close() + + def housekeeping_job() -> None: """Daily clean-up of operational rows (services/housekeeping_service.py): finished jobs, sent notifications, old bulk-upload row results and expired export archives; then @@ -404,6 +432,12 @@ def housekeeping_job() -> None: session = get_session_factory()() try: housekeeping_service.run(session) + try: + from openclear.services import backup_service + + backup_service.sqlite_copy(session.get_bind()) # a SQLite installation's nightly backup (CONTINUITY_PLAN.md BC-D8) + except Exception: # noqa: BLE001 + logger.exception("The SQLite backup copy failed") try: inactivity_service.sweep_all(session) except Exception: # noqa: BLE001 diff --git a/tests/conftest.py b/tests/conftest.py index 65ebd11e..7fcce6ff 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -109,6 +109,10 @@ def isolated_filesystem_settings(tmp_path, monkeypatch): monkeypatch.setenv("OPENCLEAR_CHECK_IMAGE_STORAGE_DIR", str(tmp_path / "check_images")) monkeypatch.setenv("OPENCLEAR_TENANT_ASSET_STORAGE_DIR", str(tmp_path / "tenant_assets")) monkeypatch.setenv("OPENCLEAR_BULK_UPLOAD_STORAGE_DIR", str(tmp_path / "bulk_uploads")) + monkeypatch.setenv("OPENCLEAR_BACKUP_MANIFEST_STORAGE_DIR", str(tmp_path / "backups")) + monkeypatch.setenv("OPENCLEAR_DATA_EXPORT_STORAGE_DIR", str(tmp_path / "exports")) + monkeypatch.setenv("OPENCLEAR_DECISION_FILE_STORAGE_DIR", str(tmp_path / "decision_files")) + monkeypatch.setenv("OPENCLEAR_AUDIT_ARCHIVE_STORAGE_DIR", str(tmp_path / "audit_archives")) # No default cut-off in tests: decisions are final when made unless a test sets a cut-off # (tests/test_decisions). Production refuses to start without one (config.assert_production_safe). monkeypatch.setenv("OPENCLEAR_DEFAULT_DECISION_CUTOFF", "") diff --git a/tests/fixtures/viewport/darwin/controls.png b/tests/fixtures/viewport/darwin/controls.png index 8af447d0..754ef2ce 100644 Binary files a/tests/fixtures/viewport/darwin/controls.png and b/tests/fixtures/viewport/darwin/controls.png differ diff --git a/tests/test_auth/test_scheduler_leader_lock.py b/tests/test_auth/test_scheduler_leader_lock.py index d3376aa7..ee06303e 100644 --- a/tests/test_auth/test_scheduler_leader_lock.py +++ b/tests/test_auth/test_scheduler_leader_lock.py @@ -120,7 +120,7 @@ def test_every_scheduled_job_goes_through_the_lock(monkeypatch): jobs = sched.get_jobs() assert {job.id for job in jobs} == { "ml_retrain_job", "dropbox_import_job", "notification_dispatch_job", - "sweep_expired_dispositions_job", "decision_lock_job", "demo_reset_job", + "sweep_expired_dispositions_job", "decision_lock_job", "continuity_snapshot_job", "demo_reset_job", } for job in jobs: job.func() diff --git a/tests/test_compliance/test_backups.py b/tests/test_compliance/test_backups.py new file mode 100644 index 00000000..f8fb2393 --- /dev/null +++ b/tests/test_compliance/test_backups.py @@ -0,0 +1,62 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""Backups measured (CONTINUITY_PLAN.md BC-D2, BC-D8): the Controls page reads the manifest the +host's backup job leaves and warns when it's missing or old; a SQLite installation makes its own +nightly copy and manifest.""" + +import json +import sqlite3 +from datetime import datetime, timedelta, timezone + +from sqlalchemy import create_engine + +from openclear import storage +from openclear.config import get_settings +from openclear.services import backup_service + + +def test_the_check_reads_the_hosts_manifest(db_session): + missing = backup_service.freshness() + assert missing.status == "WARN" and "No backup manifest" in missing.detail + + backup_service.write_manifest(database_bytes=734_003_200, database_sha256="ab" * 32, storage_areas={"check_images": {"objects": 3}}, + kept_at="s3://acme-backups/openclear/2026-10-31/", retention_days=35, made_by="backup.sh") + fresh = backup_service.freshness() + assert fresh.status == "PASS" and "734 MB" in fresh.detail and "1 storage area;" in fresh.detail and "35 days" in fresh.detail + + backup_service.write_manifest(database_bytes=1, database_sha256=None, storage_areas={}, kept_at=None, retention_days=None, + made_by="backup.sh", completed_at=datetime.now(timezone.utc) - timedelta(hours=30)) + stale = backup_service.freshness() + assert stale.status == "WARN" and "30 hours ago" in stale.detail + + storage.put(backup_service.AREA, backup_service.MANIFEST_KEY, b"not json") + assert backup_service.freshness().status == "WARN" # unreadable is as good as missing + + +def test_a_sqlite_installation_copies_itself_nightly(tmp_path, monkeypatch): + db = tmp_path / "live.sqlite" + with sqlite3.connect(db) as conn: + conn.execute("create table t (n integer)") + conn.execute("insert into t values (1), (2)") + monkeypatch.setattr(get_settings(), "database_url", f"sqlite:///{db}") + monkeypatch.setattr(get_settings(), "backup_manifest_storage_dir", str(tmp_path / "backups")) + engine = create_engine(f"sqlite:///{db}") + + copy = backup_service.sqlite_copy(engine) + assert copy is not None and copy.exists() and copy.parent == tmp_path / "backups" / "sqlite" + with sqlite3.connect(copy) as conn: + assert conn.execute("select count(*) from t").fetchone() == (2,) # a consistent copy, with the rows + manifest = json.loads(storage.read_key(backup_service.AREA, backup_service.MANIFEST_KEY)) + assert manifest["database"]["bytes"] == copy.stat().st_size and manifest["made_by"].startswith("openclear housekeeping") + assert backup_service.freshness().status == "PASS" + + old = copy.parent / "openclear-20200101T000000Z.sqlite" + old.write_bytes(b"x") + import os + os.utime(old, (0, 0)) + backup_service.sqlite_copy(engine) + assert not old.exists() and len(list(copy.parent.glob("openclear-*.sqlite"))) == 2 # pruned to the days kept + + monkeypatch.setattr(get_settings(), "database_url", "postgresql+psycopg://x:y@localhost/z") + assert backup_service.sqlite_copy(engine) is None # the host's job backs Postgres up diff --git a/tests/test_compliance/test_restore_drill.py b/tests/test_compliance/test_restore_drill.py new file mode 100644 index 00000000..5cd8f2f6 --- /dev/null +++ b/tests/test_compliance/test_restore_drill.py @@ -0,0 +1,63 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""The restore drill (CONTINUITY_PLAN.md BC-D2): a SQLite copy of the test database is restored +into a scratch file, migrated, its chains verified, its files reconciled, and the signed result +recorded; the Controls page then shows it as the last restore test.""" + +import importlib.util +import sqlite3 +from pathlib import Path + +import pytest +from sqlalchemy import create_engine, select + +from openclear.domain.restore_test import RestoreTest +from openclear.services import controls_service, evidence_service + +ROOT = Path(__file__).resolve().parents[2] + + +def _drill(): + spec = importlib.util.spec_from_file_location("restore_drill", ROOT / "scripts" / "restore_drill.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def test_a_drill_restores_migrates_verifies_and_records(db_session, tenant_factory, tmp_path, engine): + if engine.dialect.name != "sqlite": + pytest.skip("the drill's SQLite path copies the file; the Postgres path is pg_restore, exercised by the host's own drill") + tenant_factory.make(slug="drill-bank") + db_session.commit() + dump = tmp_path / "copy.sqlite" + with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn: + conn.exec_driver_sql("VACUUM INTO ?", (str(dump),)) + scratch = tmp_path / "scratch" / "restored.sqlite" + + ok, steps = _drill().run_drill(dump, f"sqlite:///{scratch}", {"completed_at": "2026-10-31T02:00:00+00:00", + "database": {"sha256": steps_sha(dump)}}, record_in=db_session) + assert ok, steps + assert steps["restore"]["method"].startswith("copied") and steps["migrate"]["to"] and steps["chains"]["banks"] >= 1 + assert steps["chains"]["broken"] == [] and steps["files"]["ok"] and steps["manifest"]["sha256_matches"] + with sqlite3.connect(scratch) as conn: + assert conn.execute("select count(*) from tenant where slug = 'drill-bank'").fetchone() == (1,) + + [record] = db_session.execute(select(RestoreTest)).scalars().all() + assert record.ok and evidence_service.verify(record.certificate) and record.backup_completed_at is not None + assert controls_service._recovery_objectives(db_session).evidence["last_restore_drill"]["certificate_sha256"] == record.certificate_sha256 + + +def steps_sha(path: Path) -> str: + import hashlib + + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def test_a_broken_dump_fails_the_drill(db_session, tmp_path): + dump = tmp_path / "broken.sqlite" + dump.write_bytes(b"not a database") + ok, steps = _drill().run_drill(dump, f"sqlite:///{tmp_path / 'scratch.sqlite'}", None, record_in=db_session) + assert not ok and "error" in steps["migrate"] + [record] = db_session.execute(select(RestoreTest)).scalars().all() + assert not record.ok diff --git a/tests/test_compliance/test_storage_reconcile.py b/tests/test_compliance/test_storage_reconcile.py new file mode 100644 index 00000000..ea85668b --- /dev/null +++ b/tests/test_compliance/test_storage_reconcile.py @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""Rows and files checked against each other (CONTINUITY_PLAN.md BC-D3): a row whose file is gone +is reported and warned about; a file no row names is reported; the platform page runs it.""" + +from openclear import storage +from openclear.domain.check_image import CheckImage +from openclear.services import storage_reconcile +from tests.test_web.test_staff_and_operator import _platform_login, _staff + + +def test_missing_files_and_orphans_are_found(db_session, tenant_factory): + tenant, account, _users = tenant_factory.make(slug="recon") + good = storage.put("check_images", f"{tenant.id}/good.png", b"png") + db_session.add(CheckImage(tenant_id=tenant.id, front_image_path=good)) + gone_ref = storage.ref("check_images", f"{tenant.id}/gone.png") + db_session.add(CheckImage(tenant_id=tenant.id, front_image_path=gone_ref)) + storage.put("bulk_uploads", f"{tenant.id}/nobody-names-me.csv", b"x") + db_session.commit() + + found = storage_reconcile.reconcile(db_session) + assert found.rows_checked >= 2 and not found.ok + assert [m["ref"] for m in found.missing] == [gone_ref] and found.missing[0]["table"] == "check_image" + assert found.orphans["bulk_uploads"] == [f"{tenant.id}/nobody-names-me.csv"] and "check_images" not in found.orphans + assert storage_reconcile.consistency_check(db_session).status == "WARN" + + storage.put("check_images", f"{tenant.id}/gone.png", b"back") + assert storage_reconcile.reconcile(db_session).missing == [] + assert storage_reconcile.consistency_check(db_session).status == "PASS" # orphans are reported, not warned about + + +def test_the_platform_page_runs_it(client, db_session, tenant_factory): + tenant_factory.make(slug="recon-page") + operator = _staff(db_session, "ops@recon.example", "operator") + _platform_login(client, operator.email) + page = client.get("/ui/platform/continuity") + assert page.status_code == 200 and "No backup manifest" in page.text and "No drill recorded yet" in page.text + csrf = client.cookies.get("csrf_token") + ran = client.post("/ui/platform/continuity/reconcile", data={"csrf_token": csrf}) + assert ran.status_code == 200 and "references checked" in ran.text diff --git a/tests/test_decisions/test_continuity_gap.py b/tests/test_decisions/test_continuity_gap.py new file mode 100644 index 00000000..e59a596b --- /dev/null +++ b/tests/test_decisions/test_continuity_gap.py @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""OpenClear down across a cut-off (CONTINUITY_PLAN.md BC-D1): the first sweep after it returns +gives the items due in the gap the bank's minimum review time again instead of applying a +default unseen, tells people, records the gap, and the Controls page shows it for a week. An +item whose midnight deadline has passed stays due at once; a sweep after a normal interval +changes nothing.""" + +from datetime import datetime, timedelta, timezone + +from sqlalchemy import select + +from openclear.domain.audit_log_entry import AuditLogEntry +from openclear.domain.exception_item import ExceptionStatus +from openclear.domain.notification import Notification, NotificationType +from openclear.domain.scheduled_job_run import ScheduledJobRun +from openclear.domain.security_event import SecurityEvent +from openclear.services import continuity_service, control_checks +from tests.test_decisions.test_changeable import _item + + +def _last_sweep(db_session, ago: timedelta) -> None: + db_session.merge(ScheduledJobRun(job_id=continuity_service.SWEEP_JOB_ID, last_started_at=datetime.now(timezone.utc) - ago, + last_finished_at=datetime.now(timezone.utc) - ago, last_ok=True)) + db_session.commit() + + +def _aware(value): + return value.replace(tzinfo=timezone.utc) if value.tzinfo is None else value + + +def test_items_due_during_an_outage_get_the_review_time_again_and_people_are_told(db_session, tenant_factory, monkeypatch, session_factory): + from openclear.workers import tasks + + tenant, account, users = tenant_factory.make(slug="gap-bank") + tenant.cutoff_min_review_minutes = 120 + db_session.commit() + now = datetime.now(timezone.utc) + during = _item(db_session, tenant, account, number="7001", due_in=timedelta(hours=-1)) # its cut-off passed while down + during.return_deadline = now + timedelta(hours=20) + too_late = _item(db_session, tenant, account, number="7002", due_in=timedelta(hours=-2)) # its midnight deadline passed too + too_late.return_deadline = now - timedelta(minutes=30) + before = _item(db_session, tenant, account, number="7003", due_in=timedelta(hours=-5)) # due before the outage: swept as usual + before.return_deadline = now + timedelta(hours=20) + db_session.commit() + _last_sweep(db_session, timedelta(hours=3)) + + monkeypatch.setattr(tasks, "get_session_factory", lambda: session_factory) + tasks.sweep_expired_dispositions_job() + db_session.expire_all() + + assert during.status == ExceptionStatus.OPEN and during.deadline_moved_reason == "outage" + assert timedelta(minutes=115) < _aware(during.decision_deadline) - now <= timedelta(minutes=125) + assert too_late.deadline_moved_reason is None and _aware(too_late.decision_deadline) < now # due at once, as RG1 says + assert before.deadline_moved_reason is None and _aware(before.decision_deadline) < now + [entry] = db_session.execute(select(AuditLogEntry).where(AuditLogEntry.tenant_id == tenant.id, + AuditLogEntry.action == "continuity.gap")).scalars().all() + assert "across a cut-off" in entry.summary and entry.metadata_json["moved"] == 1 and entry.metadata_json["items_due_in_gap"] == 2 + assert db_session.execute(select(SecurityEvent).where(SecurityEvent.name == "continuity.gap", SecurityEvent.tenant_id == tenant.id) + ).scalars().first().severity == "high" + told = db_session.execute(select(Notification).where(Notification.tenant_id == tenant.id, + Notification.notification_type == NotificationType.CONTINUITY_GAP)).scalars().all() + assert told and all("due again" in n.body for n in told) and any(n.recipient_user_id == users["admin"].id for n in told) + check = control_checks.continuity_gaps(db_session, tenant.id) + assert check.status == "FAIL" and "across a cut-off" in check.detail + + +def test_a_sweep_after_a_normal_interval_moves_nothing(db_session, tenant_factory): + tenant, account, _users = tenant_factory.make(slug="gap-none") + item = _item(db_session, tenant, account, number="7101", due_in=timedelta(minutes=-1)) + _last_sweep(db_session, timedelta(minutes=5)) + assert continuity_service.recover_from_gap(db_session) == [] + db_session.expire_all() + assert item.deadline_moved_reason is None and _aware(item.decision_deadline) < datetime.now(timezone.utc) + assert control_checks.continuity_gaps(db_session, tenant.id).status == "PASS" + assert continuity_service.gap_before(db_session, datetime.now(timezone.utc)) is None + + +def test_the_first_sweep_ever_isnt_a_gap(db_session, tenant_factory): + tenant, account, _users = tenant_factory.make(slug="gap-first") + _item(db_session, tenant, account, number="7201", due_in=timedelta(minutes=-1)) + assert continuity_service.recover_from_gap(db_session) == [] diff --git a/tests/test_web/test_continuity_snapshot.py b/tests/test_web/test_continuity_snapshot.py new file mode 100644 index 00000000..643cd5f8 --- /dev/null +++ b/tests/test_web/test_continuity_snapshot.py @@ -0,0 +1,77 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""The bank's open items as a signed file (CONTINUITY_PLAN.md BC-D5): written hourly, a dated copy +at the end of the bank's day, emailed to the incident contacts when asked, downloadable from +Settings → Continuity by the bank's staff and not by a customer's people.""" + +import csv +import io +from datetime import datetime, timezone + +from sqlalchemy import select + +from openclear.domain.notification import Notification, NotificationType +from openclear.domain.tenant import Tenant +from openclear.services import continuity_snapshot, evidence_service, incident_service +from tests.test_decisions.test_changeable import _item +from tests.test_web.test_staff_and_operator import _login + + +def test_the_snapshot_lists_open_items_signed_and_keeps_a_daily_copy(db_session, tenant_factory): + tenant, account, _users = tenant_factory.make(slug="snap") + tenant.time_zone = "America/Chicago" + _item(db_session, tenant, account, number="4101") + _item(db_session, tenant, account, number="4102") + incident_service.set_contacts(db_session, tenant.id, [incident_service.ContactInput("contact", "Ira", "ira@snap.example")], user_id=None) + tenant.continuity_snapshot_emailed = True + db_session.commit() + + midday = datetime(2026, 10, 31, 17, 0, tzinfo=timezone.utc) # 12:00 in Chicago: the hourly file, no dated copy yet + first = continuity_snapshot.write(db_session, tenant, midday) + assert first.rows == 2 and first.dated_ref is None and first.emailed == [] + rows = list(csv.DictReader(io.StringIO(continuity_snapshot.read(tenant.id).decode()))) + assert [r["item"] for r in rows] == ["4101", "4102"] and rows[0]["network"] == "check" and rows[0]["decide_by"] + described = continuity_snapshot.latest(tenant.id) + assert evidence_service.verify(described) and described["bundle"]["rows"] == 2 and described["bundle"]["sha256"] == first.sha256 + + evening = datetime(2026, 11, 1, 4, 30, tzinfo=timezone.utc) # 23:30 CDT in Chicago on the 31st: the day's copy, emailed + second = continuity_snapshot.write(db_session, tenant, evening) + db_session.commit() + assert second.dated_ref is not None and second.emailed == ["ira@snap.example"] + assert continuity_snapshot.dated(tenant.id) == ["open-items-2026-10-31.csv"] + [mail] = db_session.execute(select(Notification).where(Notification.notification_type == NotificationType.CONTINUITY_SNAPSHOT)).scalars().all() + assert mail.destination == "ira@snap.example" and "4101" in mail.body and "decide by" in mail.body + assert continuity_snapshot.write(db_session, tenant, evening).dated_ref is None # once a day + + +def test_the_settings_section_and_the_download(client, db_session, tenant_factory): + from openclear.services import security_group_service, user_service + from tests.conftest import TenantFactory + from tests.items import _make_customer_with_account + + tenant, account, users = tenant_factory.make(slug="snap-web") + _item(db_session, tenant, account, number="4201") + continuity_snapshot.write(db_session, tenant) + db_session.commit() + csrf = _login(client, tenant.slug, users["admin"].email) + section = client.get("/ui/settings?from=continuity") + assert section.status_code == 200 + got = client.get("/ui/settings/continuity/open-items.csv") + assert got.status_code == 200 and got.headers["content-type"].startswith("text/csv") and "4201" in got.text + assert client.get("/ui/settings/continuity/open-items.csv?day=2020-01-01").status_code == 404 + # Only a date names a dated copy: nothing else the browser sends reaches the file name. + assert client.get("/ui/settings/continuity/open-items.csv?day=../../open-items-latest").status_code == 404 + assert client.get("/ui/settings/continuity/open-items.csv?day=latest").status_code == 404 + saved = client.post("/ui/settings/continuity", data={"csrf_token": csrf, "emailed": "true"}, follow_redirects=False) + assert saved.status_code in (200, 303) + db_session.expire_all() + assert db_session.get(Tenant, tenant.id).continuity_snapshot_emailed + + customer, _acct = _make_customer_with_account(db_session, tenant, "SNAP-C") + group = security_group_service.get_security_group_by_name(db_session, tenant.id, "Admin") + person = user_service.create_user_with_membership(db_session, tenant.id, email="cust@snap-web.example.com", password=TenantFactory.PASSWORD, + security_group_id=group.id, customer_id=customer.id) + db_session.commit() + _login(client, tenant.slug, person.email) + assert client.get("/ui/settings/continuity/open-items.csv").status_code == 404 # the bank's whole list isn't a customer's diff --git a/tests/test_web/test_maintenance_mode.py b/tests/test_web/test_maintenance_mode.py new file mode 100644 index 00000000..bec931e7 --- /dev/null +++ b/tests/test_web/test_maintenance_mode.py @@ -0,0 +1,52 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""Maintenance mode (CONTINUITY_PLAN.md BC-D4): sign-in and reading work, writes answer 503 with +the banner's words (the data export excepted), the cut-off sweep pauses and the pause counts as a +gap when the mode ends, and the Controls page says so.""" + +from datetime import datetime, timedelta, timezone + +from openclear.config import get_settings +from openclear.domain.scheduled_job_run import ScheduledJobRun +from openclear.services import continuity_service, controls_service +from tests.test_web.test_staff_and_operator import _login + + +def test_reads_and_sign_in_work_and_writes_are_refused(client, db_session, tenant_factory, monkeypatch): + tenant, _account, users = tenant_factory.make(slug="maint") + monkeypatch.setattr(get_settings(), "maintenance_mode", True) + csrf = _login(client, tenant.slug, users["admin"].email) # signing in is allowed + page = client.get("/ui/exceptions") + assert page.status_code == 200 and "OpenClear is being restored" in page.text # reading works, with the banner + refused = client.post("/ui/settings/incident-contacts", data={"csrf_token": csrf, "contact_0_name": "x", "contact_0_email": "x@y.example"}) + assert refused.status_code == 503 and "being restored" in refused.text and refused.headers["Retry-After"] == "600" + api = client.post("/api/v1/decision-batches", headers={"Authorization": "Bearer nope"}, json={}) + assert api.status_code == 503 and "being restored" in api.json()["detail"] + export = client.post("/ui/settings/data-export", data={"csrf_token": csrf}, follow_redirects=False) + assert export.status_code != 503 # the one write that must still work + assert controls_service._maintenance_mode().status == "WARN" + monkeypatch.setattr(get_settings(), "maintenance_mode", False) + assert controls_service._maintenance_mode().status == "PASS" + + +def test_the_sweep_pauses_and_the_pause_is_a_gap_afterwards(db_session, tenant_factory, monkeypatch, session_factory): + from openclear.workers import tasks + + tenant, account, _users = tenant_factory.make(slug="maint-gap") + monkeypatch.setattr(tasks, "get_session_factory", lambda: session_factory) + monkeypatch.setattr(get_settings(), "maintenance_mode", True) + tasks.sweep_expired_dispositions_job() + db_session.expire_all() + marker = db_session.get(ScheduledJobRun, continuity_service.PAUSED_MARKER) + assert marker is not None + started = marker.last_started_at.replace(tzinfo=timezone.utc) if marker.last_started_at.tzinfo is None else marker.last_started_at + tasks.sweep_expired_dispositions_job() # still paused: the marker keeps its first time + db_session.expire_all() + assert db_session.get(ScheduledJobRun, continuity_service.PAUSED_MARKER).last_started_at.replace(tzinfo=timezone.utc) == started.replace(tzinfo=timezone.utc) + + monkeypatch.setattr(get_settings(), "maintenance_mode", False) + now = datetime.now(timezone.utc) + timedelta(minutes=1) + assert continuity_service.gap_before(db_session, now) == (started, now) # the pause, as a gap + assert db_session.get(ScheduledJobRun, continuity_service.PAUSED_MARKER) is None # once + assert continuity_service.gap_before(db_session, now) is None diff --git a/tests/test_web/test_readiness.py b/tests/test_web/test_readiness.py new file mode 100644 index 00000000..03fa5145 --- /dev/null +++ b/tests/test_web/test_readiness.py @@ -0,0 +1,48 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 Chaffed + +"""`/ready` and `openclear_component_up` (CONTINUITY_PLAN.md BC-D6): the database and storage +make the instance unavailable when they fail; Redis and the sweep only degrade it.""" + +from datetime import datetime, timedelta, timezone + +from openclear import storage +from openclear.domain.scheduled_job_run import ScheduledJobRun +from openclear.services import continuity_service, readiness + + +def test_ready_names_each_component(client, db_session): + answer = client.get("/ready") + assert answer.status_code == 200 + body = answer.json() + assert body["status"] == "ready" and body["components"]["database"]["ok"] and body["components"]["storage"]["ok"] + assert body["components"]["scheduler"] == {"ok": True, "detail": "no run recorded yet"} + assert "redis" not in body["components"] # limits are in the process here + + +def test_storage_that_cant_be_written_makes_the_instance_unavailable(client, monkeypatch): + def broken(area, key, data, **kw): + raise OSError("read-only file system") + + monkeypatch.setattr(storage, "put", broken) + answer = client.get("/ready") + assert answer.status_code == 503 and answer.headers["Retry-After"] == "30" + assert answer.json()["status"] == "unavailable" and "can't write" in answer.json()["components"]["storage"]["detail"] + + +def test_a_stale_sweep_only_degrades(client, db_session): + db_session.merge(ScheduledJobRun(job_id=continuity_service.SWEEP_JOB_ID, last_finished_at=datetime.now(timezone.utc) - timedelta(hours=2), + last_ok=True)) + db_session.commit() + answer = client.get("/ready") + assert answer.status_code == 200 and answer.json()["status"] == "degraded" + assert "minutes ago" in answer.json()["components"]["scheduler"]["detail"] + found = readiness.components(db_session) + assert readiness.status(found) == "degraded" and [c.name for c in found] == ["database", "storage", "scheduler"] + + +def test_the_metrics_carry_each_component(client, db_session, tenant_factory): + from openclear import metrics + + text = metrics.render(db_session) + assert 'openclear_component_up{component="database"} 1' in text and 'openclear_component_up{component="storage"} 1' in text