feat: add seeded snapshots, backups, secret rotation and CD

[thegreatfeez]

closes #147,
closes #146,
closes #145,
closes #144.

All four items from issues.md were implemented in OracleDesk-Backend. The
seed/snapshot workflow is deterministic and self-verifying, backups have a
checksummed manifest and a restore-time chain reconcile, secrets load from
env/files/Vault and rotate without a restart, and the CD pipeline builds a
digest-pinned signed image with migration compatibility checks and post-deploy
verification.

issues.md item	GitHub issue
Seed data + anonymised snapshot	#147
Backup/restore runbook + drill	#146
Secrets management + hot reload	#145
Continuous delivery	#144

---

#147 — Deterministic seed and anonymised snapshot

• src/seed/scenario.ts builds a repeatable dataset from a fixed PRNG:
  45 markets across every category/status, 12 users, trades, positions, traces,
  copy trades, subscriptions, payments, allowances, chain events, logs, signal
  cache and generation jobs. Same seed → byte-identical output.
• Every hash uses the production helpers: questionHash over the canonical
  question document, resolutionHash over the XDR spec, and traceHash over
  the canonical trace payload. A test recomputes all three.
• IPFS_TRANSPORT=local now selects a disk-backed transport. Seeded question
  and trace bytes are written under .ipfs-local/<cid> using a dependency-free
  CIDv1(raw, sha2-256). Config refuses local in production.
• prisma/seed.ts refuses NODE_ENV=production, CHAIN_EXECUTION_MODE=live
  and non-local database hosts (unless SEED_ALLOW_REMOTE=1). Re-runs use
  skipDuplicates; --reset truncates first. --now, --from-snapshot,
  --link-testnet and --with-raw-events are supported.
• --with-raw-events re-encodes Trade, MarketCreated, MarketResolved and
  TracePublished into the recorded getEvents shape; a test decodes every
  encoded event with the contract spec.
• scripts/snapshot-export.ts writes anonymised NDJSON. src/lib/snapshot.ts
  holds a fail-closed column allowlist (a new column stops the export), maps
  wallet addresses to deterministic throwaway G-addresses, keyed-hashes tx
  hashes, drops log payloads/errors and scans the output for secret patterns.
  signal_cache is explicitly excluded.
• prisma.config.ts wires tsx prisma/seed.ts; npm run db:seed and
  npm run snapshot:export are in package.json.

#146 — Backup/restore plus reconcile

• scripts/backup.sh takes a custom-format pg_dump, .sha256, and a manifest
  with migrations, indexer cursor(s), max chain ledger, newest payment time and
  row counts. --critical-only writes NDJSON; --encrypt-to uses age.
  Retention is controlled by BACKUP_KEEP_DAILY / BACKUP_KEEP_WEEKLY.
• scripts/restore.sh verifies the checksum, refuses a non-empty target unless
  --force, runs pg_restore, applies prisma migrate deploy and prints the
  manifest.
• src/services/reconcile.service.ts (CLI:
  scripts/post-restore-reconcile.ts) rewinds the cursor to the manifest
  boundary minus 720 ledgers, clamps to RPC retention and reports the gap,
  calls backfillEvents, re-projects every surviving event at/after the
  boundary, re-verifies CONFIRMED payments without aborting on retention loss,
  checks every resolution preimage, and optionally reports unclaimed recipient
  transfers. It exits non-zero only for missing/mismatched specs.
• src/services/indexer.service.ts exports reprojectEvent and
  decodedFromStored; trace-hash handling now accepts both Buffer and the
  hex form stored in Postgres.
• .github/workflows/restore-drill.yml seeds, backs up, drops the schema,
  restores, and reconciles against tests/fixtures/events.json monthly.
• The runbook is docs/runbooks/backup-restore.md; Compose has an optional
  backup profile.

#145 — Secrets loading and hot reload

• src/config/secrets.ts resolves <NAME>_FILE, validates at boot and refresh
  with the same rules, stores raw values as Buffers, exposes secret() /
  secretEntry() / previousSecret(), and refreshes on an interval with
  jitter. Empty or invalid refresh values keep the old value; provider failures
  keep everything.
• Providers: env (default), file (mounts re-read on refresh) and Vault KV v2
  (token from a file, never the environment). The interface allows AWS Secrets
  Manager to be added without touching consumers. SECRETS_PROVIDER,
  SECRETS_REFRESH_MS, JWT_PREVIOUS_SECRET_GRACE_SECS, VAULT_* are
  validated in src/config/index.ts.
• DATABASE_URL_FILE is resolved before Zod and in prisma.config.ts, so both
  the app and Prisma CLI work with a mounted URL. It is set on the process
  before the Prisma client is constructed.
• JWT rotation signs with the current key, accepts the previous key only inside
  the grace window, and never retries an expired token. The retired Buffer is
  zeroed. Gemini and Anthropic clients rebuild when their key version changes;
  IPFS and ingestion read at call time.
• Boot calls initSecrets and startSecretRefresh; failure exits with the
  variable named. Tests cover files, dual-source rejection, empty files, short
  JWTs, immutable/empty/failed refreshes, previous-key acceptance and expiry,
  and client rebuilds.

#144 — Continuous delivery

• .github/workflows/cd.yml verifies via CI, builds and pushes a digest,
  keylessly signs it with cosign, emits an SPDX SBOM, deploys staging then
  production through the reusable deploy-env.yml, and runs post-deploy checks
  after each.
• deploy-env.yml verifies the cosign identity, lints only the migrations
  changed in that release, applies prisma migrate deploy before Helm, and
  runs helm upgrade --install --atomic by image digest. Secrets come from
  GitHub Environments.
• scripts/check-migration-compat.mjs flags drops, renames, type changes,
  SET NOT NULL, and non-concurrent indexes. Tests cover breaking and safe SQL;
  CI runs it for newly added migrations only (historical migrations predate the
  rule).
• scripts/post-deploy-checks.sh requires readiness to hold N consecutive
  probes, compares the deployed image digest with the release digest, checks
  indexer lag and cursor advancement (status endpoint or Postgres + RPC), and
  checks Prometheus critical alerts and 5xx ratio. Tests cover pass and
  readiness failure.
• CI now runs actionlint and shellcheck. /readyz was added for readiness.
  docs/deployment.md documents the pipeline, migration direction and rollback.

---

Verification

Check	Result
npm run build	✅
npm test	✅ 50 tests passed
npx tsc --noEmit -p tsconfig.test.json	✅
CLI scripts (snapshot-export, post-restore-reconcile, prisma/seed) typecheck	✅
npx prisma validate	✅
actionlint .github/workflows/*.yml	✅
shellcheck scripts/*.sh	✅
Historical migration baseline	⚠️ predates this change; CI/CD lint only new migration files

Live Postgres/Redis were not available in this environment, so the seed write,
pg_dump/pg_restore and the hosted workflows were exercised through unit
tests and mocks rather than an end-to-end run. The restore drill is wired to
run them in CI.

Judgement calls

• Default branch. issues.md targets develop, but this checkout has only
  main (origin/HEAD and the sole local/remote branch). Per the working
  instructions, the work is committed on main.
• Remote provider (Load service secrets from files and a secret manager with rotation support #145). Vault KV v2 is implemented rather than AWS
  Secrets Manager; the SecretsProvider interface keeps AWS additive. Vault
  supplies secrets after loadConfig, so DATABASE_URL still has to arrive by
  env or file at boot (Prisma builds its client at import). This is documented.
• Agent key (Load service secrets from files and a secret manager with rotation support #145). AGENT_SECRET_KEY is intentionally outside the hot
  store: rotating it is an on-chain registration change, so it requires a
  restart. Values already copied into SDK objects cannot be zeroed; only the
  long-lived store copy is.
• Historical migrations (Build a continuous delivery pipeline with staged rollout and automatic rollback #144). The repository already contains destructive
  migrations. The linter therefore runs on the migration files changed by a PR
  or release, which matches “fail a PR that adds a destructive migration”
  without blocking every deploy on history.
• Backup retention (Define backup, restore and post-restore indexer resynchronisation procedures #146). Retention keeps the newest daily/weekly counts
  rather than implementing a calendar-aware grandfather-father-son scheme; the
  variables and runbook expose the policy.
• Restore drill RPC (Define backup, restore and post-restore indexer resynchronisation procedures #146). RECONCILE_FAKE_RPC reads the existing recorded
  event fixture. The reconcile command still uses real RPC when that variable
  is absent.
• Local CID (Add database seeding and anonymised snapshot tooling for realistic local data #147). The local transport computes CIDv1(raw, sha2-256)
  directly instead of adding an IPFS dependency; pinJson still hashes the
  same canonical bytes.
• Helm chart (Build a continuous delivery pipeline with staged rollout and automatic rollback #144). The chart belongs to the sibling infrastructure work;
  the deploy workflow expects deploy/helm/oracledesk-backend and fails clearly
  if it is missing.

[drips-wave]

@thegreatfeez Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[github-actions]

👋 Thanks for the contribution! This repository only accepts contributor pull requests into the develop branch. PRs raised against main cannot be merged, so this one has been closed automatically.

Please re-open your PR with base: develop (rebase your branch onto develop first if needed). See the note at the bottom of the issue you're working on.