Add SQLite as a second database backend alongside PostgreSQL

[felixgateru]

Summary

Full delivery of the database-backends initiative described in product-docs/development/database-backends/ (PRD/RFC/ROADMAP), as a single PR per the documented delivery model. Atom now runs on PostgreSQL or SQLite, selected by the DATABASE_URL scheme, with identical application behavior on both.

Architecture note (departs from the RFC's per-domain dual-adapter design — see product-docs/development/database-backends/IMPLEMENTATION-NOTES.md for the full rationale): rather than hand-porting every repository twice, all ~700 PostgreSQL-specific SQL constructs in the codebase are now routed through one backend-neutral query layer (crate::db::{query, query_as, query_scalar, QueryBuilder}). SQL is written once, in the PostgreSQL dialect:

• PostgreSQL runs the text unchanged — zero behavior change, confirmed by running the full pre-existing integration suite unmodified.
• SQLite runs a cached, mechanical translation (src/db/translate.rs) of the same text — arrays ↔ JSON, jsonb operators, interval arithmetic, LATERAL, ANY/ALL, data-modifying CTEs, etc. — or an explicit .sqlite("…")/.sqlite_all([...]) override where a rewrite isn't mechanical (a few dozen call sites).

What's in this PR

• DB-001/002 — facade & transaction boundary (prior commits): Database/DbTransaction/DbConn replace PgPool/sqlx::Transaction everywhere; DatabaseErrorKind classification.
• Query layer: every repository/service call goes through crate::db; scripts/check-db-boundary.sh (new CI step) fails the build if a raw sqlx::query* or driver type (PgPool, SqliteConnection, …) appears outside src/db/.
• SQLite runtime: src/db/sqlite.rs — WAL, synchronous=FULL, foreign_keys=ON, 30s busy timeout, BEGIN IMMEDIATE transactions, single-owner file lock (a second Atom process refuses to start against the same file), busy/locked errors map to HTTP 503 / gRPC UNAVAILABLE.
• SQLite baseline schema: migrations/sqlite/001_initial.sql mirrors migrations/001_initial.sql table-for-table (and 002_refresh_tokens.sql mirrors the refresh-token migration from main). Tables/indexes/views/CHECK constraints use only SQLite built-ins (so any SQLite tool — sqlite3, VACUUM INTO, backup agents — can open and copy the database); the schema's invariant triggers (protected-object registry, PKI parentage/ceilings, revocation immutability, shared-key rules, …) call a small set of atom_* SQL functions the app registers on every connection (src/db/sqlite_functions.rs, via libsqlite3-sys).
• Value encodings: UUID = 16-byte BLOB, timestamp = fixed-width RFC 3339 text (microseconds, Z), JSON = TEXT + json_valid, arrays = JSON text, bool = INTEGER.
• Error classification: error::{is_unique_violation, is_foreign_key_violation, is_check_violation} replace ad hoc SQLSTATE string comparisons everywhere (including in tests), so the same code path works on both backends' error codes.
• Dual-backend test suite: every DB-gated integration binary (63 of them) and the in-crate #[ignore]d unit tests run against both backends — cargo test -- --include-ignored (PostgreSQL, default) and ATOM_TEST_BACKEND=sqlite cargo test -- --include-ignored (SQLite, no external service — each test process gets a fresh file). CI runs both lanes. tests/sqlite_operations.rs covers what has no PostgreSQL analogue: fixed PRAGMAs, restart persistence, single-owner lock, rollback atomicity, contention/busy mapping, backup-and-restore.
• Operator docs: docs/content/docs/operations/sqlite.mdx — configuration, durability policy, backup/restore, behavioral differences, when to choose which backend.
• AGENTS.md updated with the two-backend invariants (query-layer-only rule, migration pairing, built-ins-only schema, no second pool connection under a transaction).

Merged current main (rotating refresh tokens, #107) into this branch; migrations/sqlite/002_refresh_tokens.sql added as its SQLite counterpart and the new src/identity/refresh_tokens.rs / m51_refresh_tokens.rs ported onto the query layer — all 12 refresh-token tests pass on both backends, including the concurrent-exchange and lock-ordering ones.

Test plan

• cargo fmt --check
• cargo clippy --locked -- -D warnings
• scripts/check-db-boundary.sh (driver-type boundary + PostgreSQL/SQLite schema parity)
• scripts/check-v1-contracts.sh
• Full workspace compiles: lib, bin, all integration test binaries
• cargo test --lib — 420 tests pass on both PostgreSQL and SQLite (--include-ignored)
• All 65 DB-gated integration test binaries (every one except m27_live_amqp_delivery/m28_amqp_mtls_local_principal, which need a real AMQP broker CI doesn't provision either) pass on both PostgreSQL and SQLite, each test binary against a freshly created/migrated database — including m41_pki_est (needs estclient, installed locally to verify) and m42_pki_pkcs11 (PostgreSQL only by design: it proves the HSM key backend, independent of the storage engine)
• tests/sqlite_operations.rs (SQLite-only operational contract) — 10/10
• Manual end-to-end smoke: booted the binary against a fresh SQLite file, applied the demo bootstrap, logged in, created a tenant via GraphQL, restarted the process, confirmed the tenant persisted

One CI run caught two real gaps this local matrix had missed (both fixed, both re-verified locally end to end before the fix commit, and now covered by regression tests): a json_each-scoped array_position translation that silently matched nothing rather than raising an error, masked by simpler test cases; and m41_pki_est, excluded from earlier local runs for lacking the estclient binary, which hit an SQLite-only generate_series-in-SELECT-list gap plus an unrelated SQLite INSERT...SELECT...ON CONFLICT grammar ambiguity. Both are now documented in AGENTS.md's two-backend invariants.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	atom-docs	8bf1d7c	Commit Preview URL Branch Preview URL	Sep 23 2026, 08:51 AM

[arvindh123]

Please revise this PR to use domain-level repository contracts, following the pattern used by Magistrala's alarms service, instead of making a PostgreSQL-to-SQLite query translator the main database abstraction.

The requested boundary is service -> repository contract -> backend implementation. We do not want an interface around every SQL/helper function or one giant repository interface.

Reference: Magistrala alarms

The alarms service receives a Repository interface exposing domain operations such as CreateAlarm, ViewAlarm, and ListAllAlarms; the PostgreSQL repository implements those operations and owns its SQL:

• Repository contract
• Service and dependency injection
• PostgreSQL implementation

These links use the historical revision before the service moved to EE.

Requested Rust design

1. Define small repository traits grouped by domain/use case: identity, credentials/sessions, authorization, resources, tenants, certificates, etc. Expose the operations callers need, using domain inputs/results rather than raw SQL, bind arguments, or driver-specific rows.
2. Implement those contracts separately for PostgreSQL and SQLite. Each implementation owns its native SQL, backend-specific queries, and error conversion. Keep business rules in shared services/engines; do not duplicate authorization evaluation in the adapters.
3. Select and inject the implementation at startup from DATABASE_URL. Arc<dyn Repository + Send + Sync> with async_trait is one suitable option; generics or enum dispatch are also acceptable if they preserve this domain boundary. The request is about the abstraction boundary, not requiring a particular dispatch mechanism.
4. Replace reliance on the general runtime SQL translator with explicit backend implementations. Shared internal helpers and connection/transaction wrappers are fine where they are useful, but services should call domain operations rather than assemble SQL through a generic compatibility API.

An illustrative layout (adapt the grouping to Atom's actual callers):

src/identity/
    service.rs
    repository.rs       # domain contracts and input/result types
    postgres/
        entities.rs
        credentials.rs
    sqlite/
        entities.rs
        credentials.rs

Design the transaction boundary first

Atom's mutations can span multiple domains and must publish their outbox event atomically. Do not make every repository method open and commit an independent transaction.

Choose an explicit shared transaction/unit-of-work design, or operation-level storage methods that own the entire atomic operation. Preserve:

• One transaction/connection for all writes in an operation, including its outbox row.
• Existing audit/observation commit semantics: transactional outbox, with the appropriate audit/log handling after commit.
• Nested savepoints where certificate issuance retries require them.
• No second pool acquisition while a transaction holds a connection.
• One canonical grant expansion per backend, reused by the PDP, control-plane gates, and authorized listings; no separate listing evaluator or per-reader grant expansion.

Suggested implementation sequence and acceptance criteria

• Document the proposed repository and transaction contracts, then implement one representative operation that exercises both a read and an atomic multi-write mutation on both databases.
• Port the remaining storage operations to those contracts, keeping transport/service behavior and API contracts unchanged.
• Keep PostgreSQL/SQLite driver details and native SQL within their adapters; retain paired migrations and SQLite's documented durability/single-owner behavior.
• Run the same behavioral contract/integration tests against both implementations, including authorization/listing parity, search/sorting/null semantics, rollback, savepoints, and mutation + outbox atomicity.
• Retain the SQLite operational tests and PostgreSQL regression coverage; update architecture documentation and boundary checks to match the final design.
• Remove the superseded translator/query compatibility machinery once callers have migrated.

This deliberately accepts some duplication of storage queries in exchange for explicit, independently testable backend implementations. Shared business logic should remain shared. Neon can normally reuse the PostgreSQL adapter; it does not need a separate repository implementation merely because it is a different PostgreSQL hosting provider.