docs(ops): record the Postgres 16 → 18 upgrade, add db:reindex-trgm

[gilgardosh]

Production moved from PostgreSQL 16.14 to 18.6 on 2026-09-02. This PR lands the tooling and the record — the schema and query changes shipped earlier in #4326, #4327, #4329, #4330 and #4331.

yarn db:reindex-trgm

scripts/reindex-trigram-indexes.ts rebuilds every GIN trigram index, always CONCURRENTLY.

Two design points worth reviewing:

• It discovers indexes from pg_index/pg_opclass at runtime rather than from a hardcoded list. The set grew from six to seven when idx_financial_entities_name_trgm landed in perf(migrations): add GIN trigram index on financial_entities.name #4330, and any copied list silently skips the new one. Discovery makes staleness impossible.
• CONCURRENTLY is not optional. Only that seventh index was originally built concurrently; the other six came from a plain CREATE INDEX IF NOT EXISTS. A plain REINDEX would take ACCESS EXCLUSIVE on charges, transactions and documents.

It also reports the collation provider (which decides whether the reindex is required at all), detects invalid indexes and _ccnew leftovers from a failed rebuild, disables statement_timeout for the session, times each index, and re-reads the catalog afterwards rather than trusting the absence of errors. 29 unit tests; validated against Azure 18.6 during the rehearsal.

The upgrade, measured

Rehearsal on a PITR-restored copy	17m 54s, all 11 checks passed
Production upgrade	16.14 → 18.6
ANALYZE afterwards	5.2s — Azure-mandatory, ran immediately
datcollversion	2.38 → 2.38, no drift (same base image), so no text-index reindex
Trigram reindex	not required (datlocprovider = 'c'), not run
Catalog diff, same server before/after	all 7 security-relevant sections byte-identical
Plan diff	no regressions; skip scan now live
Trigram index sizes	roughly halved by the upgrade; DB 105 → 99 MB

Azure's post-upgrade ADMIN-option privilege change altered nothing — RLS coverage stayed 101/60/60, ownership unchanged, security_invoker still unset on all views, and the prod_group → accounter_prod_user INHERIT TRUE grant intact. That grant is why FORCE ROW LEVEL SECURITY is load-bearing here, so it was the thing most worth verifying.

Measured benefit

PG18 B-tree skip scan is live on two scraper dedup queries. poalim_securities_transactions_dedup_uindex is now chosen with value_date (7th column) in the Index Cond while trade_date (6th) has no predicate — the definitional signature of skip scan. Rows Removed by Filter went from 262/261 to zero. These run per ingested row, so the benefit grows with the tables.

Findings kept in the docs

None of these are reconstructable from the code, which is the main argument for committing the docs at all:

• Under RLS, a non-leakproof qual cannot become an index condition. So a composite index whose non-leading column is an expression is unusable by the application. That is why two of perf(server): make date-range filters sargable #4331's four indexes are never chosen (Two of #4331's indexes are unusable under RLS (COALESCE predicate demoted to Filter) #4341) while the two plain-column ones work fine. Reproduced on 18.6 — the upgrade does not fix it; a STORED generated column does.
• CREATE INDEX does not populate expression-index statistics, so ANALYZE must run before capturing plans. Measured 30× misestimates otherwise, which read exactly like a planner regression. This is now the loudest warning in the command sheet.
• A PITR-restored server arrives with no firewall rules and max_connections pending restart — the latter fails the upgrade precheck outright. Both matter because PITR-restore-to-a-new-server is the only rollback after a successful upgrade.
• Production's schema has drifted from what the migrations build: 101 tables versus 99, plus three column type/nullability differences. Unread by code, but it means a rebuild from migrations is not equivalent to production.

On sensitive data

Raw EXPLAIN captures are deliberately not committed. They contained real bank account numbers, branch numbers and security identifiers. They were deleted after use; every conclusion drawn from them is in the docs.

capture-plans.psql section 3 previously printed sample rows, which is how that data reached a file — it now reports counts only, so a re-run cannot reproduce the problem. Hostnames are templated to $PROD_HOST / $RESTORED_HOST and business UUIDs to $OWNER_A / $OWNER_B.

What is committed from the run: the two catalog snapshots, whose only numbers are row counts.

Verification

prettier clean, 29 tool tests pass, package.json diff is exactly one line (db:reindex-trgm) on top of current main — including #4202's changesets upgrade, which an earlier rebase attempt would have silently reverted.

🤖 Generated with Claude Code

[github-actions]

The latest changes of this PR are not available as alpha, since there are no linked changesets for this PR.

[Copilot]

🟡 Changes recommended

The new CLI has a couple of confirmed safety/correctness issues (arg parsing edge cases, brittle “run directly” detection, and missing remote-target opt-in guard) and the rehearsal command sheet contains a hardcoded personal path / inconsistent “uncommitted” wording.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds an operator-facing Postgres maintenance tool and documents the completed Postgres 16 → 18 production upgrade, including runbook and evidence snapshots, so future upgrades/rehearsals can be repeated safely and verified against known-good baselines.

Changes:

• Added yarn db:reindex-trgm tool (REINDEX INDEX CONCURRENTLY for all trigram indexes discovered at runtime) with unit tests.
• Added/updated Postgres 18 migration audit + an executed upgrade runbook, plus baseline capture scripts and before/after catalog snapshots.
• Wired the new CLI into root package.json and recorded the change via a changeset.

File summaries

File	Description
scripts/reindex-trigram-indexes.ts	New CLI to discover and concurrently reindex all trigram indexes; prints safety/verification info.
scripts/tests/reindex-trigram-indexes.test.ts	Unit tests covering SQL generation, quoting, arg parsing, discovery mapping, and formatting helpers.
package.json	Adds db:reindex-trgm script entry to run the new tool via tsx.
docs/postgresql-v18-migration.md	Full audit/plan doc updated to reflect the v18 migration work and where the runbook lives.
docs/operations/postgres-18-upgrade.md	Operational runbook with an execution record for the 2026-09-02 production upgrade.
docs/operations/postgres-18-baselines/rehearsal-command-sheet.md	Step-by-step rehearsal command sheet used to validate the upgrade on a PITR-restored server.
docs/operations/postgres-18-baselines/README.md	Explains the baselines directory purpose, redaction rules, and key findings.
docs/operations/postgres-18-baselines/pre-upgrade-catalog.psql	PSQL script to snapshot catalog/security invariants and index validity pre/post upgrade.
docs/operations/postgres-18-baselines/capture-plans.psql	PSQL script to capture plan baselines (counts-only + skip-scan/date-range queries) under RLS context.
docs/operations/postgres-18-baselines/16-catalog-snapshot.txt	Captured production catalog snapshot before upgrade (PG 16.14).
docs/operations/postgres-18-baselines/PROD18-catalog-snapshot.txt	Captured production catalog snapshot after upgrade (PG 18.6).
.changeset/postgres-18-upgrade-record.md	Changeset recording the new tool and the upgrade documentation landing.

Review details

• Files reviewed: 12/12 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.