Repository navigation
feat: add /live and /ready health probes - #544
Conversation
`/status` answers neither question a platform asks, and the two questions are not the same one. It reads `indexer_state`, so it fails when Postgres is unreachable — which makes it useless as a liveness signal — and it says nothing about whether the process has finished booting. `app.listen()` runs part-way through `main()`, before the ingesters, the retention worker and the websocket fan-out start. Between listening and those loops running, the process accepts requests it cannot yet answer. GET /live touches no dependency and stays healthy while Postgres is down or the app is still booting. That is the point: a liveness probe that fails during a database outage has the orchestrator restarting every instance at once, turning a recoverable outage into a cold-start storm. GET /ready reports 503 with ok:false until main() finishes booting, and while Postgres is unreachable or unmigrated, or Redis is unreachable when Redis is part of readiness (READINESS_REDIS_CHECK, default true). The Postgres check is a single to_regclass query answering both reachability and whether migrations were applied — reachability alone would report a process with no tables as ready, and every route reading one would fail. Probes are bounded by READINESS_TIMEOUT_MS and answers cached for READINESS_CACHE_MS, so polling every few seconds costs at most one round trip per dependency per interval. node-postgres exposes no AbortSignal on query(), so a probe past its deadline reports down without cancelling the query underneath; the cache is what bounds how many can be outstanding. Both probes are public, allow-listed from rate limiting next to /status, sent Cache-Control: no-store, and excluded from http_requests_total — a fixed-interval poller would otherwise put a floor under the request rate that no amount of real traffic dropping could go below. /status is unchanged. The deployed smoke check now requires both probes instead of skipping /ready when absent, and fails on a failing /live.
|
@CollinsKRO Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
@fastify/rate-limit consults allowList before it reads max, so the per-route ceiling is inert while the allow-list stands.
|
Merged. You got the distinction that actually matters, and wrote down why.
The boot-state half is the subtler catch and you got that too: Also right to use tighter deadlines than the house Merge notes: README conflicted twice, both additive. Verified on the merge result rather than your branch: One thing to know: #553 (issue #352, database schema verification at startup) also adds a Closes #334. |
A GraphQL query could nest as deep as it liked, which is the standard denial-of-service shape for a public GraphQL endpoint: a short document expands into an enormous resolver tree. The implementation is careful in the ways that matter: - **Fragments are expanded.** Mercurius's built-in `queryDepth` does not, so a limit set with it is bypassed by moving the deep part into a named fragment. Here a fragment node adds no depth itself while its selected fields do, which is the correct accounting. - **Fragment cycles terminate.** A fragment already being expanded is not expanded again, so GraphQL's own cycle validation reports the cycle instead of this traversal recursing forever. - **Introspection is exempt only at the operation root**, so GraphiQL loads, but an operation that mixes `__schema` with application fields still has its application branch measured. That is exactly the hole a root-level exemption usually leaves open. - **A bad `GRAPHQL_MAX_QUERY_DEPTH` throws at startup** rather than falling back to unlimited, with the reasoning in a comment: a deployment typo must not silently become a security regression. Right instinct, and the inverse of what most config parsing does. - Rejection happens in validation, before any resolver runs, and the error names both the actual depth and the configured maximum. `.env.example` conflict resolved additively — the readiness-probe block from #544 landed while this was open, so both sections are kept. Verified on the merge result: `tsc --noEmit` silent, 896 tests pass across 83 suites, nothing red. Closes #360 Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
…key expiry
Three related things, all wanted:
- **Auth failures are now counted** — `admin_auth_failures_total` and
`api_key_auth_failures_total{reason}`, plus
`rate_limit_rejections_total{scope}` on the per-key/IP limiter. A rejected
request was previously invisible, so neither a misconfigured client nor
someone probing `ADMIN_TOKEN` showed up anywhere.
- **A per-instance request ceiling** that sheds with 503 and `Retry-After`
once this process is over `GLOBAL_RATE_LIMIT_MAX` in a rolling minute, with
expensive route templates shed at 80% of it and the probe paths exempt. This
is self-defence rather than a quota, so keeping it per-process and
in-memory is the right call — it protects *this* instance's resources.
- **`api_keys.expires_at`**, with admin key creation accepting an optional
future ISO date-time and `lookupApiKey` refusing an expired key.
`parseGlobalRateLimit` throws on an invalid value instead of falling back to a
default, which is the same instinct as #539's depth limit and correct for the
same reason: a config typo must not quietly mean "no ceiling".
Resolutions and fixes while merging:
- `src/api/usage.ts` re-added a local `isAdminAuthorized` — #517 removed
exactly that copy earlier today. Kept the shared one from `./auth` and the
new metric import.
- `src/app.ts`: kept `allowList: (req) => PROBE_PATHS.has(req.url)` from main
rather than this PR's `req.url === '/status'`, since `/live` and `/ready`
(from #544) are platform-polled and must not be limited either. Took the
PR's metric-emitting `errorResponseBuilder` and the global-ceiling
registration, and kept main's note on the probe-route ceiling.
- `.env.example` was missing `GLOBAL_RATE_LIMIT_MAX`, which the `envDrift`
guard caught. Documented, including that the fleet ceiling is this value
times the instance count by design.
- `npx prisma generate` was needed after the schema change to clear phantom
`expiresAt` errors.
⚠️ Deploy order: `api_keys.expires_at` must exist before this runs, because
`lookupApiKey` selects it — without the column, every authenticated request
fails, not just key creation. Lens has no migrations directory (schema-first
via `db push`), so:
ALTER TABLE "api_keys" ADD COLUMN IF NOT EXISTS "expires_at" TIMESTAMP(3);
Verified on the merge result: `tsc --noEmit` silent, 908 tests pass across 84
suites, nothing red.
Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
Closes #334
The gap
/statusanswers neither question a platform asks, and the two are not the same question:/live/ready/statusreadsindexer_state, so it fails when Postgres is unreachable — useless as a liveness signal — and it says nothing about boot state. That gap is real here:app.listen()runs part-way throughmain(), before the ingesters, the retention worker and the websocket fan-out start. Between listening and those loops running, the process accepts requests it cannot yet answer correctly.What was added
GET /live—200 { ok, uptimeSeconds }, touching no dependency. Reaching the handler is the signal. It stays healthy while Postgres is down and while the app is still booting, because a starting instance needs to be held out of rotation, not killed.GET /ready—200withok: truewhen it can serve,503withok: falseotherwise, plus a per-dependency breakdown:{ "ok": false, "booted": true, "checks": { "postgres": { "status": "down", "latencyMs": 2001, "reason": "unreachable or not migrated" }, "redis": { "status": "up", "latencyMs": 1, "reason": null } }, "checkedAt": "2026-10-19T10:00:00.000Z" }Unhealthy until
markBootComplete()runs at the end ofmain(). While not booted it probes nothing at all — no dependency check can make a not-yet-booted process ready, so polling it costs nothing.Acceptance criteria
/livereturns healthy without touching Postgres or Redis — asserted directly (mockQuery/mockPingnever called), and again with both dependencies rejecting, so a database outage provably cannot fail liveness on every instance at once./readyreturns unhealthy before boot completes and healthy after — 503 withbooted: falsebefore, 200 withbooted: trueaftermarkBootComplete()./readyreports unhealthy when Postgres is unreachable — plus a separate case for reachable-but-unmigrated, since connectivity alone would report a process with no tables as ready.httpMetricsroute cardinality —PROBE_PATHSinsrc/app.tsandEXCLUDED_ROUTESinsrc/middleware/httpMetrics.ts. Each has a control test so neither can pass vacuously: an ordinary route still 429s under the same limiter, and/statusstill gets counted./statusis unchanged — its allow-listing, itspublic, max-age=15policy and its shape are all asserted to be unaffected.Three decisions worth flagging
Redis in readiness, but optional. It is included by default because the snapshot retention worker and the aggregate refresh queue are Redis-backed and stop silently without it. But
main()already treats an unavailable Redis as non-fatal and serves prices uncached from Postgres — so on a deliberately Redis-less deployment, failing readiness would pull a healthy instance out of rotation.READINESS_REDIS_CHECK=falseopts out; the response reportsskippedrather than pretending it passed.Probes are bounded, but the timeout is shorter than the house 10s.
node-postgresexposes noAbortSignalonquery(), so the deadline is applied by racing. A 10s probe on a 5s poll interval would answer from the past, so the default is 2s. A probe that exceeds it reportsdownwithout cancelling the query underneath —READINESS_CACHE_MSis what bounds how many such queries can be outstanding, which is why answers are cached and concurrent polls share one in-flight run.Errors are not echoed in the response. A connection error carries the host and port, and these are public endpoints. The response carries a fixed reason phrase and the underlying error goes to the log. There is a test asserting no host, port or errno reaches the body.
Also updated
The deployed smoke check previously skipped
/readywhen absent, with a message pointing at this issue. A smoke check that silently tolerates a missing probe is how a deployment ends up unmonitored while still green, so it now fails on a 404 and also asserts/live. Both new probe paths are documented inopenapi.yaml/openapi.json(the OpenAPI coverage test enforces that), and written up indocs/health.md.npm test— 829 passed, 1 skipped (806 before, +23 new).npx tsc --noEmit,npm run buildandnpm run test:smokeall clean.