Skip to content

feat: add /live and /ready health probes - #544

Merged
Miracle656 merged 2 commits into
Divinity-Labs:mainfrom
CollinsKRO:feat/health-probes
Oct 19, 2026
Merged

Miracle656 merged 2 commits into
Divinity-Labs:mainfrom
CollinsKRO:feat/health-probes

Conversation

@CollinsKRO

Copy link
Copy Markdown
Contributor

Closes #334

The gap

/status answers neither question a platform asks, and the two are not the same question:

Question Cost of a wrong answer
/live Is this process wedged and in need of a kill? Restarting healthy processes turns a dependency blip into a fleet-wide cold-start storm
/ready Should this process receive traffic yet? Sending traffic to a process that cannot answer it

/status reads indexer_state, so it fails when Postgres is unreachable — useless as a liveness signal — and it says nothing about boot state. That gap is real here: app.listen() runs part-way through main(), before the ingesters, the retention worker and the websocket fan-out start. Between listening and those loops running, the process accepts requests it cannot yet answer correctly.

What was added

GET /live — 200 { ok, uptimeSeconds }, touching no dependency. Reaching the handler is the signal. It stays healthy while Postgres is down and while the app is still booting, because a starting instance needs to be held out of rotation, not killed.

GET /ready — 200 with ok: true when it can serve, 503 with ok: false otherwise, plus a per-dependency breakdown:

{
  "ok": false, "booted": true,
  "checks": {
    "postgres": { "status": "down", "latencyMs": 2001, "reason": "unreachable or not migrated" },
    "redis":    { "status": "up", "latencyMs": 1, "reason": null }
  },
  "checkedAt": "2026-10-19T10:00:00.000Z"
}

Unhealthy until markBootComplete() runs at the end of main(). While not booted it probes nothing at all — no dependency check can make a not-yet-booted process ready, so polling it costs nothing.

Acceptance criteria

  • /live returns healthy without touching Postgres or Redis — asserted directly (mockQuery/mockPing never called), and again with both dependencies rejecting, so a database outage provably cannot fail liveness on every instance at once.
  • /ready returns unhealthy before boot completes and healthy after — 503 with booted: false before, 200 with booted: true after markBootComplete().
  • /ready reports unhealthy when Postgres is unreachable — plus a separate case for reachable-but-unmigrated, since connectivity alone would report a process with no tables as ready.
  • Both excluded from rate limiting and from httpMetrics route cardinality — PROBE_PATHS in src/app.ts and EXCLUDED_ROUTES in src/middleware/httpMetrics.ts. Each has a control test so neither can pass vacuously: an ordinary route still 429s under the same limiter, and /status still gets counted.
  • /status is unchanged — its allow-listing, its public, max-age=15 policy and its shape are all asserted to be unaffected.

Three decisions worth flagging

Redis in readiness, but optional. It is included by default because the snapshot retention worker and the aggregate refresh queue are Redis-backed and stop silently without it. But main() already treats an unavailable Redis as non-fatal and serves prices uncached from Postgres — so on a deliberately Redis-less deployment, failing readiness would pull a healthy instance out of rotation. READINESS_REDIS_CHECK=false opts out; the response reports skipped rather than pretending it passed.

Probes are bounded, but the timeout is shorter than the house 10s. node-postgres exposes no AbortSignal on query(), so the deadline is applied by racing. A 10s probe on a 5s poll interval would answer from the past, so the default is 2s. A probe that exceeds it reports down without cancelling the query underneath — READINESS_CACHE_MS is what bounds how many such queries can be outstanding, which is why answers are cached and concurrent polls share one in-flight run.

Errors are not echoed in the response. A connection error carries the host and port, and these are public endpoints. The response carries a fixed reason phrase and the underlying error goes to the log. There is a test asserting no host, port or errno reaches the body.

Also updated

The deployed smoke check previously skipped /ready when absent, with a message pointing at this issue. A smoke check that silently tolerates a missing probe is how a deployment ends up unmonitored while still green, so it now fails on a 404 and also asserts /live. Both new probe paths are documented in openapi.yaml/openapi.json (the OpenAPI coverage test enforces that), and written up in docs/health.md.

npm test — 829 passed, 1 skipped (806 before, +23 new). npx tsc --noEmit, npm run build and npm run test:smoke all clean.

`/status` answers neither question a platform asks, and the two questions are
not the same one. It reads `indexer_state`, so it fails when Postgres is
unreachable — which makes it useless as a liveness signal — and it says
nothing about whether the process has finished booting.

`app.listen()` runs part-way through `main()`, before the ingesters, the
retention worker and the websocket fan-out start. Between listening and those
loops running, the process accepts requests it cannot yet answer.

GET /live touches no dependency and stays healthy while Postgres is down or
the app is still booting. That is the point: a liveness probe that fails
during a database outage has the orchestrator restarting every instance at
once, turning a recoverable outage into a cold-start storm.

GET /ready reports 503 with ok:false until main() finishes booting, and while
Postgres is unreachable or unmigrated, or Redis is unreachable when Redis is
part of readiness (READINESS_REDIS_CHECK, default true). The Postgres check is
a single to_regclass query answering both reachability and whether migrations
were applied — reachability alone would report a process with no tables as
ready, and every route reading one would fail.

Probes are bounded by READINESS_TIMEOUT_MS and answers cached for
READINESS_CACHE_MS, so polling every few seconds costs at most one round trip
per dependency per interval. node-postgres exposes no AbortSignal on query(),
so a probe past its deadline reports down without cancelling the query
underneath; the cache is what bounds how many can be outstanding.

Both probes are public, allow-listed from rate limiting next to /status, sent
Cache-Control: no-store, and excluded from http_requests_total — a
fixed-interval poller would otherwise put a floor under the request rate that
no amount of real traffic dropping could go below. /status is unchanged.

The deployed smoke check now requires both probes instead of skipping /ready
when absent, and fails on a failing /live.
@drips-wave

drips-wave Bot commented Oct 19, 2026

Copy link
Copy Markdown

@CollinsKRO Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@fastify/rate-limit consults allowList before it reads max, so the per-route
ceiling is inert while the allow-list stands.
@Miracle656

Copy link
Copy Markdown
Collaborator

Merged. You got the distinction that actually matters, and wrote down why.

/live not touching Postgres or Redis is the whole thing. A liveness probe that checks dependencies fails on every instance at once during a database blip, and the orchestrator responds by restarting the entire fleet — so a brief dependency outage becomes a cold-start storm, which is strictly worse than the outage. Plenty of real services ship that bug. Having the reason in the module comment means the next person won't "improve" /live by adding a database check to it.

The boot-state half is the subtler catch and you got that too: app.listen() happens part-way through main(), before the ingesters, the retention worker and the websocket fan-out start. In that window the process answers connections but cannot serve a price, so "listening" is not "ready" — and markBootComplete() at the end of the sequence is the right place to flip it.

Also right to use tighter deadlines than the house AbortSignal.timeout pattern, with the reason stated: pg has no per-query abort, and a probe slower than the platform's own poll interval reports staleness as health.

Merge notes: README conflicted twice, both additive. /snapshot and the websocket-backpressure doc link both landed on main while this was open, so the route table keeps all three rows and the operational-docs line keeps all four links. Nothing of yours changed.

Verified on the merge result rather than your branch: tsc --noEmit silent, 879 tests across 82 suites, nothing red.

One thing to know: #553 (issue #352, database schema verification at startup) also adds a /ready route, in src/api/rest.ts. Yours holds #334 and landed first, so that overlap resolves in your favour — I'll be asking that PR to drop its own /ready and surface its schema check through your readiness path instead, which is where a "the schema doesn't match the code" signal belongs anyway.

Closes #334.

Miracle656 added a commit that referenced this pull request Oct 19, 2026
A GraphQL query could nest as deep as it liked, which is the standard
denial-of-service shape for a public GraphQL endpoint: a short document
expands into an enormous resolver tree.

The implementation is careful in the ways that matter:

- **Fragments are expanded.** Mercurius's built-in `queryDepth` does not, so a
  limit set with it is bypassed by moving the deep part into a named fragment.
  Here a fragment node adds no depth itself while its selected fields do, which
  is the correct accounting.
- **Fragment cycles terminate.** A fragment already being expanded is not
  expanded again, so GraphQL's own cycle validation reports the cycle instead
  of this traversal recursing forever.
- **Introspection is exempt only at the operation root**, so GraphiQL loads,
  but an operation that mixes `__schema` with application fields still has its
  application branch measured. That is exactly the hole a root-level exemption
  usually leaves open.
- **A bad `GRAPHQL_MAX_QUERY_DEPTH` throws at startup** rather than falling
  back to unlimited, with the reasoning in a comment: a deployment typo must
  not silently become a security regression. Right instinct, and the inverse of
  what most config parsing does.
- Rejection happens in validation, before any resolver runs, and the error
  names both the actual depth and the configured maximum.

`.env.example` conflict resolved additively — the readiness-probe block from
#544 landed while this was open, so both sections are kept.

Verified on the merge result: `tsc --noEmit` silent, 896 tests pass across 83
suites, nothing red.

Closes #360

Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
Miracle656 added a commit that referenced this pull request Oct 19, 2026
…key expiry

Three related things, all wanted:

- **Auth failures are now counted** — `admin_auth_failures_total` and
  `api_key_auth_failures_total{reason}`, plus
  `rate_limit_rejections_total{scope}` on the per-key/IP limiter. A rejected
  request was previously invisible, so neither a misconfigured client nor
  someone probing `ADMIN_TOKEN` showed up anywhere.
- **A per-instance request ceiling** that sheds with 503 and `Retry-After`
  once this process is over `GLOBAL_RATE_LIMIT_MAX` in a rolling minute, with
  expensive route templates shed at 80% of it and the probe paths exempt. This
  is self-defence rather than a quota, so keeping it per-process and
  in-memory is the right call — it protects *this* instance's resources.
- **`api_keys.expires_at`**, with admin key creation accepting an optional
  future ISO date-time and `lookupApiKey` refusing an expired key.

`parseGlobalRateLimit` throws on an invalid value instead of falling back to a
default, which is the same instinct as #539's depth limit and correct for the
same reason: a config typo must not quietly mean "no ceiling".

Resolutions and fixes while merging:

- `src/api/usage.ts` re-added a local `isAdminAuthorized` — #517 removed
  exactly that copy earlier today. Kept the shared one from `./auth` and the
  new metric import.
- `src/app.ts`: kept `allowList: (req) => PROBE_PATHS.has(req.url)` from main
  rather than this PR's `req.url === '/status'`, since `/live` and `/ready`
  (from #544) are platform-polled and must not be limited either. Took the
  PR's metric-emitting `errorResponseBuilder` and the global-ceiling
  registration, and kept main's note on the probe-route ceiling.
- `.env.example` was missing `GLOBAL_RATE_LIMIT_MAX`, which the `envDrift`
  guard caught. Documented, including that the fleet ceiling is this value
  times the instance count by design.
- `npx prisma generate` was needed after the schema change to clear phantom
  `expiresAt` errors.

⚠️ Deploy order: `api_keys.expires_at` must exist before this runs, because
`lookupApiKey` selects it — without the column, every authenticated request
fails, not just key creation. Lens has no migrations directory (schema-first
via `db push`), so:

    ALTER TABLE "api_keys" ADD COLUMN IF NOT EXISTS "expires_at" TIMESTAMP(3);

Verified on the merge result: `tsc --noEmit` silent, 908 tests pass across 84
suites, nothing red.

Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
@Miracle656 Miracle656 mentioned this pull request Oct 19, 2026
5 of 11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

There is no readiness check, only a status page

2 participants