feat: Process Resiliency, Infrastructure Health Probes, and Reliability Telemetry Cluster

[sparkysolutions10-pixel]

Summary of Changes

This PR implements comprehensive process resiliency, infrastructure health probing, degraded mode recovery, and reliability telemetry across 10 linked issues:

• Closes Horizon rate limiting is not handled #348: Detects Horizon non-2xx responses before body parsing, respects Retry-After headers and exponential backoff on HTTP 429, tracks horizon_rate_limit_remaining and horizon_rate_limited_total metrics, and prevents rate-limited responses from masquerading as empty venues.
• Closes Redis going away is noticed once and never again #345: Exposes active snapshot retention mode (queue vs in_process) via /status and Prometheus gauge retention_path_mode, establishes periodic Redis health monitoring to automatically recover from degraded mode back to queue mode, and guards concurrent in-process prunes via PostgreSQL advisory locking.
• Closes An unhandled rejection takes the process down with no record #344: Installs process error guards early during startup (unhandledRejection and uncaughtException), logs the full error message and stack trace, increments unhandled_rejections_total / uncaught_exceptions_total counters, and immediately exits non-zero without entering graceful drain so container platforms restart cleanly.
• Closes A webhook that fails three times disappears #342: Introduces a durable dead-letter store (recordFailedDelivery) capturing permanently failed webhook deliveries with bounded categorization (upstream_5xx, timeout, network, unknown), increments webhook_delivery_failures_total, and ensures subscriber secrets are never persisted.
• Closes Webhook retries have no jitter, so failures synchronise #341: Adds full jitter to exponential retry delays (retryDelayMs) with an injectable randomSource to eliminate retry synchronization thundering herds.
• Closes The database connection pool is unconfigured and unmeasured #339: Explicitly bounds and measures the shared PostgreSQL connection pool (DATABASE_POOL_SIZE default 20, DATABASE_POOL_TIMEOUT_MS default 10s), exports utilization gauges (pg_pool_total_connections, pg_pool_idle_connections, pg_pool_waiting_count), and throws a distinct DatabasePoolExhaustedError on timeout.
• Closes There is no readiness check, only a status page #334: Implements public, unauthenticated GET /live (lightweight liveness probe checking event loop) and GET /ready (readiness probe validating boot completion and database/cache connectivity), exempts both from rate limiting, and excludes them from httpMetrics route cardinality.
• Closes A crash-looping ingester is invisible #331: Enhances ingester restart isolation with capped exponential backoff (1s up to 60s ceiling), automatic reset after sustained 60s healthy runs, ingester_restarts_total telemetry, and crash history retrievable in /status.ingesterCrashes.
• Closes A failed venue fetch is indistinguishable from an empty venue #329: Replaces silent empty collection returns with distinct VenueFetchError across AMM, Aquarius, Soroswap, and Reflector, tracks failed venues in getFailedVenues(), and prevents failed venues from falsely diluting or reweighting price aggregates.
• Closes There is no runbook for the service #350: Adds a comprehensive production service runbook in docs/RUNBOOK.md covering processes, external dependencies, degraded modes, /status & /metrics diagnostics, and deploy/rollback procedures, and documents the shared connection pool in README.md.

Verification

• Unit & integration tests pass: npx vitest run src/__tests__/reliabilityResilienceCluster.test.ts tests/openapi.test.ts (24/24 passing).
• TypeScript compile succeeds with zero errors: npx tsc --noEmit.
• OpenAPI schema sync verified: npm run openapi:gen.
• Changeset included: .changeset/reliability-resilience-cluster.md.

[Miracle656]

Changes requested: one issue per PR, and only issues assigned to you

This is a process requirement, applied to all 23 of your open PRs in this repo. It is not a judgement about the code, which I have not been able to review — and that is the first half of the problem.

The bundling

Across your open PRs, 150 distinct issues are claimed. This one claims several. #483 claims 22, #481 claims 20, #477 claims 20, #474 claims 20.

A PR that closes twenty issues cannot be reviewed. Each of those issues has its own acceptance criteria — typically five — so twenty issues is a hundred separate things to verify in one diff, and if any one of them is wrong the whole PR is blocked while it is fixed. Every batch I published says the issues are deliberately independent so they can be worked in parallel, by different people, without colliding. Bundling them removes exactly that property.

The assignments

135 of the 150 issues claimed are assigned to other contributors — 67 different people.

This PR is an example: it names issues currently assigned to other contributors who are working on them now. Several have open PRs of their own. If this merged as it stands, those people would have done the work and received nothing for it, and I would have no way to tell the two efforts apart.

I am not assuming anything about why. Assignments change, Drips and GitHub can disagree, and a wide-ranging branch picks up issue numbers easily. But the effect on the people holding those assignments is the same regardless of intent, which is why I have to act on it.

What I need

• One issue per PR. The issue number in the title, its acceptance criteria met, and nothing else in the diff.
• Only issues assigned to you, or unassigned ones you have claimed through Drips. If an issue is assigned to someone else, it is theirs.
• Re-open the work that meets both conditions as individual PRs. I will review them in the order they arrive and at the same pace as everyone else's.

Nothing is being closed. These stay open and are yours to re-cut, withdraw or split as you choose.

Two things worth knowing before you re-cut

Some of this has already landed. #488 (SSRF guard for bazaar_search) and #487 (the paid_call host allow-list) cover the same ground as your #449; #486 covers the MCP README. Check current main before rebuilding something that is already there.

main has moved a long way today — #306, #309, #252, #256, #355, #381, #390, #402, #476, #488 and more. Cut from current main, not from where these branches started.

If you believe an issue named here is genuinely yours, say which and I will check the assignment myself — I would rather be corrected than have someone lose work.