feat(ingest): add schema verification, cycle deadlines, lag telemetry, and /status health breakdown

[sparkysolutions10-pixel]

Summary of Changes

This PR introduces end-to-end ingestion resiliency, startup database verification, cycle deadline controls, granular lag metrics, and /status venue health visibility.

• Startup Database Schema Verification & /ready Probe (Closes Startup does not verify the database schema matches the code #352):

  • Added verifyDatabaseSchema and assertSchemaValid in src/db/schemaVerify.ts querying Postgres information_schema on boot to verify tables (price_points, pool_snapshots, price_aggregates, price_snapshots, indexer_state) and required columns.
  • Implemented DatabaseSchemaMismatchError to fail fast on boot or expose schema readiness via GET /ready (200 when schema is valid, 503 Service Unavailable with missing tables/columns on mismatch).
  • Documented migration architecture rationale in docs/database-schema.md detailing the path from prisma db push to versioned prisma migrate.

• Ingester Single-Cycle Deadline Timeout (Closes Ingester loops have no upper bound on a single cycle #351):

  • Added runCycleWithDeadline in src/ingest/venues/runner.ts using Promise.race against configurable deadlines ([VENUE]_CYCLE_TIMEOUT_MS or INGESTER_CYCLE_TIMEOUT_MS, default 30s).
  • Hanging or stalled cycles are cleanly abandoned without partial writes, incrementing the new Prometheus metric ingestion_cycles_abandoned_total.
  • Ensures cycles execute serially with zero overlapping state drift.

• Ingestion Lag & Cycle Telemetry (Closes Nothing measures how far behind ingestion is #340):

  • Exported Prometheus metrics in src/metrics.ts:
    • ingestion_lag_seconds: Gauge measuring seconds elapsed since last write per venue and network.
    • ingestion_cycle_duration_seconds: Histogram with tuned buckets (50ms to 30s) measuring poll duration.
    • ingestion_errors_total: Counter categorizing failures into bounded label categories (network, upstream_4xx, upstream_5xx, db, parse, timeout, unknown).

• /status Venue Breakdown & Health Reporting (Closes /status does not say whether ingestion is actually working #333):

  • Enhanced GET /status to report venues breakdown with per-venue health (healthy, stale, offline), lastWrite timestamp, ageSeconds, and overall ingestionHealth (healthy, degraded, unhealthy).
  • Preserved all existing response fields (ok, network, watchedPairs, lastIndexedLedger, lastProcessedAt, ingestLagSeconds).
  • Updated statusResponseSchema and readyResponseSchema with comprehensive validation.

Testing & Verification

• Unit test suite src/__tests__/ingestionResiliencyHealth.test.ts covering schema verification, /ready endpoint, timeout abandonment, error categorization, and /status breakdown.
• Verified OpenAPI contract regeneration and route coverage test passing.

[Miracle656]

Changes requested: one issue per PR, and only issues assigned to you

This is a process requirement, applied to all 23 of your open PRs in this repo. It is not a judgement about the code, which I have not been able to review — and that is the first half of the problem.

The bundling

Across your open PRs, 150 distinct issues are claimed. This one claims several. #483 claims 22, #481 claims 20, #477 claims 20, #474 claims 20.

A PR that closes twenty issues cannot be reviewed. Each of those issues has its own acceptance criteria — typically five — so twenty issues is a hundred separate things to verify in one diff, and if any one of them is wrong the whole PR is blocked while it is fixed. Every batch I published says the issues are deliberately independent so they can be worked in parallel, by different people, without colliding. Bundling them removes exactly that property.

The assignments

135 of the 150 issues claimed are assigned to other contributors — 67 different people.

This PR is an example: it names issues currently assigned to other contributors who are working on them now. Several have open PRs of their own. If this merged as it stands, those people would have done the work and received nothing for it, and I would have no way to tell the two efforts apart.

I am not assuming anything about why. Assignments change, Drips and GitHub can disagree, and a wide-ranging branch picks up issue numbers easily. But the effect on the people holding those assignments is the same regardless of intent, which is why I have to act on it.

What I need

• One issue per PR. The issue number in the title, its acceptance criteria met, and nothing else in the diff.
• Only issues assigned to you, or unassigned ones you have claimed through Drips. If an issue is assigned to someone else, it is theirs.
• Re-open the work that meets both conditions as individual PRs. I will review them in the order they arrive and at the same pace as everyone else's.

Nothing is being closed. These stay open and are yours to re-cut, withdraw or split as you choose.

Two things worth knowing before you re-cut

Some of this has already landed. #488 (SSRF guard for bazaar_search) and #487 (the paid_call host allow-list) cover the same ground as your #449; #486 covers the MCP README. Check current main before rebuilding something that is already there.

main has moved a long way today — #306, #309, #252, #256, #355, #381, #390, #402, #476, #488 and more. Cut from current main, not from where these branches started.

If you believe an issue named here is genuinely yours, say which and I will check the assignment myself — I would rather be corrected than have someone lose work.