feat(solvers): liveness heartbeats with automatic offline detection

[benedictworks-home]

Summary

Implements #445: solver liveness heartbeats with automatic offline detection. Dead solvers stopped being quoted — markLive/markOffline are now driven by heartbeats instead of explicit calls.

What's new

Heartbeat protocol (negotiated cadence, default 10 s)

• The server advertises heartbeatIntervalMs in the WS connected frame, in auth_ok, and in every ack (that's the negotiation: server declares, client adapts).
• Primary channel: { "type": "heartbeat" } on the authenticated socket → { "type": "heartbeat_ack", "accepted": true, "heartbeatIntervalMs": 10000 } (unauthenticated connections get accepted: false so a bot notices it never completed auth).
• REST alternative: POST /api/v1/solvers/:address/heartbeat with { timestamp, signature } over the canonical heartbeat:<address>:<timestamp>; the timestamp must be fresh within max(60 s, offline window) so a captured beat cannot keep a dead solver live.

Offline after 3 misses (SOLVER_HEARTBEAT_MISSES, window = interval × misses = 30 s)

• Two-phase sweep every interval: read all shared verdicts first (per-solver Redis key with the window as TTL), apply transitions second.
• Partition guard: if any shared read reports the store unreachable, the cycle aborts with no transitions; a fresh local beat also wins at transition re-check. One partitioned replica can never mass-flip its solvers.
• Boot grace (one window after process start) + per-solver proof-of-life (lastActiveAt within one window from register/reactivate/fill) give bots time to connect; a solver that is demonstrably filling is never withdrawn from quotes.
• Store choice: SOLVER_HEARTBEAT_REDIS_URL (defaults to REDIS_URL) → shared Redis; empty → process-local memory (single replica only). Periodic scan instead of keyspace notifications — no notify-keyspace-events config burden, fire-and-forget loss on subscriber reconnect, and it can distinguish "expired" from "unreachable" (the guard's precondition). Reasoning recorded in docs/adr/0007-solver-liveness-heartbeats.md.

Exclusion + events

• Offline transitions flip isActive=false, so existing gates exclude the solver for free: POST /api/v1/intents/quote (RFQ), GET /solvers/:address/eligible-intents, and WS re-authentication all check isActive.
• Capability filtering on live connections: buildMatchPredicate(solver, isLive?) gains an optional gate consulted on every matches() — installed predicates go quiet without a rebuild (gateway + feed both pass the closure).
• solver_status_changed is broadcast on the WS/SSE feed (solver, status, lastActiveAt, reason, at); lastActiveAt is stamped on every status transition (reactivate now stamps it too, like every other transition).
• Auto-offline is reversible, deliberate deactivation is not: the auto-offline flag lets the next heartbeat or a re-authentication heal the record; deactivate/deregister clear the flag so heartbeats can never undo an operator decision.

Metrics

• vortex_solver_live_by_chain{chain} — live (= active and beating) solver count per supported chain, * expanded to all chains, refreshed each sweep and on heal.
• vortex_solver_status_changes_total{status} — transition counter.

New environment variables

Var	Default	Notes
SOLVER_HEARTBEAT_INTERVAL_MS	10000	Client cadence + sweep period (joi: 1000–600000)
SOLVER_HEARTBEAT_MISSES	3	Misses tolerated before auto-offline (joi: 1–60)
SOLVER_HEARTBEAT_REDIS_URL	REDIS_URL if set, else ""	Shared liveness store; "" = process-local memory

Added to src/config/env.validation.ts and all four .env*.example files (keeps check:env-drift green).

Files

• New: src/solvers/solver-liveness.service.ts, src/solvers/liveness.store.ts (memory + Redis stores), src/solvers/dto/heartbeat.dto.ts, src/solvers/solver-liveness.service.spec.ts, docs/adr/0007-solver-liveness-heartbeats.md
• Modified: gateway (heartbeat message, auth-time touch/heal, predicate gates), feed (predicate gate), solver-intent-matcher (isLive param), solvers controller/module/service, stellar-signature (buildHeartbeatMessage), metrics, config trio + env examples, docs (solver-onboarding.md §3, runbook Scenario G + key config, README endpoint list)

Verification

• Tests written (run in CI): solver-liveness.service.spec.ts — fake-timer miss detection (live at 2 misses, offline at 3), boot grace, partition guard (whole-cycle abort, both solvers survive), store-flush protection (local beats win), auto-offline heal vs deliberate deactivation, concurrent-heartbeat single-emit, unknown-solver ack, per-chain metric counts (incl. * expansion), capability-predicate gate, and a two-replica integration scenario (two service instances over one shared store + one repository: beats only on A keep both online; after silence B transitions exactly once and A's sweep does not double-emit; reconnect on A heals globally).
• Local checks run on merge commit 64e5a03 (Node 24, Windows): tsc --noEmit clean; eslint src test scripts packages … 0 errors (144 warnings, same baseline as main); check:env-drift pass; check:quarantine pass; check:migrations --base c48de27 pass (every changed migration has down.sql, no unsafe DDL); jest --config scripts/jest.config.js pass (43 tests); full unit suite in 4 shards (--shard=n/4 --maxWorkers=3): 1727 passed, 97 skipped, 0 failed.
• Not run locally: docker-based jobs, psql-backed integration tests, gitleaks, semgrep, and coverage upload (no docker/psql in this environment) — CI is the authority for those.

Merge with upstream/main (64e5a03)

The branch merges upstream main at c48de27. That tree itself ships ~10 parse-broken files (leaderboard-query.ts, in-memory-intents.repository.spec.ts, intents.types.ts, list-intents.dto.ts, configuration.ts, env.validation.ts, stats.service.ts, intents.controller.ts, intents.gateway.ts, intents.module.ts, stellar-signature.ts) plus 14 zero-byte files (incl. .github/workflows/ci.yml, src/soroban/soroban.service.ts, src/treasury/treasury.service.ts), so conflicts were resolved conservatively:

• Files this PR does not own are taken from 2a922ee — the simulation-harness tree from feat(tools): solver simulation harness and strategy backtesting tool #575, which is c48de27 plus those torn/empty files repaired (independently verified green: tsc, eslint, 1746 passing unit tests).
• Files [High] Solver Liveness Heartbeats with Automatic Offline Detection #445 owns (liveness service/store/DTO, matcher, metrics, solvers controller/module/service, gateway, feed, stellar-signature, config trio, .env* examples, README/docs) keep the branch content merged with main's additions.
• intents.gateway.ts keeps the harness structure and re-applies only [High] Solver Liveness Heartbeats with Automatic Offline Detection #445's own delta: solverForAuth at re-auth (touch heals auto-offline), the isLive-gated match predicate, heartbeatIntervalMs negotiation in the connected/auth_ok frames, and the { "type": "heartbeat" } ack path.
• prisma/schema.prisma = harness schema + main's @@index([solver]).
• package.json / package-lock.json are byte-identical to the feat(tools): solver simulation harness and strategy backtesting tool #575 tree — this PR adds no dependency of its own and the lockfile is not hand-edited (the @aws-sdk/client-s3 entry present in it originates from feat(tools): solver simulation harness and strategy backtesting tool #575).

Fixes made during verification (needed for a type-clean, green tree):

• solver-liveness.service.spec.ts: three feed.broadcast = makeFeed() assignments stored a whole fake feed where a broadcast function is expected — a pre-existing type error in this branch (the file never compiled under ts-jest diagnostics); replaced with fresh jest.fn() spies.
• solver-registry.service.spec.ts: the AppConfig literal gained the three SOLVER_HEARTBEAT_* keys that configuration.ts now requires.
• solver-liveness.service.ts: IntentFeedService is now @Optional() (the current IntentsModule registers no feed provider — the same treatment the gateway's feed param already applies), and boot grace keeps the optimistic live view so first-sweep metrics and predicate gating match the spec.
• intents.gateway.ts: the WS message listener returns the dispatch promise (failures catch-logged) so the async auth path settles deterministically for the existing gateway spec.
• .env.example: backfilled the solver-reputation ([High] Solver Reputation Score v2 with Time Decay and Transparency #444) variables (from fd6bff1, via the feat(tools): solver simulation harness and strategy backtesting tool #575 tree) so check:env-drift stays green; the other three .env*.example files carry both that block and this PR's heartbeat block.

(The two fix: commits ac129c1/8fb9a87 that the branch started with remain in history; after the merge, shared files follow the repaired 2a922ee content.)

Assumptions & edge cases

• Issue text says "RFQ and capability filtering": RFQ exclusion rides on isActive (which this PR's sweeper now drives); capability filtering gets an explicit liveness gate for already-connected solvers.
• "Network partitions of a single replica must not flap all solvers offline" is enforced three ways: store-unreachable → abort cycle; fresh local beat → never offline; boot grace → no cold-start mass flap. Documented in the ADR.
• Bots must be upgraded to beat within one window of deployment (behaviour change, documented in docs/solver-onboarding.md §3): a non-heartbeating solver goes offline interval × misses + ≤interval after the grace, which is exactly the issue's intent.
• Out of scope per the issue: penalising downtime (no slashing changes here).

Checklist

• Tests written for new behaviour (fake timers + two-replica integration)
• TSDoc on public APIs
• README / runbook (Scenario G) / ADR / docs/solver-onboarding.md updated
• New env vars in env.validation.ts + every .env*.example
• Local parse / duplicate-binding / env-drift checks clean
• npm run lint / typecheck / test run locally — tsc clean, eslint 0 errors, 1727 unit tests green on the merge commit
• docker / psql / gitleaks / semgrep / coverage jobs — not runnable in this environment; CI is the authority

Closes #445