feat: socket hardening, throttling, chain read cache, and HTTP cache policies

[rubbathabrahband]

Summary

This PR resolves issues #96, #97, #98, and #99 by implementing socket update coalescing and slow consumer protection, hardening the socket server against floods and abuse, adding a Redis-backed chain read-through cache with single-flight and ledger invalidation, and establishing explicit HTTP cache policies with strong ETags and conditional request (304) handling.

Key Changes

1. Coalesce High-Frequency Socket Updates & Slow Consumer Protection (#96)

• Market Price Coalescing: Introduced MARKET_PRICE events coalesced latest-wins per market and emitted at most once per SOCKET_PRICE_TICK_MS (default 1000 ms).
• Volatile Delivery: Configured TRADE_EXECUTED as volatile delivery (SOCKET_TRADE_VOLATILE=true) emitted only to sockets in the realtime:full room.
• Liveness & Catchup Classification: Added ledger lag classification in src/services/indexer.service.ts (SOCKET_LIVE_LEDGER_LAG, default 12 ledgers). Historical trades during catch-up are suppressed, emitting a single CATCHUP_COMPLETE summary instead.
• Slow Consumer Guard: Added attachSlowConsumerGuard to monitor socket buffered amounts. Sockets above SOCKET_SLOW_WARN_BYTES (256 KB) degrade to receiving only price ticks; sockets above SOCKET_SLOW_DROP_BYTES (2 MB) are disconnected.
• Unit & Soak Tests: Created unit tests in tests/socket-throttle.test.ts and load soak script scripts/socket-load.mjs.

2. Harden Socket.io Server Against Connection Floods & Oversized Messages (#97)

• Engine Hardening: Explicitly configured maxHttpBufferSize (SOCKET_MAX_BUFFER_BYTES, default 16 KB), pingInterval, pingTimeout, connectTimeout, transports, and disabled perMessageDeflate.
• Origin Validation: Shared isAllowedOrigin() between Express CORS and socket engine allowRequest to reject unauthorized WebSocket upgrades.
• Connection Caps & Rate Limiting: Enforced distributed global, per-IP, and connection rate limits via Redis with local memory fallback (redisConnectionLimiter / memoryConnectionLimiter).
• Client IP & Proxies: Implemented clientIp() walking X-Forwarded-For right-to-left skipping proxies in TRUSTED_PROXIES and normalizing IPv6-mapped IPv4 addresses.
• Inbound Event Limiter: Blocked unauthorized inbound application events with token-bucket rate limiting and disconnection on repeated abuse. Exposed socket connection metrics on /health for non-production environments.
• Unit Tests: Created tests/socket-limits.test.ts.

3. Cache Live Chain Reads in Redis with Ledger Invalidation & Single-Flight (#98)

• Lossless Codec: Created src/lib/chain-cache.ts supporting bigint, Buffer/Uint8Array, tagged unions, and contract types.
• Tag-Generation Invalidation: Implemented tag generations (chain:gen:<tag>) and ledger watermark checks (chain:minLedger:<tag>) to prevent race conditions from stale simulations overwriting fresh updates.
• Single-Flight Coalescing: Coalesced concurrent identical requests in-process and cross-instance using Redis locks.
• Read & Write Integration: Added getOnChainMarketCached, resolverStateCached, and treasuryAvailableCapitalCached. Invalidate affected tags on live agent writes and indexer events.
• Unit Tests: Created tests/chain-cache.test.ts.

4. HTTP Cache Middleware, Strong ETags & Conditional Requests (#99)

• Per-Route Cache Policies: Implemented cachePolicy middleware supporting shared, immutable, personalised, and no-store.
• Pre-execution Validators: Built cheap DB aggregation validators for market lists, detail, traces, and portfolio positions/stats that calculate strong ETags and answer If-None-Match with 304 Not Modified before running expensive handlers.
• Canonical Body Fallback: Added canonical JSON fallback ETags in sendSuccess when cheap validators are not present.
• Security & Authorization Isolation: Disabled default weak ETags (app.set('etag', false)). Requests carrying Authorization are never served public cache headers. Compressed responses append encoding suffixes with Vary: Accept-Encoding.
• Unit Tests: Created tests/http-cache.test.ts.

Closes #96,
Closes #97,
Closes #98,
Closes #99

[drips-wave]

@rubbathabrahband Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits