Harden the socket.io server against connection floods and oversized messages #97

Overview

The HTTP side of the backend has a reasonable baseline: helmet, an explicit CORS allowlist, a 1 MB JSON body limit and express-rate-limit limiters (globalLimiter 200 requests / 15 min per IP, authLimiter, generateLimiter). The socket.io server created in src/server.ts has none of this. It runs with library defaults: both polling and websocket transports, a 1 MB maxHttpBufferSize per inbound message, no limit on connections per IP or in total, no limit on inbound events per socket, and it logs every connect and disconnect at info level.

Because socket.io traffic is upgraded or long-polled outside the Express middleware chain, none of the HTTP rate limiters apply to it. A single client (or a small botnet) can open thousands of websocket connections and exhaust file descriptors and memory, keep thousands of polling sessions alive, send 1 MB frames, or simply connect and disconnect in a loop to flood the logs and the log pipeline. The CORS origin check on the socket server is also configured separately from the HTTP allowlist and only restricts browsers, not raw clients.

This issue makes the socket layer defensible: explicit transport and buffer limits, per-IP and global connection caps enforced across instances through Redis, per-socket inbound event rate limiting, origin validation identical to the HTTP CORS allowlist, sampled connection logging, and connection metrics. The hard parts are deriving the correct client IP behind proxies, avoiding false positives for users behind carrier-grade NAT, and proving the limits with adversarial tests.

Background & current behavior

• src/server.ts:

  const io = new Server(server, { cors: { origin: config.CORS_ORIGINS, methods: ['GET', 'POST'] } });
  io.on('connection', (socket) => {
    logger.info({ socketId: socket.id }, 'New client connected to socket.io');
    socket.on('disconnect', () => { logger.info({ socketId: socket.id }, 'Client disconnected from socket.io'); });
  });

  No allowRequest, no maxHttpBufferSize, no transports, no connectTimeout tuning, no middleware (io.use).
• src/app.ts CORS: requests with no Origin are allowed (server-to-server), otherwise config.CORS_ORIGINS.includes(origin). socket.io's cors option only sets response headers for polling/preflight; it does not reject websocket upgrades from other origins.
• src/middlewares/rate-limit.middleware.ts: express-rate-limit with the in-memory store, skipped when NODE_ENV === 'test'; keyed by req.ip. Express's trust proxy is not set anywhere, so behind a proxy every client shares the proxy's IP (the proxy-trust issue covers HTTP; sockets need the same derivation).
• Clients never send application events today (the server only emits), so inbound events can be tightly limited without breaking the frontend; wave issue Feature/integration and fixes #4 will add a subscribe message.
• There is no metrics endpoint yet (a Prometheus issue is planned separately).

Goals

• Explicit engine options: maxHttpBufferSize, pingInterval, pingTimeout, connectTimeout, transport policy, and perMessageDeflate disabled.
• Origin enforcement for both polling and websocket upgrades using the same allowlist function as HTTP CORS.
• Per-IP and global concurrent connection caps, shared across instances via Redis, with local fallback when Redis is unavailable.
• Connection-rate limiting (new connections per IP per minute).
• Per-socket inbound event rate limiting and an allowlist of accepted inbound event names.
• Correct client IP behind configured trusted proxies.
• Sampled logging and connection counters exposed for metrics.
• Adversarial tests.

Non-goals

• Socket authentication (separate issue) and per-user quotas.
• HTTP trust proxy configuration and security headers (separate issue), though the IP derivation helper should be reusable there.
• DDoS protection at the network edge; document that a CDN/WAF is still expected in production.
• Outbound throttling and slow consumers (separate coalescing issue).

Proposed implementation

1. Config in src/config/index.ts and .env.example:
   SOCKET_MAX_CONNECTIONS (global per instance, default 5000), SOCKET_MAX_PER_IP (default 20), SOCKET_CONNECT_RATE_PER_MIN (default 60), SOCKET_MAX_BUFFER_BYTES (default 16384), SOCKET_TRANSPORTS (websocket,polling default; websocket to disable polling), SOCKET_INBOUND_EVENTS_PER_10S (default 20), SOCKET_LOG_SAMPLE_RATE (default 0.01), TRUSTED_PROXIES (comma list of CIDRs, default empty), SOCKET_IP_EXEMPT_CIDRS (default empty, for known NAT ranges or monitoring).
2. New src/lib/socket-limits.ts:

   export function clientIp(req: IncomingMessage, trusted: string[]): string; // walks X-Forwarded-For right-to-left skipping trusted proxies
   export function isAllowedOrigin(origin: string | undefined): boolean;      // shared with app.ts CORS
   export interface ConnectionLimiter { acquire(ip: string, socketId: string): Promise<{ ok: true } | { ok: false; reason: 'GLOBAL' | 'PER_IP' | 'RATE' }>; release(ip: string, socketId: string): Promise<void> }
   export function redisConnectionLimiter(redis: Redis, opts): ConnectionLimiter;
   export function memoryConnectionLimiter(opts): ConnectionLimiter;
   export function inboundEventLimiter(socket: Socket, opts): void;  // token bucket per socket
   export function socketMetrics(): { connected: number; rejected: Record<string, number>; disconnectedForAbuse: number };

   Move the origin callback from src/app.ts into isAllowedOrigin() and use it in both places.
3. allowRequest on the engine (runs for polling handshakes and websocket upgrades before a socket exists):

   const io = new Server(server, {
     transports: config.SOCKET_TRANSPORTS,
     maxHttpBufferSize: config.SOCKET_MAX_BUFFER_BYTES,
     pingInterval: 25_000, pingTimeout: 20_000, connectTimeout: 10_000,
     perMessageDeflate: false,
     cors: { origin: (o, cb) => cb(null, isAllowedOrigin(o)), methods: ['GET', 'POST'], credentials: true },
     allowRequest: (req, cb) => {
       if (req.headers.origin && !isAllowedOrigin(req.headers.origin)) return cb('ORIGIN_NOT_ALLOWED', false);
       cb(null, true);
     },
   });

   Requests without an Origin (non-browser clients) are allowed, consistent with HTTP, but still subject to connection caps.
4. Connection caps in an io.use middleware (async, has the socket id): compute ip = clientIp(socket.request, TRUSTED_PROXIES), call limiter.acquire(ip, socket.id); on rejection next(Object.assign(new Error('CONNECTION_LIMIT'), { data: { reason } })). Release on disconnect. Redis implementation:
   • Per-IP set: SADD socklim:ip:<ip> <instanceId>:<socketId> with EXPIRE refreshed by a heartbeat; SCARD compared to the cap, done atomically in a Lua script that also checks the global counter.
   • Global per-cluster: a hash socklim:instances of instanceId → count, refreshed every 10 s with a TTL so a crashed instance's counts expire instead of leaking forever.
   • Connection rate: INCR socklim:rate:<ip>:<minute> with EXPIRE 120.
     If Redis errors, fall back to the in-memory limiter with the same caps (fail closed on global per-instance limits, fail open on cluster-wide counts) and log once per minute.
5. Inbound events: inboundEventLimiter uses socket.use(([event], next) => ...) to reject any event not in ALLOWED_INBOUND_EVENTS (initially empty; wave Feature/integration and fixes #4 adds subscribe/unsubscribe) and to enforce a token bucket; after 3 violations in a minute, socket.disconnect(true) and count disconnectedForAbuse.
6. Logging: replace the per-connection logger.info with logger.debug plus sampled info (Math.random() < SOCKET_LOG_SAMPLE_RATE), always log rejections at warn but rate-limited per reason (one line per reason per 10 s with a count). Never log full headers.
7. Metrics: socketMetrics() updated by the middleware and engine events (io.engine.on('connection_error', ...) for handshake failures). Expose it on the existing /health response under data.sockets only when NODE_ENV !== 'production', and leave the export for the Prometheus issue.
8. Graceful shutdown: in the shutdown handler in src/server.ts, call io.close() before server.close(), and release all Redis slots for this instance (delete its entry in socklim:instances).
9. Docs: add a "Realtime limits" subsection to docs/api.md (error codes clients may receive on connect: ORIGIN_NOT_ALLOWED, CONNECTION_LIMIT with reason) and a deployment note in docs/architecture.md about TRUSTED_PROXIES and NAT exemptions.

Edge cases & pitfalls

• Spoofed X-Forwarded-For: only honour it when the immediate peer (req.socket.remoteAddress) is in TRUSTED_PROXIES; otherwise use the peer address. IPv4-mapped IPv6 (::ffff:1.2.3.4) must normalise to IPv4.
• Users behind CGNAT or a corporate proxy share an IP; SOCKET_MAX_PER_IP = 20 must be generous enough for many tabs, and SOCKET_IP_EXEMPT_CIDRS gives operators an escape hatch. Consider keying authenticated sockets by user once socket auth exists (note as follow-up).
• Polling sessions create several HTTP requests per socket; caps must count sockets, not requests, which is why they live in io.use rather than allowRequest.
• A crashed instance leaves set members in Redis; the per-member heartbeat TTL must reclaim them within a minute.
• Test environments: express-rate-limit is skipped under NODE_ENV=test, but these limits must be testable; inject options explicitly in tests instead of reading config.

Testing plan

New tests/socket-limits.test.ts (in-process http.Server + Server on port 0, socket.io-client added to devDependencies, memoryConnectionLimiter by default and the Redis limiter when REDIS_URL is reachable):

• clientIp ignores X-Forwarded-For from untrusted peers and honours trusted proxies
• websocket upgrade from a disallowed origin is rejected
• connections without Origin are accepted and counted
• 21st connection from one IP is rejected with CONNECTION_LIMIT PER_IP
• global cap rejects new connections and recovers after disconnects
• connection rate limit rejects rapid reconnect loops
• messages larger than maxHttpBufferSize close the connection
• unknown inbound events are rejected and repeated violations disconnect the socket
• two server instances sharing Redis enforce one per-IP cap (Redis-gated)
• crashed instance slots expire after the heartbeat TTL (Redis-gated, fake time via short TTL)

Commands: docker compose up -d redis, npm test, npm run build.

Acceptance criteria

• The socket.io server is created with explicit transport, buffer, ping, timeout and compression options from validated config.
• Origin validation for sockets and HTTP uses one shared isAllowedOrigin() function, including websocket upgrades.
• Per-IP, global and connection-rate limits are enforced in a socket middleware and released on disconnect.
• Limits are shared across instances through Redis with crash-safe expiry, and fall back to in-memory limits when Redis is down.
• Client IP derivation honours only TRUSTED_PROXIES and normalises IPv6-mapped addresses.
• Inbound events are allowlisted and rate-limited per socket.
• Connection logging is sampled and rejection logs are rate-limited; counters are exposed for metrics.
• Adversarial tests pass locally and in CI (Redis-backed tests run in CI's Redis service).
• .env.example, docs/api.md and docs/architecture.md are updated.
• npm run build, npm test and CI pass.

Resources

• src/server.ts, src/app.ts, src/middlewares/rate-limit.middleware.ts, src/lib/redis.ts, src/lib/logger.ts, src/config/index.ts, .env.example
• docs/wave-issues/04-socket-market-rooms.md, docs/wave-issues/07-rate-limit-tuning.md
• socket.io server options (allowRequest, maxHttpBufferSize, transports): https://socket.io/docs/v4/server-options/
• socket.io middlewares: https://socket.io/docs/v4/middlewares/
• Express behind proxies (trust proxy): https://expressjs.com/en/guide/behind-proxies.html
• OWASP WebSocket Security Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/WebSocket_Security_Cheat_Sheet.html

Generated by Claude Code