Multithreaded TCP Load Balancer (C++)

A Layer-4 (TCP) load balancer built from raw POSIX sockets, with a custom thread pool for concurrent connection handling and round-robin backend selection. Built to demonstrate systems/backend fundamentals — concurrency, synchronization, and network I/O — as a complement to AI/ML coursework.

Architecture

client(s) ──TCP──> [load balancer :8080] ──TCP──> backend_1 :9001
                                          ──TCP──> backend_2 :9002
                                          ──TCP──> backend_N :900N

• Raw POSIX sockets (no Boost.Asio) — the socket layer is hand-rolled.
• TCP-level (L4) forwarding, not HTTP-level (L7). This is deliberate: it avoids writing an HTTP parser (partial reads, keep-alive, chunked encoding are a well-known time sink) and mirrors how early HAProxy/nginx-stream operate at the TCP layer.
• A fixed-size thread pool handles accepted connections concurrently; a single accept-loop thread keeps calling accept() and hands each connection off to the pool rather than blocking on it.
• Backends are selected via round-robin over a shared index.

Project structure — four milestones, kept as a progression

File	What it is
proxy_v1_singlethread.cpp	Single-threaded, single-backend proxy. Proves out the socket plumbing (partial read/write handling) before any concurrency is introduced.
thread_pool.h	Fixed-size worker pool: std::mutex + std::condition_variable + std::queue<std::function<void()>>.
proxy_v2_threadpool.cpp	v1 + thread pool. Accept loop no longer blocks on handle_client(); it submits a task and immediately loops back to accept().
proxy_v3_racy.cpp	v2 + multiple backends + round-robin selection over a deliberately unsynchronized shared counter. Kept as "before" evidence — not fixed in this file on purpose.
proxy_v4_fixed.cpp	v3 with the round-robin counter protected by a std::mutex. Kept as a separate file so both versions exist as a before/after pair.
thread_pool_smoke_test.cpp	Standalone correctness checks for the pool: concurrency actually overlaps, every task runs exactly once, shutdown drains the queue without hanging.
proxy_v5_epoll.cpp	Single-threaded, event-driven proxy using epoll (level-triggered) instead of a thread pool. Per-connection state machine, non-blocking sockets throughout. See EPOLL_README.md for a full walkthrough.
backend_server.cpp	Trivial test-fixture backend. Echoes a response tagged with its own port, so you can verify which backend served a given request.

The race condition: before and after

proxy_v3_racy.cpp selects a backend with a plain (non-atomic, non-mutex-protected) global counter:

int chosen_index = g_next_backend;
g_next_backend = (g_next_backend + 1) % g_backend_ports.size();

Multiple worker threads call this concurrently. The read and the write are two separate, non-atomic memory operations, so this is a classic read-modify-write race — undefined behavior, not just "occasionally picks the wrong backend."

Confirmed with ThreadSanitizer (g++ -fsanitize=thread), under concurrent client load:

• proxy_v3_racy.cpp: 4 confirmed data races, all localized to pick_backend_port_RACY() at proxy_v3_racy.cpp:132-133, on the global g_next_backend. TSan reports both read/write and write/write races between different worker threads on the same load run.
• proxy_v4_fixed.cpp: 0 data races under the identical load pattern and client count.

Excerpt from the TSan report on v3:

WARNING: ThreadSanitizer: data race (pid=31020)
  Read of size 4 at 0x55555555c280 by thread T2:
    #0 pick_backend_port_RACY proxy_v3_racy.cpp:132
  Previous write of size 4 at 0x55555555c280 by thread T1:
    #0 pick_backend_port_RACY proxy_v3_racy.cpp:133
  Location is global '(anonymous namespace)::g_next_backend'
SUMMARY: ThreadSanitizer: data race proxy_v3_racy.cpp:132 in pick_backend_port_RACY

The fix (proxy_v4_fixed.cpp) wraps the entire read-index/compute-next/write-back/read-port sequence in a single std::lock_guard<std::mutex>, making it indivisible from any other thread's point of view:

int pick_backend_port() {
    std::lock_guard<std::mutex> lock(g_backend_mutex);
    int chosen_index = g_next_backend;
    g_next_backend = (g_next_backend + 1) % g_backend_ports.size();
    return g_backend_ports[chosen_index];
}

Same load pattern, same client count, run against v4: zero TSan reports.

Why a mutex over std::atomic<int>: std::atomic<int>::fetch_add() would make the increment itself atomic, but the critical section here is "reserve an index AND use it to index into the backend list" as one indivisible step — a plain atomic increment covers the first part but not the dependency between the two. A mutex is the more conservative, easier-to-reason-about choice for a read-modify-write-then-use sequence, and it's consistent with the synchronization primitive already used elsewhere in the project (the thread pool's task queue). atomic<int> was considered and is a reasonable alternative worth discussing, since the critical section here is small.

Reproducing this

g++ -std=c++17 -Wall -Wextra -pthread -fsanitize=thread -g -O1 \
    proxy_v3_racy.cpp -o proxy_v3_tsan
g++ -std=c++17 -Wall -Wextra -pthread -fsanitize=thread -g -O1 \
    proxy_v4_fixed.cpp -o proxy_v4_tsan

Run either binary, point several backend instances at it, and drive concurrent client load (see race_repro.sh in this repo). TSan reports races directly to stderr as they're detected — no diffing or special tooling needed beyond -fsanitize=thread.

Benchmarking

Measured on Intel i5-9300H (4c/8t), 200 requests per run against a 50ms-artificial-delay backend (I/O-bound workload — makes thread overlap visible rather than measuring raw CPU work).

Version	Threads	Throughput (req/s)	Avg latency (ms)
v1 (single-threaded)	1	18.94	52.81
v4 (thread pool)	1	19.55	51.14
v4	2	39.12	25.56
v4	4	74.99	13.34
v4	8	137.55	7.27
v4	16	239.52	4.17
v4	32	445.43	2.25
v4	64	583.09	1.72
v5 (epoll, 1 thread)	1	696.86	1.44

v5 (single-threaded epoll) outperforms v4 at every thread count tested, including 64 threads — higher throughput and lower latency using one thread instead of 64. This is the expected outcome for an I/O-bound workload: v4 scales by adding OS threads (each with real scheduling and context-switch overhead) to cover blocking time, while v5 eliminates blocking time entirely, so one thread can juggle all in-flight connections with no per-thread overhead at all.

Throughput scales near-linearly with thread count well past the 8 hardware threads available (35x at 64 threads vs. 1 thread). This is expected for an I/O-bound workload: each request spends most of its time blocked on recv() waiting for the backend, not consuming CPU, so threads far beyond the physical core count still do useful work by overlapping their wait time. A CPU-bound workload would plateau at ~8 threads instead (see backend_server_cpu.cpp, written but not yet run, for testing that distinction separately).

v1 and v4-at-1-thread match closely (19.03 vs 19.54 req/s), confirming the thread pool adds no meaningful overhead when concurrency isn't used — the gains are entirely from parallelism, not from switching frameworks.

Reproduce with benchmark.sh in this repo.

Known limitations / explicitly out of scope

• No HTTP-level parsing (headers, keep-alive, chunked encoding) — this is a Layer-4 proxy by design.
• No backend health-checking or fault tolerance.
• epoll-based variant: planned as a follow-up extension, not part of the core deliverable.
• No distributed coordination — single-node load balancer only.
• v1/v2/v3/v4 relay request-then-response in a simple alternating fashion, suitable for short-lived request/response traffic. A fully general bidirectional relay would need select/poll/epoll.

Build

g++ -std=c++17 -Wall -Wextra -pthread proxy_v4_fixed.cpp -o proxy
g++ -std=c++17 -Wall -Wextra -pthread backend_server.cpp -o backend_server