feat: add overload control and measured coordinator benchmarks - #2
Merged
Conversation
New 'client bench' command: submits a batch of deterministic jobs from several client connections at once (one CoordinatorClient per generator thread — the client is a one-request-in-flight blocking protocol), waits for every job to reach a terminal state, and reports throughput, submit-acknowledgement latency, and submit-to-observed-terminal latency as nearest-rank p50/p95/p99. An optional untimed warm-up batch runs first so the timed phase is not measuring a stone-cold JIT. End-to-end numbers are an upper bound (quantized by the 20 ms status-poll interval), and generator and system share one machine; this is a local engineering benchmark for comparing the system against itself under different configurations, not a rigorous performance benchmark.
Baseline measurements from the new load generator, before admission control, on an Apple M1 Pro (3 workers x capacity 4 = 12 slots): - Durable write-through costs ~5.4x throughput on cheap SHA256 jobs (896 jobs/s file-backed vs 4843 jobs/s in-memory): synchronous per-transition SQLite writes on the single core thread are the bottleneck, not the task work. - Under sustained overload (240x 500ms sleep jobs, 12 slots), the coordinator accepts every job in ~4ms while end-to-end latency climbs to ~5s median / ~10s tail. Submit ack stays flat as the queue grows: the client gets no backpressure signal and the job set grows unbounded. This is the limitation the admission-control change addresses. Records methodology and honest benchmark limitations.
Add --max-active-jobs (default 10000): the coordinator holds at most this many active (non-terminal) jobs before rejecting new submissions, so it sheds load instead of buffering unboundedly. Accounting is O(1). The core keeps a running activeJobCount in its single-writer state rather than scanning the jobs map on every submit (which would make admission most expensive exactly when it is most needed). It is incremented once when a job enters the active set (a new submission or a non-terminal job recovered from storage) and decremented once when a job leaves it. Every active -> terminal transition funnels through one persistRetired() helper, so the decrement lives in exactly one place and a guard makes underflow impossible. Retries stay active (RUNNING -> RETRY_WAIT -> QUEUED) and never touch the counter. Because all of this runs on the existing single core thread, the check and the counter need no extra locking. Rejections use a new typed SUBMIT_REJECTED protocol message carrying reason, activeCount, limit, and retryable=true - deliberately distinct from ErrorReply so a client can tell "valid request, back off and retry" apart from "malformed request". CoordinatorClient surfaces it as a dedicated SubmitRejectedException; the CLI prints it and exits non-zero, and the load generator counts it as a rejection. Overload semantics: only new submissions are admission-controlled; existing jobs are never discarded; a recovered coordinator may sit above a since-lowered limit and simply rejects new work until the count falls back below it.
Adds 14 tests for the active-job admission counter:
- SubmitRejected wire round-trip and type discriminator (common).
- max-active-jobs config parsing and rejection of non-positive values.
- Integration (AdmissionControlIT), using no-worker coordinators to pin
the active set and a ScriptedWorker for exact retry timing:
* accepts exactly up to the limit, then rejects with a typed
SubmitRejectedException carrying activeCount/limit/retryable, and the
rejection persists nothing;
* completion, permanent failure, and cancellation each free one slot;
* a retry (RUNNING -> RETRY_WAIT -> RUNNING) neither frees nor consumes
an extra slot, and the slot is freed exactly once on completion;
* 8 clients racing 80 submissions at a limit of 20 admit exactly 20 -
the single-writer core serialises admission without extra locking;
* recovery reconstructs the count from non-terminal jobs only, ignoring
terminal ones;
* recovery above a since-lowered limit keeps the recovered jobs, rejects
new work, and resumes accepting once the backlog drains below the limit.
Exposes Coordinator.activeJobCount() so tests (and operators) can read the
counter directly. No timing-based performance assertions.
- MEASURED_BEHAVIOR.md: post-change section. Same overload load, bounded to --max-active-jobs 24, drops end-to-end p99 from ~10s to ~1s while shedding 216 of 240 as typed rejections; throughput unchanged (~23 jobs/s). Healthy SHA256 runs confirm the O(1) check adds no measurable overhead below the limit (896->897 file, 4843->4751 memory). - README: bench command, admission-control / overload semantics with the measured before/after, an honest-semantics bullet, updated limitations, and a MEASURED_BEHAVIOR link. - PROTOCOL.md: SUBMIT_REJECTED message and why it is distinct from ERROR. - DESIGN_DECISIONS.md: ADR 14 — reject-new over buffer-forever / block / drop-accepted, typed rejection, and O(1) race-free accounting on the single-writer thread. - FAILURE_MODEL.md: overload as a defined, bounded condition. - ARCHITECTURE.md: admission control among the core-thread components.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a measured overload-control milestone to the distributed task processor.
Benchmarking
Admission control
--max-active-jobsSUBMIT_REJECTEDprotocol response for retryable overloadMeasured result
For 240 × 500 ms tasks with 12 worker execution slots:
The change therefore defines overload behavior and bounds queueing latency rather than claiming to make worker execution faster.
Documentation
Updated:
Validation
./mvnw clean verify✅git diff --checkclean