Feat/lease deadlines cancellation - #1
Merged
Conversation
- CANCELLED terminal state, reachable from QUEUED, RETRY_WAIT, and RUNNING; enforced by the state machine like every other transition - Attempt leases are now time-bounded: assignTo stamps a deadline (coordinator clock + configurable execution timeout, default 10 min, --task-timeout-millis); the deadline is cleared with the lease - Job.cancel revokes any active lease; Job.isDeadlineExpired documents that expiry is suspicion of a stuck task, not proof of worker death - SQLite schema gains execution_timeout and deadline columns, with an in-place migration so pre-deadline databases remain loadable - Unit tests: CANCELLED transitions, deadline stamping/clearing, cancellation from each state, legacy-database migration
Closes the liveness gap where a task wedges on a worker that stays alive and keeps heartbeating: worker liveness no longer implies task progress. Coordinator (single-writer core, unchanged model): - the sweep gains detectExpiredDeadlines(): a RUNNING job past its coordinator-clock deadline has its lease revoked and is retried or failed via the existing retry policy, exactly like a lost worker - revokeAttemptOnWorker() sends a best-effort TASK_CANCEL and frees the worker's capacity slot; a send failure is harmless because the lease is already revoked, so any late result is stale-rejected as before - cancelJob() supports client cancellation: QUEUED/RETRY_WAIT/RUNNING -> CANCELLED (revoking any lease), terminal jobs reported unchanged Protocol (additive, unknown-type-tolerant): - TASK_CANCEL (coordinator->worker: jobId + attemptId) - CANCEL_JOB / JOB_CANCEL (client request/reply) Worker: - cooperative cancellation via thread interruption; InFlightTasks tracks each future by (jobId, attemptId) so a cancel for a superseded attempt can never interrupt a newer one. No hard preemption; a task that ignores interruption still has its lease revoked coordinator-side. Client CLI: new 'cancel <job-id>' command.
…sults New DeadlineAndCancellationIT (7 tests): - task exceeds its deadline on a healthy, still-heartbeating worker -> TASK_CANCEL sent, job reassigned, completes on attempt 2 - late success from a timed-out attempt is stale-rejected - deadline expiry on the final attempt -> FAILED with a timeout error - cancel a QUEUED job -> CANCELLED, never scheduled even once a worker joins - cancel a RUNNING SLEEP -> real worker interrupted (finishes in <5s not 30s), capacity freed, state CANCELLED - a worker that ignores the cancel and reports success cannot resurrect a CANCELLED job - cancelling terminal/unknown jobs is reported, not misapplied CoordinatorRestartIT: a restart discards the old wall-clock deadline and requeues the in-flight job (rather than letting a stale deadline expire it immediately) — the fresh assignment gets a fresh deadline. ScriptedWorker records received TASK_CANCEL messages (awaitCancel); MessageSerializationTest round-trips the three new message types.
- README: CANCELLED in the state diagram, time-bounded lease explanation, execution-deadline run instructions, cancel command, updated semantics and limitations - PROTOCOL.md: TASK_CANCEL, CANCEL_JOB/JOB_CANCEL, CANCELLED in snapshots - FAILURE_MODEL.md: stuck-task-on-healthy-worker scenario, client cancellation, stale-deadline recovery rule, tuning entry - DESIGN_DECISIONS.md: ADR 13 on time-bounded leases — coordinator-clock expiry, reuse over new machinery, cooperative (not preemptive) cancel, and the explicit non-claims (still at-least-once, not real-time) - ARCHITECTURE.md: deadline enforcer in the sweep and core components
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds time-bounded attempt leases and cooperative cancellation to close the
liveness gap where a task can remain stuck on a healthy worker indefinitely.
Changes
cancel <job-id>supportVerification
./mvnw clean verifypassesgit diff --checkpassesSemantics
Cancellation is cooperative, not hard preemption.
Execution remains at-least-once.
Late results from revoked attempts are rejected using the existing attempt-lease mechanism.