Skip to content

feat: add durable idempotent job submissions - #4

Merged
achraf059 merged 4 commits into
mainfrom
feat/idempotent-submissions
Sep 7, 2026
Merged

feat: add durable idempotent job submissions#4
achraf059 merged 4 commits into
mainfrom
feat/idempotent-submissions

Conversation

@achraf059

Copy link
Copy Markdown
Owner

Summary

Adds optional durable idempotency keys for client job submissions.

A client may now associate a stable idempotency key with a logical submission. Repeating the same submission with the same key returns the original job ID instead of creating another logical job.

The mapping is persisted in SQLite and restored after coordinator restart.

What changed

  • Added optional idempotencyKey to SubmitJob
  • Added client and CLI support for --idempotency-key
  • Added coordinator-side key → job ID deduplication
  • Deduplication occurs before admission control
  • Added conflict detection when a key is reused with a different submission
  • Persisted idempotency keys in SQLite
  • Added backward-compatible database migration
  • Added a partial unique index for non-null keys
  • Rebuilds the in-memory deduplication map during recovery
  • Preserved existing no-key submission behavior

Semantics

  • Same key + same submission → returns original job ID
  • Same key + different payload/task/maxAttempts → error
  • Known-key duplicates bypass admission control because they create no new job
  • New keys still pass through normal admission control
  • Terminal jobs retain their keys
  • Execution retries remain separate

Important limitation

This provides submission deduplication only.

It does not provide exactly-once task execution. Worker execution remains at-least-once.

Automatic retry of ambiguous transport failures is also still intentionally disabled.

Verification

  • ./mvnw clean verify — BUILD SUCCESS
  • git diff --check — clean
  • Integration suite: 49 tests
  • 12 new end-to-end idempotent-submission tests
  • Serialization, migration, recovery, conflict, admission, lifecycle, and concurrency behavior covered

@achraf059
achraf059 merged commit b558cfc into main Sep 7, 2026
1 check passed
@achraf059
achraf059 deleted the feat/idempotent-submissions branch September 7, 2026 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant