IDDQueue

Documentation · Step-by-step setup

A PostgreSQL broker and Results backend for Dramatiq, built on synchronous Psycopg 3. Tasks, results and coordination live in PostgreSQL; no Redis or ORM is required by IDDQueue.

Current release: 0.13.0 — the first stable IDDQueue release. GitHub assets include the wheel, sdist and SHA256 checksums.

IDDQueue is a fork of DALIBO's dramatiq-pg (original package). It preserves the PostgreSQL license and original contributor credits.

What it provides

• Durable JSONB task/results storage, delayed tasks and crash recovery.
• Transactional publication alongside business writes; atomic batches and deduplication.
• PostgreSQL rate limiters, barriers and Dramatiq group callbacks.
• Pause/resume, cooperative cancellation, failed-task inspection/retry and attempt history.
• Fixed-interval scheduling and optional Prometheus queue metrics.
• Schema/prefix isolation and a tested FastAPI example.

Delivery is at least once: actors and callbacks must be idempotent. LISTEN/NOTIFY carries message IDs as wakeup hints; consumers claim authoritative SQL rows with session advisory locks. See behavior and limits.

Install

Requires Python 3.10+, Dramatiq 2.2.1+ and Psycopg 3.3.6+. CI tests Python 3.10 / 3.13 / 3.14 with PostgreSQL 14 / 18. See compatibility and API stability.

uv venv
uv pip install "iddqueue[binary]==0.13.0"

The binary extra supplies libpq. Base installs require system libpq; [binary,monitoring] also enables Prometheus support.

Migrating from dramatiq-pg? Follow the migration guide before changing imports, pools or database storage. Stop all participants, back up data, run the upgrade and restart consistently. Fresh initialization and an existing-storage upgrade are different operations.

Quickstart

Configure PostgreSQL using PGHOST, PGPORT, PGUSER, PGDATABASE and your normal credential mechanism. Initialize fresh storage:

uv run --no-sync iddqueue init

For existing storage, use iddqueue upgrade after following the migration guide. Both commands manage the complete feature schema.

Save as tasks.py:

import dramatiq
from iddqueue import PostgresBroker

broker = PostgresBroker()
dramatiq.set_broker(broker)

@dramatiq.actor(broker=broker, store_results=True)
def add(left: int, right: int):
    return left + right

Start a separate worker from the same directory/environment:

uv run --no-sync dramatiq --use-spawn --processes=1 --threads=1 tasks

In another terminal, publish and read the result:

uv run --no-sync python - <<'PY'
from tasks import add, broker

try:
    message = add.send(2, 3)
    print(message.message_id)
    print(message.get_result(backend=broker.backend, block=True, timeout=10000))
finally:
    broker.close()
PY

The result is 5; timeouts are milliseconds. Each process creates its own pool. Close broker-owned pools on shutdown; callers close pools they supply.

Guides and examples

Domain actors and one worker bootstrap are included since RC3.

• Documentation index, user guide, API.
• Detailed recipes: transactions, middleware, controls, metrics and schedules.
• FastAPI: lifespan, async endpoint thread offload and separate worker.
• Deployment and retention, why PostgreSQL.
• Release notes, changelog, release checks.

Ordinary broker publication, consumers and result APIs are synchronous. Async actors use Dramatiq's AsyncIO middleware; FastAPI async endpoints offload publication to a thread. 0.13.0 includes explicit awaitable single/batch transactional publication on Psycopg AsyncConnection; see transactional recipes.

Session-bound locks/listeners require persistent database sessions; PgBouncer transaction pooling is unsuitable. Namespace separation is not access control. The legacy django-dramatiq-pg integration has been archived since September 3, 2024; IDDQueue compatibility is unverified. No tested Django integration is currently provided.

Compared with dramatiq-pg

Baseline: dramatiq-pg 0.12.0, compared with the published IDDQueue 0.13.0 release. This describes the published version, not every future upstream revision. Both projects provide a PostgreSQL Dramatiq broker.

Area	dramatiq-pg 0.12.0	IDDQueue 0.13.0 / practical benefit
Core storage and delivery	JSONB tasks/results, delayed tasks, LISTEN/NOTIFY, advisory locks, recovery and maintenance CLI	Preserved; at-least-once delivery still requires idempotent actors
Actor initialization	Broker configured before actor imports	Opt-in Domain declarations import before DSN; explicit startup registration and domain queues
Runtime	Python >=3.6,<4; Dramatiq >=1.5,<2; Psycopg 2	Python >=3.10,<4; Dramatiq >=2.2.1,<3; synchronous Psycopg 3 and psycopg-pool
Pool lifecycle	Psycopg 2 pools	Lazy owned pools, explicit close, caller-owned pool support
Transactional publication	Regular enqueue	Caller-owned sync/async Psycopg 3 or optional sync SQLAlchemy transaction: commit business data and tasks together
Coordination	No PostgreSQL rate-limit/barrier backend	PostgreSQL rate limits, durable barriers and standard GroupCallbacks without Redis
Task operations	stats, purge, recover, flush	Failed-task inspection/targeted retry, pause/resume, cooperative cancellation and opt-in attempt history
Publication controls	Single-message enqueue	Deduplication keys/TTL and atomic batches of up to 1000; deduplication is not exactly-once execution
Scheduling	Per-message delay	Also a multi-process fixed-interval scheduler; no cron/calendar expressions
Observability	Basic CLI statistics	Queue/domain backlog and age, native error/retry counters, optional Prometheus collector and Grafana kit
Storage namespaces	Configurable schema/table prefix; shared channel/lock domains	Instance-specific queries and schema/prefix-specific channels/locks; storage separation is not access control
Dramatiq integration	Broker/results implementation	Verified pipelines, groups, AsyncIO and standard middleware; these are Dramatiq features, not a replacement orchestration engine
Compatibility evidence	Historical upstream tests	CI on Python 3.10/3.13/3.14 × PostgreSQL 14/18, functional tests and installed-wheel checks

See migration from dramatiq-pg for dependency, pool, import, CLI and database changes, including backup and rollback. No throughput advantage is claimed here. The 0.13.0 release is available on PyPI; the legacy Django integration is archived and has not been verified with IDDQueue.

Development and support

Use CONTRIBUTING for locked uv setup and the full release gate. Functional tests require a dedicated PostgreSQL instance: they terminate sessions and crash/restart workers. Project planning and evidence live in OpenSpec. Report reproducible issues on GitHub.

License and credits

Released under the PostgreSQL license, preserving Copyright (c) 2019, DALIBO and the original author Étienne BERSAC.

Thanks to upstream contributors Andy Freeland, Curtis Maloney (Django support), Federico Caselli, Giuseppe Papallo and Rafal Kwasny. The historical upstream logo was created by Damien CAZEILS. Original changelog remains available as historical context.