feat(hub): add PostgreSQL background jobs

hub/.env.example

@@ -4,6 +4,10 @@ PORT=3001
 # API logging threshold: trace, debug, info, warn, error, fatal, or silent.
 LOG_LEVEL=info
 
+# Independent worker concurrency. Both default to one (supported range: 1-100).
+ON_DEMAND_JOB_CONCURRENCY=1
+SCHEDULED_JOB_CONCURRENCY=1
+
 # Local development only. The devcontainer supplies its own database hostname.
 DATABASE_URL=postgresql://replica:replica@127.0.0.1:5432/replica
 

hub/README.md

@@ -4,7 +4,7 @@ Hub is being rebuilt on `build/hub/new` using TypeScript, Express, React, Vite,
 
 The API has one route: `GET /api/health`. The frontend displays the existing Replica logo and supports light and dark themes. Existing components, features, and pages are kept in Storybook, but they are not connected to the new API.
 
-Contributors can use PostgreSQL and Prisma locally or in the devcontainer. The database has no product models yet. Sign-in, saved data, live updates, background jobs, and deployment are not implemented.
+Contributors can use PostgreSQL and Prisma locally or in the devcontainer. The [background-job foundation](docs/background-jobs.md) provides PostgreSQL-backed BullMQ queues and independent on-demand and scheduled workers, without product jobs yet. The database has no product models. Sign-in, saved product data, live updates, and deployment are not implemented.
 
 Use [Development](docs/development.md) to start the API and frontend. Review [Architecture](docs/architecture.md), [code architecture conventions](docs/backend-architecture.md), and [coding conventions](docs/coding-conventions.md) before adding behavior.
 

hub/docs/README.md

@@ -8,6 +8,7 @@ Hub is being rebuilt. The [Hub overview](../README.md) describes what the baseli
 - [Development](development.md): local API and frontend setup.
 - [Database](database.md): Prisma schema conventions, migrations, and the shared client.
 - [Logging](logging.md): application events, automatic request records, and payload safety.
+- [Background jobs](background-jobs.md): PostgreSQL queues, workers, schedules, setup, and test isolation.
 - [Testing and operations](testing-and-operations.md): baseline checks and their limits.
 - [Storybook](storybook.md): retained presentation, fixtures, and screenshot evidence.
 - [Prototype canvas](prototype-canvas.md): the separately retained extension.

hub/docs/architecture.md

@@ -8,7 +8,7 @@ The team has chosen TypeScript, Express for the API, React and Vite for the fron
 
 The API responds to a health check, and the frontend displays the logo with light and dark themes. Storybook keeps the existing components and pages available for reuse.
 
-Contributors can use PostgreSQL and Prisma locally or in the devcontainer. The database has no product models yet. Sign-in, deployment, background jobs, and product APIs have not been built. This app cannot yet find repositories, import Builds, host previews, save product data, or support reviews.
+Contributors can use PostgreSQL and Prisma locally or in the devcontainer. The database has no product models yet. Sign-in, deployment, and product APIs have not been built. This app cannot yet find repositories, import Builds, host previews, save product data, or support reviews.
 
 ## Backend
 
@@ -20,6 +20,8 @@ The [database guide](database.md) defines the multi-file Prisma schema and share
 
 The [Pino integration](logging.md) writes structured application and HTTP logs to stdout. `createApp()` installs request logging and JSON parsing before routes, so new endpoints inherit completion records and server-generated request IDs without explicit logging calls.
 
+The [background-job foundation](background-jobs.md) uses BullMQ with PostgreSQL and separate on-demand and scheduled worker processes. Setup migrates BullMQ's own schema independently of Prisma. Product processors and schedules are not registered yet.
+
 ## Frontend
 
 ```mermaid

hub/docs/background-jobs.md

@@ -0,0 +1,93 @@
+# Background jobs
+
+Hub uses BullMQ's PostgreSQL backend for internal background work. The foundation provides queues, validated job payloads, recurring schedules, and independent worker commands. No product jobs or production schedules are registered yet. The Rails jobs have not been ported.
+
+## Queues and workers
+
+Two stable queue names separate execution capacity:
+
+| Queue | Work | Command from `hub/` |
+| --- | --- | --- |
+| `on-demand` | Work requested by an application action | `npm run worker:on-demand` |
+| `scheduled` | Recurring work and work spawned by recurring sweeps | `npm run worker:scheduled` |
+
+Each command starts one Node.js process consuming only its assigned queue. The API starts neither worker. Multiple processes may consume the same queue when more capacity is needed. Separate queues do not isolate PostgreSQL, CPU, or external service quotas.
+
+`ON_DEMAND_JOB_CONCURRENCY` and `SCHEDULED_JOB_CONCURRENCY` independently set the number of active jobs per worker process. Both default to one and accept integers from 1 through 100. CPU-heavy JavaScript still blocks that worker's event loop; move that work to a subprocess or an appropriate execution boundary when adding the domain.
+
+`server/workers/configuration.ts` is the composition point for processors and schedules. Its empty registrations are intentional until product domains exist. Worker startup logs readiness after connecting; unknown job names fail permanently rather than disappearing.
+
+Workers stop accepting work on `SIGINT` or `SIGTERM` and wait for active jobs. After 30 seconds, the command exits unsuccessfully rather than waiting forever. Interrupted work can be retried by BullMQ's stalled-job recovery. Deployment stop timeouts must allow this shutdown window. Long-running domain work needs its own bounded execution and interruption policy; a queue lock is not a business lock.
+
+## Database and setup
+
+Jobs use `DATABASE_URL`, the same database as application data, with BullMQ-owned tables and functions in the `bullmq` schema. Prisma owns only the application schema. Do not copy BullMQ tables into Prisma models or change its migration files.
+
+From `hub/`, after installing dependencies and starting PostgreSQL:
+
+```sh
+npm run setup
+```
+
+This runs `prisma migrate deploy`, then BullMQ migrations, then schedule registration. A failed step stops setup. For local work that only needs the job schema, use `npm run jobs:setup`.
+
+Run setup once as a coordinated release step before starting the new application version. Do not run it in every API or worker startup. Migrations are not one transaction across Prisma and BullMQ, and changes must remain compatible with old processes during a rolling upgrade. BullMQ schema downgrades are not supported.
+
+After building, the compiled equivalents are:
+
+```sh
+npm run start:setup
+npm start
+npm run start:worker:on-demand
+npm run start:worker:scheduled
+```
+
+Run the last three commands as separate services, not sequential commands in one shell. These roles can use one application image in a future Compose deployment, with PostgreSQL shared and only the web service exposed. Production frontend hosting, an application image, service health checks, and release orchestration are not implemented by this foundation.
+
+Each queue or worker owns a small query pool and BullMQ's additional notification connections. The query pool is limited to two connections; concurrency does not increase that limit. Budget database connections across all API, producer, and worker instances. Creation functions require an explicit URL so tests and other callers cannot accidentally inherit a previously imported database client.
+
+## Defining work
+
+Keep stable names, a Zod payload schema, and retry options in a job definition. Producers need the definition, not the domain's execution code. Register a processor at the worker composition point using `createJobProcessor(definition, process, shouldRetry)`. The optional `shouldRetry(error)` predicate makes an execution failure permanent when it returns false; errors otherwise follow the job's retry options.
+
+Use the functions under `server/integrations/bullmq/`:
+
+| Function | Contract |
+| --- | --- |
+| `migrateJobDatabase(databaseUrl)` | Explicitly migrate BullMQ's schema and close the migration connection. |
+| `createJobQueue(options)` | Open a client for one named PostgreSQL queue. The caller must close it. |
+| `enqueueJob(queue, definition, payload, options)` | Validate JSON payload data and enqueue work with job-specific retries and optional delay, priority, or job ID. Failure rejects the call. |
+| `createJobProcessor(definition, process, shouldRetry)` | Bind typed execution to runtime payload validation. Invalid stored payloads fail permanently. |
+| `createJobWorker(options)` | Construct a stopped consumer for registered processors. The caller starts and closes it. Duplicate processor names are rejected. |
+| `upsertJobSchedule(queue, schedule)` | Validate and register an interval or timezone-explicit cron schedule on `scheduled`. |
+
+`runJobWorker()` owns process signals and consumer lifetime outside the integration. Integration code does not import domains, repositories, or the shared logger; worker composition injects processors and logging.
+
+Payloads must serialize as JSON. Prefer record IDs and necessary revision identifiers over snapshots, credentials, or private text. Load current records and recheck access when execution begins. Do not rely on TypeScript alone: enqueue and execution both validate input.
+
+Retries default to one attempt. A definition can specify `attempts`, `backoff`, and `priority`; an enqueue can override priority. Choose retries according to the operation, not a universal policy. Retention keeps at most 1,000 successful jobs for up to one day and 5,000 failed jobs for up to seven days. BullMQ applies removal lazily as jobs finish, not as a strict expiry timer. Job records can retain payloads and failure messages, so restrict database access and keep secrets out of thrown errors as well as payloads.
+
+Worker logs contain queue, job name, job ID, attempt counts, and error types, never payloads, results, or raw exception messages. Domain functions should report safe failure codes and necessary context through the existing logging policy. Connection failures and command failures produce explicit error records.
+
+## Schedules and delivery guarantees
+
+A schedule has a stable ID within a stable queue. Setup calls `upsertJobSchedule()` with that identity, so repeated releases update one schedule instead of adding another. Removing an entry from code does not remove its persisted schedule: explicitly call BullMQ's `removeJobScheduler()` when retiring or renaming one. Coordinate updates through one release step so old and new processes do not overwrite each other's definitions.
+
+Keep work spawned by a recurring sweep on `scheduled`; otherwise a sweep can consume on-demand capacity indirectly. Scheduled times are not execution deadlines, and a busy queue can run less often than its configured interval. Define missed-run and overlap rules when each domain is introduced.
+
+Assume at-least-once execution. A stable job or scheduler ID does not replace idempotent handlers or keyed exclusion across workers and queues. Sharing PostgreSQL also does not make a Prisma write and a BullMQ enqueue atomic. Future domains that need guaranteed submission must explicitly implement an outbox or another durable recovery contract.
+
+## Tests
+
+Ordinary `npm test` requires no database. Use `.db.test.ts` cases and `npm run test:database` for real BullMQ behavior. The [database harness](testing-and-operations.md#parallel-database-tests) applies Prisma and BullMQ setup to a unique template before cloning a database for each Vitest worker.
+
+Pass the current worker's resolved `DATABASE_URL` to every queue and consumer. Do not use the administrative `TEST_DATABASE_URL` or a production connection. Each test registers its own synthetic processors and schedules; setup does not install production schedules.
+
+Close consumers, queue clients, and any owned connections before the next case resets data. Clearing Vitest's module cache does not stop consumers. Cleanup clears job and schedule tables but preserves `public._prisma_migrations` and `bullmq.migration`; preserving a migration ledger is not preserving queued work.
+
+## References
+
+- [BullMQ PostgreSQL backend](https://docs.bullmq.io/guide/postgresql)
+- [Job Schedulers](https://docs.bullmq.io/guide/job-schedulers)
+- [Managing schedules](https://docs.bullmq.io/guide/job-schedulers/manage-job-schedulers)
+- [Worker shutdown](https://docs.bullmq.io/guide/workers/graceful-shutdown)

hub/docs/database.md

@@ -2,6 +2,8 @@
 
 Hub uses PostgreSQL and Prisma. Product models and migrations have not been added.
 
+[Background jobs](background-jobs.md) use the same database with BullMQ-owned tables and functions in the `bullmq` schema. Prisma does not own those objects. `npm run setup` applies Prisma migrations before BullMQ setup and schedule registration.
+
 [Compose](../.devcontainer/compose.yaml) pins the development database image. [package.json](../package.json) pins the Prisma packages to the same version. Dependency updates must follow the [seven-day dependency policy](../../docs/testing-and-review.md#update-dependencies).
 
 Choose the newest stable PostgreSQL major available from both Azure Database for PostgreSQL and Amazon RDS or Aurora. Managed services can offer different patch releases; the local image tracks a stable patch of that shared major. Check provider availability again when changing the major, rather than assuming that an upstream release is available everywhere.

hub/docs/development.md

@@ -40,6 +40,12 @@ Open <http://127.0.0.1:5173>. Vite serves the frontend and forwards `/api` reque
 
 The API reads an optional `PORT` environment variable, defaulting to 3001. If you change it, update the local Vite proxy target to match. The editor forwards ports 3001, 5173, and 6006 from the devcontainer. API startup reads `.env` when present; Compose's environment takes precedence.
 
+## Background workers
+
+The API and frontend do not start background consumers. After starting PostgreSQL, run `npm run setup` once, then start `npm run worker:on-demand` and `npm run worker:scheduled` in separate terminals. Both use `DATABASE_URL`; their concurrency settings are in `.env.example`.
+
+No product jobs or recurring schedules are installed yet. See [Background jobs](background-jobs.md) for definitions, shutdown, migrations, and the compiled commands intended for separate deployment services.
+
 ## Database lifecycle
 
 PostgreSQL stores data in a named Compose volume. Stopping or rebuilding the devcontainer preserves it. For a local Node.js setup, stop the database with `docker compose -f .devcontainer/compose.yaml down`; this also preserves data. Do not add `--volumes` unless you intend to permanently delete the local database.

hub/docs/testing-and-operations.md

@@ -51,15 +51,15 @@ Use `.test.ts` or `.test.tsx` for ordinary cases. Use `.db.test.ts` or `.db.test
 
 Start the PostgreSQL service as described in [Development](development.md) and set `TEST_DATABASE_URL` using `.env.example`. The devcontainer supplies this variable. Use a local database user with permission to create and drop databases, never a production connection.
 
-The database command uses `--passWithNoTests` so an empty selection exits successfully. It still fails when tests fail. There are no database test cases yet; an empty run does not prepare or exercise the databases.
+The database command uses `--passWithNoTests` so an empty selection exits successfully. It still fails when tests fail. The job integration cases exercise real PostgreSQL queues, schedules, and database resets.
 
 Database-enabled jsdom tests use the browser setup and database hooks together. They can arrange records through server code, but frontend code must not import server modules. The setup does not start an API or connect frontend requests to one.
 
 The database setup follows this lifecycle:
 
-1. Before workers start, setup creates a template database with a unique name for each project and run. It applies the current Prisma schema with `prisma db push`, then copies the template for every worker slot.
+1. Before workers start, setup creates a template database with a unique name for each project and run. It applies the current Prisma schema with `prisma db push` and BullMQ's migrations, closes those setup connections, then copies the template for every worker slot.
 2. Each worker receives its own `DATABASE_URL` before application modules load. Setup uses Vitest's [`VITEST_POOL_ID`](https://vitest.dev/guide/migration/jest#envs), which stays within the configured worker limit.
-3. Before every case, setup empties application tables across user schemas and resets generated identity counters. It preserves Prisma's migration records. Each case creates its own records; there is no shared seed data.
+3. Before every case, setup empties application and job tables across user schemas and resets generated identity counters. It preserves `public._prisma_migrations` and `bullmq.migration`. Each case creates its own records; there is no shared seed data.
 4. After the run, teardown drops only the databases it created, even if tests left connections open.
 
 Unique database names also isolate simultaneous runs. Before clearing tables, cleanup checks that the database name belongs to the test namespace and matches the connected database.
@@ -68,6 +68,10 @@ The helpers under `test/` prepare databases before Prisma Client and application
 
 Hub CI runs database tests against PostgreSQL on every supported Node major. Only that command starts the database service.
 
+Job tests use each worker's resolved `DATABASE_URL`, never `TEST_DATABASE_URL`. Close their consumers, queues, and owned connections before the next case's reset; clearing the module cache does not stop running jobs. See [Background jobs](background-jobs.md#tests) for job-specific lifecycle rules.
+
+Use `test/database/get-database-url` to read the worker URL and `test/jobs/open-queue` for a queue that closes at test completion. Tests that reset data mid-case must create and close their queues explicitly before resetting.
+
 ## Shared code
 
 The extension and build action still use Hub's Controls and canvas code. Keep their checks working: