Worker split

.env.production.example

@@ -16,11 +16,13 @@ ACME_EMAIL=you@yourcompany.com
 COOKIE_DOMAIN=.content.yourcompany.com
 COOKIE_SAMESITE=lax
 
-# Encrypts the API keys stored in the settings table.
+# Encrypts the API keys stored in the settings table. REQUIRED in production —
+# the API refuses to start without it.
 #
-# MOVE THE EXISTING VALUE from your local .env. A different secret does not
-# fail loudly — it makes every key already saved unreadable, which reads as
-# "the key vanished".
+# MOVE THE EXISTING VALUE from your local .env. Leaving this blank is caught at
+# boot, but setting the WRONG value is not: a different secret does not fail
+# loudly, it makes every key already saved unreadable, which reads on the page
+# as "the key vanished".
 CONTENTOS_SECRET=
 
 # How the model is reached on a server.

.gitignore

@@ -61,3 +61,14 @@ next-env.d.ts
 # the NestJS API has its own install and build output
 api/node_modules
 api/dist
+
+# the Tauri desktop app: Rust build output, and the ACL schemas tauri-build
+# writes from tauri.conf.json on every compile. Cargo.lock is NOT here — this
+# is an application, so the exact versions it was built against are part of it.
+src-tauri/target
+src-tauri/gen/schemas
+
+# The Node the desktop app ships. ~90 MB, fetched and checksummed by
+# src-tauri/scripts/fetch-node.mjs at build time — a runtime is not a thing to
+# keep in a repository.
+src-tauri/binaries

.vscode/settings.json

@@ -0,0 +1,9 @@
+{
+  // Use the TypeScript in node_modules (5.9.3) for IntelliSense, not the one
+  // bundled with the editor. A newer bundled compiler reports options this
+  // repo's compiler still accepts — `moduleResolution: node` and `baseUrl` in
+  // api/tsconfig.json are deprecated in TypeScript 7 — as errors in files that
+  // build and typecheck clean. Editor and CI now agree on one compiler.
+  "typescript.tsdk": "node_modules/typescript/lib",
+  "typescript.enablePromptUseWorkspaceTsdk": true
+}

AGENTS.md

@@ -10,11 +10,17 @@ This block is written and re-added by `next dev` — verify at `node_modules/nex
 
 # Read these first
 
-Four short files, written so a session does not have to rediscover the same
-things by grepping and clicking. Read the one that matches the job.
+Written so a session does not have to rediscover the same things by grepping
+and clicking. Read the one that matches the job.
 
-- **`docs/ARCHITECTURE.md`** — what lives where, the routes, the data flow, and
-  the vocabulary (the UI says "Template" where the code says `pack`).
+- **`docs/ARCHITECTURE.md`** — what lives where, the routes, the two data flows,
+  and the vocabulary (the UI says "Template" where the code says `pack`, and
+  "Machine" where it says `worker`).
+- **`docs/WORKER.md`** — why writing a section and fetching a reel became jobs
+  on a queue that teammates' laptops claim: the `workers` and `jobs` tables, the
+  claim query, the leases, and what a machine's bearer token may reach. Read it
+  before touching `worker/`, `api/src/workers/`, `lib/server/repos/jobs.ts` or
+  `advance()` in `lib/server/services/runs.ts`.
 - **`docs/DEVELOPING.md`** — how to start it (port 3333), what to run to check
   work, why `next build` must not touch the dev server, and what to do when
   another session is editing the same files.
@@ -42,3 +48,25 @@ Compose from the UI kit in `components/ui` — `Page`, `PageHeader`, `Toolbar`,
   listed in `components/ui/README.md` — read that file before adding anything.
 - Every icon-only control takes a `label`, which becomes its tooltip and its
   accessible name.
+
+# Code that runs on somebody else's machine
+
+`worker/` is a sidecar on a teammate's laptop. It claims jobs over HTTPS, spawns
+the CLIs, and posts the results back. It has no database and must never look
+like it wants one.
+
+- From `lib/server` it may import `tools.ts`, `brain-defs.ts` and
+  `brain-transports.ts`, and **nothing else**. Everything else there reaches a
+  repo, and an import of a repo puts `pg` on somebody's desktop. That is how the
+  worker ended up carrying a hand-copied brain table once already — and the copy
+  drifted, on the half of the system that does the work.
+- Use relative specifiers in anything the worker imports, not `@/`. It runs
+  under Node's own type stripping through `worker/resolve-ts.mjs`, which knows
+  nothing about tsconfig `paths`: an `@/` import typechecks, passes review, and
+  is a module-not-found the first time somebody starts the sidecar.
+- The server decides and the worker executes. A payload carries finished text
+  and a command — never a pack, a rule, a dependency graph or the brand voice —
+  and a worker that cannot honour what it was told fails the job rather than
+  quietly answering some other way.
+- `npm run worker` starts it; there is no build step. `docs/WORKER.md` lists the
+  environment it takes and why two of the variables have no default.

README.md

@@ -259,16 +259,28 @@ gone, so an old run can still be rewritten.
 
 ### Running one
 
-**No API key needed.** The default brain shells out to the `claude` CLI already
-installed on this machine, which is what the old system called its local brain.
-Set `ANTHROPIC_API_KEY` and it uses the Anthropic SDK instead.
-
-Sections run **one request each**, driven from the browser in dependency order.
-That is deliberate: you watch it happen, and a failure costs one section rather
-than the run. Rewriting a section is the same request again. The server checks
-the dependencies rather than trusting the caller — asking for the Bangla script
-first would otherwise hand the model an empty "already generated" block, and it
-would invent the English script it was meant to be translating.
+**No API key needed.** The brain shells out to the `claude` CLI already signed
+in on a machine, billed to that subscription rather than per token. Which model
+writes is a property of the **workspace**, not of the console.
+
+Pressing Run does not write anything itself. `advance()` looks at the template's
+dependency waves, and for every section whose inputs are written it queues a
+**job**; a worker — a process on somebody's own computer, with the CLIs and the
+GPU on it — claims the job, writes the section, and posts it back, at which
+point `advance()` runs again. So a run survives the tab closing, the API
+restarting, and the machine that started it going to sleep: another one picks
+it up. `docs/WORKER.md` is the whole design.
+
+A failure still costs one section rather than the run, and rewriting one is
+still a single request. The server checks the dependencies rather than trusting
+the caller — asking for the Bangla script first would otherwise hand the model
+an empty "already generated" block, and it would invent the English script it
+was meant to be translating.
+
+`ANTHROPIC_API_KEY` still works and is **off by default**. A workspace can turn
+on API fallback, which lets the server write a section when no machine is
+awake; it is opt-in because a fallback that fired by itself would spend money
+at the moment nobody was watching.
 
 ### First run
 

api/Dockerfile

@@ -14,10 +14,21 @@ WORKDIR /app
 COPY --from=deps /app/api/node_modules ./api/node_modules
 # The shared code, and the root packages it imports (pg, the Anthropic SDK).
 COPY package.json package-lock.json* ./
-RUN npm ci --omit=dev
+# Everything, including devDependencies — `tsc` needs them.
+#
+# This was `--omit=dev`, which looks right for an image and is not: the type
+# packages for what lib/server imports live in the root's devDependencies, so
+# `@types/pg` was absent and the build below failed with a screenful of
+# implicit-any on code that compiles fine on a laptop. The dev dependencies
+# are needed to BUILD and not to RUN, which is what `npm prune` is for.
+RUN npm ci
 COPY lib ./lib
 COPY api ./api
 RUN cd api && npm run build
+# Now that tsc is done, take them back out. The runtime stage copies this
+# node_modules wholesale, so pruning here is what keeps the shipped image to
+# what a request actually reaches.
+RUN npm prune --omit=dev
 
 FROM node:24-bookworm-slim AS runtime
 WORKDIR /app
@@ -28,12 +39,22 @@ ENV TS_NODE_BASEURL=./dist
 # Downloads and frames live here; mount a volume on it.
 ENV CONTENTOS_DATA_DIR=/app/.data
 
-# The tools the ingest runs. Without them the API still starts; source-video
-# features report the binary as missing, which is the truth.
+# ffmpeg, and only ffmpeg.
+#
+# This image used to carry python3 and a downloaded yt-dlp as well, because
+# fetching a link ran here. It does not any more: an `ingest_source` job goes
+# to a machine that has yt-dlp, the download happens there, and what comes
+# back is the stills and a 16 kHz WAV. Nothing on this side touches a URL's
+# media.
+#
+# ffmpeg stays for the one ingest that is still local — a BROWSER UPLOAD,
+# whose bytes already arrived here, and where posting them back out to a
+# laptop to cut eight stills would double the transfer to learn nothing.
+# `ffprobe` ships in the same package. Whisper is deliberately absent:
+# transcription is a job over that WAV, aimed at a desktop with a GPU, and a
+# CPU transcriber on a 2GB VPS is the slowest way to get the worst answer.
 RUN apt-get update \
- && apt-get install -y --no-install-recommends ffmpeg python3 ca-certificates curl \
- && curl -fsSL https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp \
- && chmod a+rx /usr/local/bin/yt-dlp \
+ && apt-get install -y --no-install-recommends ffmpeg ca-certificates \
  && rm -rf /var/lib/apt/lists/*
 
 COPY --from=build /app/node_modules ./node_modules

api/src/app.module.ts

@@ -15,15 +15,18 @@ import { IntegrationsController } from "./integrations/integrations.controller";
 import { RunsController } from "./runs/runs.controller";
 import { SourcesController } from "./sources/sources.controller";
 import { ResearchController } from "./research/research.controller";
+import { WorkersController } from "./workers/workers.controller";
+import { MachinesController } from "./machines/machines.controller";
+import { Reaper } from "./workers/reaper";
 
 /**
  * One module, because there is one database and one set of rules.
  *
  * Nest's usual shape is a module per feature, each with its own providers.
  * Here the "providers" are the functions in `lib/server` — repos and services
  * shared with the repo root — and a controller is a thin translation from HTTP
- * to those. Ten controllers in one module is honest about that; ten modules
- * each importing nothing would be ceremony.
+ * to those. Thirteen controllers in one module is honest about that;
+ * thirteen modules each importing nothing would be ceremony.
  *
  * The session guard is global. Every route is behind it unless the handler
  * says otherwise with `@Public()` — the auth gap that the audit found was
@@ -43,7 +46,22 @@ import { ResearchController } from "./research/research.controller";
     RunsController,
     SourcesController,
     ResearchController,
+    WorkersController,
+    /*
+     * The console's view of the machines, not the machines' own protocol.
+     * `MachinesController` is behind the global session guard like every other
+     * console route; `WorkersController` above is bearer-token only. Two
+     * controllers because they are two principals — see docs/WORKER.md.
+     */
+    MachinesController,
   ],
-  providers: [{ provide: APP_GUARD, useClass: SessionGuard }],
+  /*
+   * The reaper is a provider rather than something started in `main`, so that
+   * its interval is owned by the application: Nest constructs it once and,
+   * with shutdown hooks on, clears it on the way out. A `setInterval` in
+   * `main` would survive `app.close()` and hold a database pool open forever
+   * in tests and on a dev reload.
+   */
+  providers: [{ provide: APP_GUARD, useClass: SessionGuard }, Reaper],
 })
 export class AppModule {}