feat: generate offline Qwen narration clips (#4)

.github/workflows/ci.yml

@@ -8,7 +8,7 @@ on:
 jobs:
   test:
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 45
     steps:
       - uses: actions/checkout@v5
       - uses: actions/setup-node@v5

README.md

@@ -4,8 +4,8 @@ Record reproducible feature demos from a short Playwright script. A coding agent
 writes the script, and `flow-reel` replays it in a sandbox on a virtual display while recording the screen. The same
 script gives the same video, step for step.
 
-This release records **silent** video with narration text, captions and step timings. Voice, cursor and zoom polish,
-and sound effects come in later releases.
+This release records **silent** video with narration text, captions, step timings and separate speech clips.
+Fitting and mixing the speech into the MP4, cursor and zoom polish, and sound effects come in later releases.
 
 ## Requirements
 
@@ -14,6 +14,8 @@ and sound effects come in later releases.
   pulled, `flow-reel` builds it locally once (a few minutes).
 - About 2.5 free CPU cores while recording. On an overloaded machine the recording stops with an error rather than
   save a video whose timing is off.
+- The image is about 7 GB unpacked; its build downloads about 2.5 GB of model weights. Local speech synthesis uses
+  four CPU threads after capture and can require more than 5 GiB of process memory, in addition to your app and OS.
 
 ## Install
 
@@ -62,12 +64,30 @@ Next to the script:
 | `settings.events.json` | Every move, click, keystroke run and scroll the runner made, timed the same way (schema version 1). |
 | `settings.srt` | Captions for a subtitle-capable video player. |
 | `settings.vtt` | WebVTT captions for a browser's `<video>` / `<track>`. |
+| `settings.audio/000.wav` | Local speech clips, named by zero-based step index. |
 
 Each caption shows one step's narration from its `startMs` to its `endMs`. Caption whitespace is normalized and the
 player wraps the text; the transcript keeps the original narration. Blank narration and steps lasting zero
 milliseconds stay in the transcript but produce no caption. Short actions can be too brief to read: step holds for
 voice clips come in a later release.
 
+Narration uses Qwen3-TTS 0.6B CustomVoice, Ryan in English, through the official Python runtime on CPU. Model and
+tokenizer assets are pinned and downloaded when the image is built; synthesis runs offline without a key. Speech
+is generated after capture finishes. The transcript records the voice, seed, and each clip's relative file path,
+duration in milliseconds and complete-file SHA-256 hash. Copy the audio folder with its transcript when sharing it.
+
+Synthesis normalizes whitespace without interpreting caption escapes; the transcript preserves the original text.
+Blank narration produces no clip. Clip numbers retain step indexes, so skipped steps leave gaps. Rerunning replaces
+the generated numeric WAVs in this demo's audio folder; keep personal audio under other filenames or folders.
+Nonblank narration on a zero-duration action still produces speech. Clips can outlast their action windows until
+the next release adds holds and muxing; the MP4 has no audio track yet. A synthesis failure fails the command.
+
+The local generator resets seed 0 for each clip and uses pinned settings. Matching hashes are checked in fresh
+processes on the tested CPU/container platform; byte equality across hardware or architectures is not guaranteed.
+On the tested Ryzen 5 3500U, one fresh process generated three short clips (6.4 seconds of speech) in 147 seconds,
+with about 5.4 GiB peak process memory. Runtime varies with host load. A synthesis job has a 15-minute timeout;
+individual steps also fail if generation reaches its limit, rather than returning truncated speech.
+
 Captions escape `&`, `<` and `>` so narration is emitted without raw tags. WebVTT displays the literal text; some
 SRT players display escape sequences such as `&lt;` instead. Use WebVTT in a browser when literal symbols matter.
 
@@ -86,7 +106,7 @@ a script have the same steps, each starting within ±150 ms.
 
 ## Known gaps
 
-- No synthesized voice or audio yet; narration is text only.
+- Speech clips are separate; the MP4 remains silent until step holds and audio muxing are added.
 - No cursor drawn in the video yet: the runner logs the motion in `events.json` for the polish step.
 - Playwright calls the runner doesn't cover run without any mouse motion (the warning names them).
 
@@ -96,7 +116,7 @@ a script have the same steps, each starting within ±150 ms.
 npm ci
 npm run typecheck
 npm test            # unit tests
-npm run test:e2e    # builds the image and records a fixture app twice (needs Docker and ffmpeg)
+npm run test:e2e    # builds the image, tests offline synthesis, and records twice (needs Docker and ffmpeg)
 ```
 
 Planning lives in `planning/` (see `groundwork.md`).

docker/Dockerfile

@@ -2,14 +2,19 @@
 FROM mcr.microsoft.com/playwright:v1.64.1-noble
 
 RUN apt-get update \
- && apt-get install -y --no-install-recommends xvfb x11-utils ffmpeg \
+ && apt-get install -y --no-install-recommends xvfb x11-utils ffmpeg python3-venv sox libsndfile1 time \
  && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /opt/flow-reel
+COPY docker/tts/requirements.txt docker/tts/requirements.txt
+RUN python3 -m venv /opt/flow-reel/tts-venv \
+ && /opt/flow-reel/tts-venv/bin/pip install --no-cache-dir -r docker/tts/requirements.txt
+COPY docker/tts ./docker/tts
+RUN /opt/flow-reel/tts-venv/bin/python docker/tts/download.py
 COPY package.json ./
 RUN npm install --omit=dev --no-fund --no-audit
 COPY dist ./dist
 
-ENV FLOW_REEL_IN_CONTAINER=1 HOME=/tmp
+ENV FLOW_REEL_IN_CONTAINER=1 HOME=/tmp HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 TOKENIZERS_PARALLELISM=false
 USER pwuser
 ENTRYPOINT ["node", "/opt/flow-reel/dist/cli.js"]

docker/tts/assets.json

@@ -0,0 +1,72 @@
+{
+  "repository": "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice",
+  "revision": "85e237c12c027371202489a0ec509ded67b5e4b5",
+  "files": [
+    {
+      "file": "config.json",
+      "size": 4908,
+      "hash": "5e8d5fc100536984e3a89bd87ef08e59e322b29b",
+      "algorithm": "git-blob-sha1"
+    },
+    {
+      "file": "generation_config.json",
+      "size": 245,
+      "hash": "1872b16d4535564abf2db5d6debe6cddd82b7f2e",
+      "algorithm": "git-blob-sha1"
+    },
+    {
+      "file": "merges.txt",
+      "size": 1671839,
+      "hash": "20024bfe7c83998e9aeaf98a0cd6a2ce6306c2f0",
+      "algorithm": "git-blob-sha1"
+    },
+    {
+      "file": "model.safetensors",
+      "size": 1811626576,
+      "hash": "bc3c7e785eb961179c25450d1acff03f839e0002f2f3a5aeb67b5735c0fa2adb",
+      "algorithm": "sha256"
+    },
+    {
+      "file": "preprocessor_config.json",
+      "size": 127,
+      "hash": "0525dd953bb9241912f7147666f0d535165d5d4f",
+      "algorithm": "git-blob-sha1"
+    },
+    {
+      "file": "speech_tokenizer/config.json",
+      "size": 2336,
+      "hash": "06cc8dc4c5ec8a1929086b71b98c313020d9268b",
+      "algorithm": "git-blob-sha1"
+    },
+    {
+      "file": "speech_tokenizer/configuration.json",
+      "size": 76,
+      "hash": "ab58e2eaf53cd14a1a2a7527d9261ceea93a24cd",
+      "algorithm": "git-blob-sha1"
+    },
+    {
+      "file": "speech_tokenizer/model.safetensors",
+      "size": 682293092,
+      "hash": "836b7b357f5ea43e889936a3709af68dfe3751881acefe4ecf0dbd30ba571258",
+      "algorithm": "sha256"
+    },
+    {
+      "file": "speech_tokenizer/preprocessor_config.json",
+      "size": 234,
+      "hash": "ba40914f4f49ab98a8ca545d4892ef7291a39592",
+      "algorithm": "git-blob-sha1"
+    },
+    {
+      "file": "tokenizer_config.json",
+      "size": 7344,
+      "hash": "6ff9fd60cc623bb54bbd603cbd418c97a11528d7",
+      "algorithm": "git-blob-sha1"
+    },
+    {
+      "file": "vocab.json",
+      "size": 2776833,
+      "hash": "4783fe10ac3adce15ac8f358ef5462739852c569",
+      "algorithm": "git-blob-sha1"
+    }
+  ]
+}

docker/tts/download.py

@@ -0,0 +1,28 @@
+"""Build-time only: fetch immutable model assets and verify their published hashes."""
+import hashlib
+import json
+from pathlib import Path
+from urllib.request import urlopen
+
+from huggingface_hub import hf_hub_download
+
+ROOT = Path("/opt/flow-reel/tts-model")
+manifest = json.loads(Path(__file__).with_name("assets.json").read_text())
+for asset in manifest["files"]:
+    path = Path(hf_hub_download(manifest["repository"], asset["file"], revision=manifest["revision"], local_dir=ROOT))
+    digest = hashlib.sha256() if asset["algorithm"] == "sha256" else hashlib.sha1()
+    if asset["algorithm"] == "git-blob-sha1":
+        digest.update(f"blob {asset['size']}\0".encode())
+    with path.open("rb") as stream:
+        while chunk := stream.read(1024 * 1024):
+            digest.update(chunk)
+    if path.stat().st_size != asset["size"] or digest.hexdigest() != asset["hash"]:
+        raise RuntimeError(f"model asset verification failed: {asset['file']}")
+
+# The checkpoint is Apache-2.0; retain its model card and the upstream runtime licence.
+for filename, url in {
+    "MODEL_CARD.md": f"https://huggingface.co/{manifest['repository']}/resolve/{manifest['revision']}/README.md",
+    "LICENSE": "https://raw.githubusercontent.com/QwenLM/Qwen3-TTS/022e286b98fbec7e1e916cb940cdf532cd9f488e/LICENSE",
+}.items():
+    with urlopen(url) as response:
+        (ROOT / filename).write_bytes(response.read())

docker/tts/requirements.in

@@ -0,0 +1,5 @@
+qwen-tts==0.1.1
+torch @ https://download-r2.pytorch.org/whl/cpu/torch-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl#sha256=ee40b8a4b4b2cf0670c6fd4f35a7ef23871af956fecb238fbf5da15a72650b1d ; platform_machine == "x86_64"
+torch @ https://download-r2.pytorch.org/whl/cpu/torch-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_aarch64.whl#sha256=179451716487f8cb09b56459667fa1f5c4c0946c1e75fbeae77cfc40a5768d87 ; platform_machine == "aarch64"
+torchaudio @ https://download-r2.pytorch.org/whl/cpu/torchaudio-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl#sha256=502b19a5bd0ef8fc97adf06c24146c5e2b94bc3c43b768174407c031cd21c3a5 ; platform_machine == "x86_64"
+torchaudio @ https://download-r2.pytorch.org/whl/cpu/torchaudio-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_aarch64.whl#sha256=2ef1f8d0f0feba5a9364620925c864890b90abc98a65f35895256dd6aa1710be ; platform_machine == "aarch64"

docker/tts/requirements.txt

@@ -0,0 +1,92 @@
+--index-url https://pypi.org/simple
+
+accelerate==1.12.0
+annotated-doc==0.0.5
+annotated-types==0.8.0
+anyio==4.16.0
+brotli==1.2.1
+certifi==2026.10.23
+cffi==2.1.1
+charset-normalizer==3.5.2
+click==8.5.1
+cloudpickle==3.1.2
+decorator==5.3.1
+einops==0.8.2
+fastapi==0.162.2
+filelock==4.2.10
+flatbuffers==25.12.19
+fsspec==2026.9.0
+gradio==6.17.3
+gradio-client==2.5.0
+groovy==0.1.2
+h11==0.16.0
+hf-gradio==0.4.1
+hf-xet==1.8.0
+httpcore==1.0.9
+httpx==0.28.1
+huggingface-hub==0.36.2
+idna==3.20
+jinja2==3.1.6
+joblib==1.6.0
+lazy-loader==0.6
+librosa==1.0.0
+llvmlite==0.50.0
+markdown-it-py==4.2.0
+markupsafe==3.0.4
+mdurl==0.1.2
+mpmath==1.3.0
+msgpack==1.2.4
+narwhals==2.28.0
+networkx==3.7
+numba==0.68.1
+numpy==2.5.4
+onnxruntime==1.31.1
+opentelemetry-api==1.45.2
+orjson==3.13.0
+packaging==26.3
+pandas==3.1.1
+pillow==12.3.0
+platformdirs==4.14.5
+pooch==1.9.0
+protobuf==7.36.2
+psutil==7.2.2
+pycparser==3.12
+pydantic==2.14.3
+pydantic-core==2.50.4
+pydub==0.25.1
+pygments==2.21.0
+python-dateutil==4.0.0
+python-multipart==0.0.33
+pytz==2026.6
+pyyaml==6.0.3
+qwen-tts==0.1.1
+regex==2026.11.9
+requests==2.34.2
+rich==15.0.0
+safehttpx==0.1.7
+safetensors==0.8.0
+scikit-learn==1.9.1
+scipy==1.18.1
+semantic-version==2.10.0
+setuptools==84.2.1
+shellingham==1.5.4
+soundfile==0.14.0
+sox==1.5.0
+soxr==1.1.0
+starlette==1.10.0
+sympy==1.14.0
+threadpoolctl==3.8.0
+tokenizers==0.22.2
+tomlkit==0.14.0
+torch @ https://download-r2.pytorch.org/whl/cpu/torch-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl#sha256=ee40b8a4b4b2cf0670c6fd4f35a7ef23871af956fecb238fbf5da15a72650b1d ; platform_machine == "x86_64"
+torch @ https://download-r2.pytorch.org/whl/cpu/torch-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_aarch64.whl#sha256=179451716487f8cb09b56459667fa1f5c4c0946c1e75fbeae77cfc40a5768d87 ; platform_machine == "aarch64"
+torchaudio @ https://download-r2.pytorch.org/whl/cpu/torchaudio-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl#sha256=502b19a5bd0ef8fc97adf06c24146c5e2b94bc3c43b768174407c031cd21c3a5 ; platform_machine == "x86_64"
+torchaudio @ https://download-r2.pytorch.org/whl/cpu/torchaudio-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_aarch64.whl#sha256=2ef1f8d0f0feba5a9364620925c864890b90abc98a65f35895256dd6aa1710be ; platform_machine == "aarch64"
+tqdm==4.70.1
+transformers==4.57.3
+typer==0.29.0
+typing-extensions==4.16.0
+typing-inspection==0.4.5
+tzdata==2026.6
+urllib3==2.8.0
+uvicorn==0.54.0

docker/tts/worker.py

@@ -0,0 +1,67 @@
+"""One model load, serial clips, fixed CPU generation; JSON stdin/stdout for the TS adapter."""
+import contextlib
+import json
+import random
+import resource
+import sys
+import time
+
+START = time.monotonic()
+MAX_TOKENS = 1024
+
+
+def synthesize(jobs):
+    # Upstream prints optional-dependency notices to stdout; keep our JSON channel clean.
+    with contextlib.redirect_stdout(sys.stderr):
+        import numpy as np
+        import soundfile as sf
+        import torch
+        from qwen_tts import Qwen3TTSModel
+
+        torch.set_num_threads(4)
+        torch.set_num_interop_threads(1)
+        torch.use_deterministic_algorithms(True)
+        model = Qwen3TTSModel.from_pretrained(
+            "/opt/flow-reel/tts-model", device_map="cpu", dtype=torch.float32,
+            attn_implementation="eager", local_files_only=True,
+        )
+        model.model.eval()
+        # The pinned official wrapper discards codec counts before returning WAVs.
+        # Check those counts at its generation boundary to reject token-limit truncation.
+        generate = model.model.generate
+
+        def checked_generate(**kwargs):
+            codes, hidden = generate(**kwargs)
+            if any(len(code) >= MAX_TOKENS - 1 for code in codes):
+                raise RuntimeError("speech reached its generation limit; shorten this step's narration")
+            return codes, hidden
+
+        model.model.generate = checked_generate
+        for job in jobs:
+            try:
+                random.seed(0)
+                np.random.seed(0)
+                torch.manual_seed(0)
+                wavs, rate = model.generate_custom_voice(
+                    text=job["text"], speaker="Ryan", language="English", non_streaming_mode=True,
+                    do_sample=True, top_k=50, top_p=1.0, temperature=0.9, repetition_penalty=1.05,
+                    subtalker_dosample=True, subtalker_top_k=50, subtalker_top_p=1.0,
+                    subtalker_temperature=0.9, max_new_tokens=MAX_TOKENS,
+                )
+                if not len(wavs[0]) or not np.isfinite(wavs[0]).all():
+                    raise RuntimeError("speech was empty or invalid")
+                sf.write(job["file"], wavs[0], rate, subtype="PCM_16")
+            except Exception as error:
+                raise RuntimeError(f"step {job['index']}: {error}") from error
+
+
+if __name__ == "__main__":
+    try:
+        jobs = json.load(sys.stdin)
+        synthesize(jobs)
+        print(json.dumps({"clips": len(jobs)}))
+        print(json.dumps({"ttsSeconds": round(time.monotonic() - START, 3),
+                          "peakMemoryKiB": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss}), file=sys.stderr)
+    except Exception as error:
+        print(f"Qwen TTS: {error}", file=sys.stderr)
+        sys.exit(1)

package.json

@@ -24,7 +24,7 @@
     "build": "tsc -p tsconfig.build.json",
     "typecheck": "tsc -p tsconfig.json --noEmit",
     "test": "vitest run test/unit",
-    "test:e2e": "npm run build && vitest run test/e2e --testTimeout=900000 --hookTimeout=900000"
+    "test:e2e": "npm run build && vitest run test/e2e --no-file-parallelism --testTimeout=900000 --hookTimeout=900000"
   },
   "dependencies": {
     "playwright": "1.64.1"

planning/GH-1-narrated-reproducible-feature-demos/decisions/004-pluggable-tts-local-default.md

@@ -1,5 +1,7 @@
 # 004: TTS sits behind an adapter, with a local open voice by default
 
+**Partly superseded by GH-4 decision 002** (the starting local model is Qwen3-TTS 0.6B instead of Kokoro/Piper).
+
 **Date:** 2026-11-02 (shaping, Q4; agreed: "yes!")
 
 **Question:** local TTS, cloud TTS, or pluggable?

planning/GH-1-narrated-reproducible-feature-demos/decisions/011-transcript-json-schema.md

@@ -1,5 +1,7 @@
 # 011: `transcript.json` has one versioned schema; each ticket fills its own fields
 
+**Partly superseded by GH-4 decision 001** (GH-4 writes `voice` and step `audio`; GH-5 still owns held `endMs`).
+
 **Date:** 2026-11-02 (global pass, Q9; agreed: "ok")
 
 **Question:** what does `transcript.json` hold, given four tickets read or write it?