feat: fit narration, mux audio and compare recordings (#5)

README.md

@@ -4,8 +4,8 @@ Record reproducible feature demos from a short Playwright script. A coding agent
 writes the script, and `flow-reel` replays it in a sandbox on a virtual display while recording the screen. The same
 script gives the same video, step for step.
 
-This release records **silent** video with narration text, captions, step timings and separate speech clips.
-Fitting and mixing the speech into the MP4, cursor and zoom polish, and sound effects come in later releases.
+This release records narrated video with captions, fitted step timings and original speech clips.
+Cursor and zoom polish, and sound effects come in later releases.
 
 ## Requirements
 
@@ -59,7 +59,7 @@ Next to the script:
 
 | File | What it is |
 |---|---|
-| `settings.mp4` | The silent demo, 1920×1080, 30 fps. |
+| `settings.mp4` | The narrated demo, 1920×1080, 30 fps, with AAC speech audio. |
 | `settings.transcript.json` | Each step's original narration and `startMs`, `actionEndMs` and `endMs` on the video's timeline (schema version 1). |
 | `settings.events.json` | Every move, click, keystroke run and scroll the runner made, timed the same way (schema version 1). |
 | `settings.srt` | Captions for a subtitle-capable video player. |
@@ -68,8 +68,8 @@ Next to the script:
 
 Each caption shows one step's narration from its `startMs` to its `endMs`. Caption whitespace is normalized and the
 player wraps the text; the transcript keeps the original narration. Blank narration and steps lasting zero
-milliseconds stay in the transcript but produce no caption. Short actions can be too brief to read: step holds for
-voice clips come in a later release.
+milliseconds stay in the transcript but produce no caption. Narrated steps include the full speech clip and
+at least 200 ms afterward.
 
 Narration uses Qwen3-TTS 0.6B CustomVoice, Ryan in English, through the official Python runtime on CPU. Model and
 tokenizer assets are pinned and downloaded when the image is built; synthesis runs offline without a key. Speech
@@ -79,8 +79,20 @@ duration in milliseconds and complete-file SHA-256 hash. Copy the audio folder w
 Synthesis normalizes whitespace without interpreting caption escapes; the transcript preserves the original text.
 Blank narration produces no clip. Clip numbers retain step indexes, so skipped steps leave gaps. Rerunning replaces
 the generated numeric WAVs in this demo's audio folder; keep personal audio under other filenames or folders.
-Nonblank narration on a zero-duration action still produces speech. Clips can outlast their action windows until
-the next release adds holds and muxing; the MP4 has no audio track yet. A synthesis failure fails the command.
+Nonblank narration on a zero-duration action still produces speech. Each action runs once at normal speed.
+After its callback returns (`actionEndMs`), the recorder waits for network quiet (500 ms, bounded at 10 seconds),
+then captures another 100 ms. Scripts must still await the UI state they intend to show; network quiet does not
+prove that an operation succeeded. A timeout prints a warning and continues.
+
+After capture and synthesis finish, the renderer adds only the whole frames needed to fit speech plus its tail,
+freezing the last frame of that settled window. Narration starts alongside its action at `startMs`. Longer actions
+continue normally after speech finishes. Transcript windows, captions, events and the master all use this final
+timeline. Blank narration gets no speech padding; an all-blank demo has no audio stream.
+
+The assembled voice is normalized to -16 LUFS with a -1.5 dBTP peak ceiling, then encoded as 48 kHz stereo AAC-LC
+at 192 kb/s. Decoded output is checked within 1 LU of the loudness target and 0.1 dB of the peak ceiling. Original
+WAVs, durations and hashes remain unchanged. Synthesis, rendering or sync failures fail the command before a new
+success transcript is written.
 
 The local generator resets seed 0 for each clip and uses pinned settings. Matching hashes are checked in fresh
 processes on the tested CPU/container platform; byte equality across hardware or architectures is not guaranteed.
@@ -91,13 +103,32 @@ individual steps also fail if generation reaches its limit, rather than returnin
 Captions escape `&`, `<` and `>` so narration is emitted without raw tags. WebVTT displays the literal text; some
 SRT players display escape sequences such as `&lt;` instead. Use WebVTT in a browser when literal symbols matter.
 
-In `.flow-reel/settings/`: the 2560×1440 master the polish steps will render from. The `.flow-reel/` folder ignores
-itself in git; the files next to the script are yours to commit or not.
+In `.flow-reel/settings/`: `capture.mp4` is the original 2560×1440 recording; `master.mp4` has the added holds for
+later polish; `master.json.startMs` gives delivery time zero within the master, preserving its calibration pre-roll.
+`voice.wav` is the normalized voice bus on the delivery timeline. The `.flow-reel/` folder ignores itself in git;
+the files next to the script are yours to commit or not.
 
 To keep runs the same, the runner fixes the viewport, device scale, `prefers-reduced-motion`, the page's clock
 (`2026-01-05T09:00:00Z`) and `Math.random`'s seed, and waits for the network to go idle before each step. Two runs of
 a script have the same steps, each starting within ±150 ms.
 
+## Compare recordings
+
+```sh
+npx flow-reel compare demos/a.transcript.json demos/b.transcript.json
+```
+
+Comparison runs on the host without Docker or FFmpeg. It validates each transcript's schema and sync, then checks
+video dimensions/FPS, step order, exact original narration, voice identity, source audio hashes/durations and step
+starts within ±150 ms inclusive. Script and output paths may differ. Legacy silent transcripts and blank-only
+recordings are supported. Exit status is 0 for a match, 1 for valid recordings that differ, and 2 for invalid input,
+failed sync or usage errors; diagnostics identify the file or step.
+
+These checks read transcript data only. They do not open referenced audio, caption or video files, prove visual/UI
+equality, or promise identical encoded MP4 bytes. Valid action/end/video durations may differ between recordings;
+only step starts have the 150 ms comparison tolerance. Recording automatically checks that all clips fit their
+step windows, including the speech tail, before publishing its transcript.
+
 ## Platforms
 
 - **Linux** (Docker or Podman): the container uses the host's network, so `localhost` is your machine. Tested.
@@ -106,7 +137,6 @@ a script have the same steps, each starting within ±150 ms.
 
 ## Known gaps
 
-- Speech clips are separate; the MP4 remains silent until step holds and audio muxing are added.
 - No cursor drawn in the video yet: the runner logs the motion in `events.json` for the polish step.
 - Playwright calls the runner doesn't cover run without any mouse motion (the warning names them).
 

planning/GH-1-narrated-reproducible-feature-demos/decisions/008-typescript-cli-on-npm.md

@@ -1,5 +1,7 @@
 # 008: The CLI is TypeScript on npm; on the host it only launches the image
 
+**Partly superseded by GH-5 decision 006** (the read-only `compare` command runs on the host; recording stays in the image).
+
 **Date:** 2026-11-02 (global pass, Q6; agreed: "ok")
 
 **Question:** which language is the CLI written in, and how does it reach the container?

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/build-notes.md

@@ -0,0 +1,87 @@
+# GH-5 build notes
+
+Base: main at 76a31531ae34cec8ae238c1188016423579ee025. Branch: feat/5-voice-fit-mux-sync-check.
+
+## Implemented
+
+Single capture with bounded post-action network quiet plus 100 ms; pure integer-frame hold planning and one
+final event mapping; assembled/normalized delivery voice; retimed master preserving calibration pre-roll;
+shared semantic sync checks; host-only transcript comparison with 0/1/2 exit statuses; README contract.
+The original Qwen settings, public WAVs/hashes, overload guard and unrelated follow-ups are unchanged.
+
+## Verification so far
+
+- `npm test`: 63 tests pass; `npm run typecheck` and `npm run build` pass.
+- Focused real-browser settling fixture: old code failed with 0 ms between callback return and visual endpoint;
+  updated code captures the delayed response and passes. Initial overloaded-host run failed the existing guard
+  (24% shortfall); user freed CPU and it reached the intended assertion. Guard/tolerances were retained.
+- Synthetic media tests: exact source/master/delivery frame counts, red-frame holds before a blue transition,
+  final hold, pre-roll preservation, blank-only output, stream codec/rate/channels, raw WAV hashes unchanged,
+  decoded clip starts/ends within one frame, delivered loudness/peak measurement, and invalid-media/silence errors.
+  All seven checks, including review regressions and combined holds/audio placement, pass on both host FFmpeg
+  and the recording image's FFmpeg.
+- Built host comparison runs with PATH containing no external executables and FLOW_REEL_IN_CONTAINER=1: equal,
+  different, invalid JSON, failed sync, missing file and usage cases pass (0/1/2 statuses).
+- `npm pack --dry-run --json`: includes dist/compare.js, dist/schema/sync.js, dist/runner/timeline.js and
+  dist/runner/render.js. The installed npm outputs a package-name-keyed object; its files were checked directly.
+- Corrected real-speech preview (`test/e2e/out/preview/demo.mp4`): video and 48 kHz stereo AAC both 12.500 s;
+  decoded -16.40 LUFS / -2.00 dBTP. Rendered from the first run's held master and unchanged source WAVs without
+  another action replay or TTS call. Person approved playback: "Playback sounds good" (all three lines, screen holds and pauses).
+- Offline TTS fresh processes: 140.835 s / 5,677,132 KiB and 120.254 s / 5,789,436 KiB. First recording synthesis:
+  181.003 s / 5,791,216 KiB. Full serial Docker suite and final fixed-image recording pair remain pending.
+
+## Review and corrections
+
+Standards review found an orphaned validation branch after validation moved before publication; replaced it with
+writeJson. Both reviewers found stale voice intermediates on blank reruns; renderer removes only its owned
+voice.wav and voice.wav.bus.wav when clips are absent. Regression fails before the change and passes afterward.
+
+Spec review found that loudnorm can switch to dynamic normalization at 192 kHz. Sample-based padding/trimming now
+explicitly resamples to 48 kHz first. A regression with extreme transients verifies the full 5.5-second bus and
+its late clip survive. Its delivered level is about -20 LUFS, so the command correctly rejects it under the
+unchanged loudness target; this fixture tests failure and duration retention, not a relaxed normalization target.
+
+Frame-derived times can contain fractional milliseconds; caption serialization rounds them to valid whole-ms
+SRT/WebVTT timestamps, covered across a second boundary. No design decisions were changed.
+
+The first full-suite image was built before review corrections. Its first real demo exposed the dynamic-mode
+bug directly: 12.500 s video but only 3.125 s audio, despite valid transcript durations and acceptable loudness.
+The renderer now also checks normalized-bus duration to one sample and AAC stream duration to one frame against
+the planned timeline. The recording E2E asserts actual audio duration, in addition to schema and fit checks.
+The final full suite is running with rebuilt fixed image 44fb4f54671da97b433f457bab9ad1cdf24299d982351803a89077223bf03c98; early-image success is not final evidence.
+
+Final-source checks: `npm test` passes all 63 tests (12 files, 1.82 s); `npm run typecheck` and build pass.
+Final `npm pack --dry-run --json`: 48 files, 28,265 bytes, all new compiled modules included.
+First full media pass: 19 tests passed in 821.81 s. A fresh final full media pass is running after the normalization
+fix; the earlier run's audio-duration defect is documented above rather than counted as final delivery evidence.
+Review rechecks: Standards 0 unresolved; Spec 0 unresolved (2 corrected defects). Human playback approved.
+
+Final full-suite run with the fixed image: 12 passed / 7 skipped in 991.33 s. The shared recording hook failed
+on **recording b's capture overload guard**: 6,933 ms video for 12,014 ms wall time (42% shortfall). Host load
+reached about 29 on eight CPUs. The first recording succeeded; the initial suspicion of a render failure was
+corrected once the suite printed its shared-hook error. No source changes were made for this resource failure.
+The image's renderer was independently exercised against the completed master/WAVs and passed at -16.43 LUFS /
+-1.99 dBTP. The normalized bus and AAC have full planned duration. Asked the person to free CPU before rerunning
+the recording pair; the full suite's offline TTS, renderer and CLI checks already passed on the fixed source.
+
+## Final result
+
+After the person freed CPU again, the unchanged recording file was rerun:
+`node_modules/.bin/vitest run test/e2e/record.test.ts --hookTimeout=900000 --testTimeout=900000`:
+**9 passed in 488.01 s**. Together with the fixed-image full run's other passing files (offline TTS 2, renderer 7,
+CLI 1), **all 19 unique media checks pass**. The earlier full command's resource failure remains documented above;
+it was not hidden or fixed by weakening a guard. No source/test changes occurred between that full run and retry.
+
+- Both final demos have full AAC duration, fitted steps, unchanged source WAV hashes and valid native captions.
+- Host `node dist/cli.js compare .../a/demo.transcript.json .../b/demo.transcript.json`: exit 0, transcripts match.
+- Step start differences: 5 ms, 40.333 ms, 9 ms (maximum 40.333 ms < 150 ms).
+- Clip tails, run a: 203.667 / 208.333 / 242.667 ms; run b: 232 / 201.333 / 233.667 ms (all >= 200 ms).
+- First final demo: -16.42 LUFS / -1.98 dBTP. Magenta click discrepancy:
+  -12.493 ms (limit ±66.667 ms / two frames). Retimed master and delivery frame checks pass.
+- Final TTS synthesis: a 198.801 s / 5,783,284 KiB; b 180.446 s / 5,694,716 KiB peak process memory.
+- Person approved the corrected narrated MP4: "Playback sounds good". Same voice/source clips in final pair.
+- Final unit/type/build/package/whitespace checks and both review axes passed as recorded above.
+
+No implementation steps left undone. Small implementation notes (fractional caption timestamps, explicit AAC peak
+margin, normalization resampling/duration guards and clearing stale buses) retain the approved design. The
+odd-viewport encoder and optional 4K follow-ups remain open and untouched. Ready for commit approval.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/001-single-replay-postprocessing-holds.md

@@ -0,0 +1,22 @@
+# 001: Add voice-fit holds after a single replay
+
+**Date:** 2026-11-11 (ticket pass, Q1; agreed: "ok")
+
+**Question:** how can the runner fit speech when narration becomes known only as the script calls `step()`?
+
+**Decision:** **capture the script once, synthesize narration after capture, then extend the recorded video with
+frame holds where speech needs more time.** Keep the existing imperative script API and execute each action once.
+The screen is frozen during added holds. Shift transcript times, captions and events onto the final video timeline
+together; the master consumed by later polish must agree with that timeline too.
+
+This replaces the current GH-5 draft's proposal to synthesize all narration before recording. Voice/audio metadata
+and the version 1 schema remain as shipped by GH-4. CPU synthesis stays outside real-time capture.
+
+- **Why:** narration is discovered at runtime, and Qwen synthesis can take minutes. Postprocessing keeps capture
+  free of that work and avoids another execution of application actions.
+- **Not chosen:** a preparation replay, which repeats side effects and needs application resets; pausing capture
+  to synthesize each step, which lets application state change during long unrecorded gaps.
+- **To settle next:** narration placement, hold boundaries/padding, audio mix, and sync/comparison checks.
+
+Feasibility reference: [FFmpeg tpad source](https://ffmpeg.org/doxygen/trunk/vf__tpad_8c.html) supports cloning
+frames for holds. This decision does not prescribe the final filter graph.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/002-narration-starts-with-action.md

@@ -0,0 +1,14 @@
+# 002: Start narration with its action
+
+**Date:** 2026-11-11 (ticket pass, Q2; agreed: "ok")
+
+**Question:** does narration begin at the step's start or after the action finishes?
+
+**Decision:** **start each clip at the step's final `startMs`, alongside the action.** Preserve the action's
+normal playback speed. Add a frame hold when speech needs more time than the captured action window. When the
+action lasts longer than speech, it continues after the clip ends. Padding and exact frame boundaries are settled
+separately.
+
+- **Why:** a natural walkthrough pace with the explanation accompanying its action.
+- **Not chosen:** completing the action first and then playing its entire narration over a held screen, which
+  lengthens every narrated step.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/003-capture-settled-result-before-hold.md

@@ -0,0 +1,17 @@
+# 003: Capture a settling window before choosing the held frame
+
+**Date:** 2026-11-11 (ticket pass, Q3; agreed: "ok")
+
+**Question:** freeze the frame at action callback completion, or give the resulting UI time to settle?
+
+**Decision:** **after the action returns, use the existing bounded network-idle check, then capture another
+100 ms before ending the step's visual window.** The idle check requires 500 ms of quiet and waits at most
+10 seconds. Warn and continue if the page remains busy, matching the existing before-step timeout behavior.
+An added postprocessing hold uses the final frame of this captured visual window.
+
+Keep `actionEndMs` as the callback's actual finish time. The settling window and any added hold contribute to
+the step's final `endMs`; they do not redefine when the action completed. Scripts must still explicitly await
+the UI state they intend to demonstrate; network quiet is not proof that a business operation succeeded.
+
+- **Why:** avoid freezing a loading indicator or unfinished transition when the callback returns promptly.
+- **Not chosen:** freezing immediately at callback return, which is quicker but can hold an incomplete view.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/004-speech-tail-padding.md

@@ -0,0 +1,14 @@
+# 004: Leave at least 200 ms after speech
+
+**Date:** 2026-11-11 (ticket pass, Q4; agreed: "ok")
+
+**Question:** fit speech exactly into its step, or leave a short pause before the next step?
+
+**Decision:** **each narrated step lasts at least through its clip plus 200 ms**, measured from its final
+`startMs`. Existing captured time counts toward this minimum; insert only the extra hold needed. The step must
+also include its captured action and settling window. Round added holds upward to whole video frames, so speech
+and its minimum tail are never truncated by rounding. Blank narration gets no speech-tail padding.
+
+- **Why:** give successive narration lines breathing room while retaining longer captured actions.
+- **Not chosen:** fitting clips exactly to step boundaries with no extra pause, which is more compact but can
+  sound rushed.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/005-normalize-assembled-voice-track.md

@@ -0,0 +1,18 @@
+# 005: Normalize the assembled voice track and preserve source clips
+
+**Date:** 2026-11-11 (ticket pass, Q5; agreed: "ok")
+
+**Question:** normalize delivered narration volume or retain Qwen's original output level?
+
+**Decision:** **normalize the assembled voice track to a target of -16 LUFS with a -1.5 dBTP peak ceiling.**
+Keep original generated WAV clips unchanged, including their recorded durations and complete-file SHA-256 hashes.
+Normalization belongs to the assembled delivery audio, after its placement on the final timeline. Source clip
+hashes remain the basis for the inherited same-audio comparison.
+
+- **Why:** a consistent delivery level across demos while keeping the reproducible source clips available.
+- **Not chosen:** using the original Qwen volume for delivery, which can vary between demos.
+- **Verification:** measure the rendered audio rather than assuming filter options prove the result. Exact
+  encoding settings, measurement tolerances and treatment of unmeasurable input belong in the implementation plan.
+
+Source: [FFmpeg loudnorm documentation](https://ffmpeg.org/ffmpeg-filters.html#loudnorm), checked during the
+ticket pass; supports measured two-pass loudness normalization. The numerical targets are this product's choice.