From 23c3b420d0e47c41e2e2200ecfa20c32de31b680 Mon Sep 17 00:00:00 2001 From: codeonym Date: Wed, 11 Nov 2026 11:13:28 +0000 Subject: [PATCH 1/2] feat: fit narration, mux audio and compare recordings (#5) --- README.md | 50 ++++-- .../decisions/008-typescript-cli-on-npm.md | 2 + .../build-notes.md | 87 ++++++++++ .../001-single-replay-postprocessing-holds.md | 22 +++ .../002-narration-starts-with-action.md | 14 ++ .../003-capture-settled-result-before-hold.md | 17 ++ .../decisions/004-speech-tail-padding.md | 14 ++ .../005-normalize-assembled-voice-track.md | 18 ++ .../decisions/006-host-compare-command.md | 18 ++ .../GH-5-voice-fit-mux-sync-check/draft.md | 55 ++++-- .../history/2026-11-11_0842_draft.md | 28 ++++ .../history/2026-11-11_0842_meta.md | 12 ++ .../history/2026-11-11_0925_meta.md | 12 ++ .../history/2026-11-11_1018_impl.md | 148 +++++++++++++++++ .../history/CHANGES.md | 12 ++ .../GH-5-voice-fit-mux-sync-check/impl.md | 157 ++++++++++++++++++ .../GH-5-voice-fit-mux-sync-check/meta.md | 6 +- src/cli.ts | 23 ++- src/compare.ts | 29 ++++ src/runner/captions.ts | 1 + src/runner/recorder.ts | 10 -- src/runner/render.ts | 116 +++++++++++++ src/runner/run.ts | 43 +++-- src/runner/timeline.ts | 46 +++++ src/schema/sync.ts | 29 ++++ test/e2e/compare.test.ts | 25 +++ test/e2e/record.test.ts | 50 +++++- test/e2e/render.test.ts | 154 +++++++++++++++++ test/unit/captions.test.ts | 6 + test/unit/compare.test.ts | 40 +++++ test/unit/sync.test.ts | 45 +++++ test/unit/timeline.test.ts | 56 +++++++ 32 files changed, 1285 insertions(+), 60 deletions(-) create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/build-notes.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/001-single-replay-postprocessing-holds.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/002-narration-starts-with-action.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/003-capture-settled-result-before-hold.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/004-speech-tail-padding.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/005-normalize-assembled-voice-track.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/006-host-compare-command.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0842_draft.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0842_meta.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0925_meta.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_1018_impl.md create mode 100644 planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/impl.md create mode 100644 src/compare.ts create mode 100644 src/runner/render.ts create mode 100644 src/runner/timeline.ts create mode 100644 src/schema/sync.ts create mode 100644 test/e2e/compare.test.ts create mode 100644 test/e2e/render.test.ts create mode 100644 test/unit/compare.test.ts create mode 100644 test/unit/sync.test.ts create mode 100644 test/unit/timeline.test.ts diff --git a/README.md b/README.md index 116cefc..3a3b1ee 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,8 @@ Record reproducible feature demos from a short Playwright script. A coding agent writes the script, and `flow-reel` replays it in a sandbox on a virtual display while recording the screen. The same script gives the same video, step for step. -This release records **silent** video with narration text, captions, step timings and separate speech clips. -Fitting and mixing the speech into the MP4, cursor and zoom polish, and sound effects come in later releases. +This release records narrated video with captions, fitted step timings and original speech clips. +Cursor and zoom polish, and sound effects come in later releases. ## Requirements @@ -59,7 +59,7 @@ Next to the script: | File | What it is | |---|---| -| `settings.mp4` | The silent demo, 1920×1080, 30 fps. | +| `settings.mp4` | The narrated demo, 1920×1080, 30 fps, with AAC speech audio. | | `settings.transcript.json` | Each step's original narration and `startMs`, `actionEndMs` and `endMs` on the video's timeline (schema version 1). | | `settings.events.json` | Every move, click, keystroke run and scroll the runner made, timed the same way (schema version 1). | | `settings.srt` | Captions for a subtitle-capable video player. | @@ -68,8 +68,8 @@ Next to the script: Each caption shows one step's narration from its `startMs` to its `endMs`. Caption whitespace is normalized and the player wraps the text; the transcript keeps the original narration. Blank narration and steps lasting zero -milliseconds stay in the transcript but produce no caption. Short actions can be too brief to read: step holds for -voice clips come in a later release. +milliseconds stay in the transcript but produce no caption. Narrated steps include the full speech clip and +at least 200 ms afterward. Narration uses Qwen3-TTS 0.6B CustomVoice, Ryan in English, through the official Python runtime on CPU. Model and tokenizer assets are pinned and downloaded when the image is built; synthesis runs offline without a key. Speech @@ -79,8 +79,20 @@ duration in milliseconds and complete-file SHA-256 hash. Copy the audio folder w Synthesis normalizes whitespace without interpreting caption escapes; the transcript preserves the original text. Blank narration produces no clip. Clip numbers retain step indexes, so skipped steps leave gaps. Rerunning replaces the generated numeric WAVs in this demo's audio folder; keep personal audio under other filenames or folders. -Nonblank narration on a zero-duration action still produces speech. Clips can outlast their action windows until -the next release adds holds and muxing; the MP4 has no audio track yet. A synthesis failure fails the command. +Nonblank narration on a zero-duration action still produces speech. Each action runs once at normal speed. +After its callback returns (`actionEndMs`), the recorder waits for network quiet (500 ms, bounded at 10 seconds), +then captures another 100 ms. Scripts must still await the UI state they intend to show; network quiet does not +prove that an operation succeeded. A timeout prints a warning and continues. + +After capture and synthesis finish, the renderer adds only the whole frames needed to fit speech plus its tail, +freezing the last frame of that settled window. Narration starts alongside its action at `startMs`. Longer actions +continue normally after speech finishes. Transcript windows, captions, events and the master all use this final +timeline. Blank narration gets no speech padding; an all-blank demo has no audio stream. + +The assembled voice is normalized to -16 LUFS with a -1.5 dBTP peak ceiling, then encoded as 48 kHz stereo AAC-LC +at 192 kb/s. Decoded output is checked within 1 LU of the loudness target and 0.1 dB of the peak ceiling. Original +WAVs, durations and hashes remain unchanged. Synthesis, rendering or sync failures fail the command before a new +success transcript is written. The local generator resets seed 0 for each clip and uses pinned settings. Matching hashes are checked in fresh processes on the tested CPU/container platform; byte equality across hardware or architectures is not guaranteed. @@ -91,13 +103,32 @@ individual steps also fail if generation reaches its limit, rather than returnin Captions escape `&`, `<` and `>` so narration is emitted without raw tags. WebVTT displays the literal text; some SRT players display escape sequences such as `<` instead. Use WebVTT in a browser when literal symbols matter. -In `.flow-reel/settings/`: the 2560×1440 master the polish steps will render from. The `.flow-reel/` folder ignores -itself in git; the files next to the script are yours to commit or not. +In `.flow-reel/settings/`: `capture.mp4` is the original 2560×1440 recording; `master.mp4` has the added holds for +later polish; `master.json.startMs` gives delivery time zero within the master, preserving its calibration pre-roll. +`voice.wav` is the normalized voice bus on the delivery timeline. The `.flow-reel/` folder ignores itself in git; +the files next to the script are yours to commit or not. To keep runs the same, the runner fixes the viewport, device scale, `prefers-reduced-motion`, the page's clock (`2026-01-05T09:00:00Z`) and `Math.random`'s seed, and waits for the network to go idle before each step. Two runs of a script have the same steps, each starting within ±150 ms. +## Compare recordings + +```sh +npx flow-reel compare demos/a.transcript.json demos/b.transcript.json +``` + +Comparison runs on the host without Docker or FFmpeg. It validates each transcript's schema and sync, then checks +video dimensions/FPS, step order, exact original narration, voice identity, source audio hashes/durations and step +starts within ±150 ms inclusive. Script and output paths may differ. Legacy silent transcripts and blank-only +recordings are supported. Exit status is 0 for a match, 1 for valid recordings that differ, and 2 for invalid input, +failed sync or usage errors; diagnostics identify the file or step. + +These checks read transcript data only. They do not open referenced audio, caption or video files, prove visual/UI +equality, or promise identical encoded MP4 bytes. Valid action/end/video durations may differ between recordings; +only step starts have the 150 ms comparison tolerance. Recording automatically checks that all clips fit their +step windows, including the speech tail, before publishing its transcript. + ## Platforms - **Linux** (Docker or Podman): the container uses the host's network, so `localhost` is your machine. Tested. @@ -106,7 +137,6 @@ a script have the same steps, each starting within ±150 ms. ## Known gaps -- Speech clips are separate; the MP4 remains silent until step holds and audio muxing are added. - No cursor drawn in the video yet: the runner logs the motion in `events.json` for the polish step. - Playwright calls the runner doesn't cover run without any mouse motion (the warning names them). diff --git a/planning/GH-1-narrated-reproducible-feature-demos/decisions/008-typescript-cli-on-npm.md b/planning/GH-1-narrated-reproducible-feature-demos/decisions/008-typescript-cli-on-npm.md index 24b0afc..934c42b 100644 --- a/planning/GH-1-narrated-reproducible-feature-demos/decisions/008-typescript-cli-on-npm.md +++ b/planning/GH-1-narrated-reproducible-feature-demos/decisions/008-typescript-cli-on-npm.md @@ -1,5 +1,7 @@ # 008: The CLI is TypeScript on npm; on the host it only launches the image +**Partly superseded by GH-5 decision 006** (the read-only `compare` command runs on the host; recording stays in the image). + **Date:** 2026-11-02 (global pass, Q6; agreed: "ok") **Question:** which language is the CLI written in, and how does it reach the container? diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/build-notes.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/build-notes.md new file mode 100644 index 0000000..ca8e4df --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/build-notes.md @@ -0,0 +1,87 @@ +# GH-5 build notes + +Base: main at 76a31531ae34cec8ae238c1188016423579ee025. Branch: feat/5-voice-fit-mux-sync-check. + +## Implemented + +Single capture with bounded post-action network quiet plus 100 ms; pure integer-frame hold planning and one +final event mapping; assembled/normalized delivery voice; retimed master preserving calibration pre-roll; +shared semantic sync checks; host-only transcript comparison with 0/1/2 exit statuses; README contract. +The original Qwen settings, public WAVs/hashes, overload guard and unrelated follow-ups are unchanged. + +## Verification so far + +- `npm test`: 63 tests pass; `npm run typecheck` and `npm run build` pass. +- Focused real-browser settling fixture: old code failed with 0 ms between callback return and visual endpoint; + updated code captures the delayed response and passes. Initial overloaded-host run failed the existing guard + (24% shortfall); user freed CPU and it reached the intended assertion. Guard/tolerances were retained. +- Synthetic media tests: exact source/master/delivery frame counts, red-frame holds before a blue transition, + final hold, pre-roll preservation, blank-only output, stream codec/rate/channels, raw WAV hashes unchanged, + decoded clip starts/ends within one frame, delivered loudness/peak measurement, and invalid-media/silence errors. + All seven checks, including review regressions and combined holds/audio placement, pass on both host FFmpeg + and the recording image's FFmpeg. +- Built host comparison runs with PATH containing no external executables and FLOW_REEL_IN_CONTAINER=1: equal, + different, invalid JSON, failed sync, missing file and usage cases pass (0/1/2 statuses). +- `npm pack --dry-run --json`: includes dist/compare.js, dist/schema/sync.js, dist/runner/timeline.js and + dist/runner/render.js. The installed npm outputs a package-name-keyed object; its files were checked directly. +- Corrected real-speech preview (`test/e2e/out/preview/demo.mp4`): video and 48 kHz stereo AAC both 12.500 s; + decoded -16.40 LUFS / -2.00 dBTP. Rendered from the first run's held master and unchanged source WAVs without + another action replay or TTS call. Person approved playback: "Playback sounds good" (all three lines, screen holds and pauses). +- Offline TTS fresh processes: 140.835 s / 5,677,132 KiB and 120.254 s / 5,789,436 KiB. First recording synthesis: + 181.003 s / 5,791,216 KiB. Full serial Docker suite and final fixed-image recording pair remain pending. + +## Review and corrections + +Standards review found an orphaned validation branch after validation moved before publication; replaced it with +writeJson. Both reviewers found stale voice intermediates on blank reruns; renderer removes only its owned +voice.wav and voice.wav.bus.wav when clips are absent. Regression fails before the change and passes afterward. + +Spec review found that loudnorm can switch to dynamic normalization at 192 kHz. Sample-based padding/trimming now +explicitly resamples to 48 kHz first. A regression with extreme transients verifies the full 5.5-second bus and +its late clip survive. Its delivered level is about -20 LUFS, so the command correctly rejects it under the +unchanged loudness target; this fixture tests failure and duration retention, not a relaxed normalization target. + +Frame-derived times can contain fractional milliseconds; caption serialization rounds them to valid whole-ms +SRT/WebVTT timestamps, covered across a second boundary. No design decisions were changed. + +The first full-suite image was built before review corrections. Its first real demo exposed the dynamic-mode +bug directly: 12.500 s video but only 3.125 s audio, despite valid transcript durations and acceptable loudness. +The renderer now also checks normalized-bus duration to one sample and AAC stream duration to one frame against +the planned timeline. The recording E2E asserts actual audio duration, in addition to schema and fit checks. +The final full suite is running with rebuilt fixed image 44fb4f54671da97b433f457bab9ad1cdf24299d982351803a89077223bf03c98; early-image success is not final evidence. + +Final-source checks: `npm test` passes all 63 tests (12 files, 1.82 s); `npm run typecheck` and build pass. +Final `npm pack --dry-run --json`: 48 files, 28,265 bytes, all new compiled modules included. +First full media pass: 19 tests passed in 821.81 s. A fresh final full media pass is running after the normalization +fix; the earlier run's audio-duration defect is documented above rather than counted as final delivery evidence. +Review rechecks: Standards 0 unresolved; Spec 0 unresolved (2 corrected defects). Human playback approved. + +Final full-suite run with the fixed image: 12 passed / 7 skipped in 991.33 s. The shared recording hook failed +on **recording b's capture overload guard**: 6,933 ms video for 12,014 ms wall time (42% shortfall). Host load +reached about 29 on eight CPUs. The first recording succeeded; the initial suspicion of a render failure was +corrected once the suite printed its shared-hook error. No source changes were made for this resource failure. +The image's renderer was independently exercised against the completed master/WAVs and passed at -16.43 LUFS / +-1.99 dBTP. The normalized bus and AAC have full planned duration. Asked the person to free CPU before rerunning +the recording pair; the full suite's offline TTS, renderer and CLI checks already passed on the fixed source. + +## Final result + +After the person freed CPU again, the unchanged recording file was rerun: +`node_modules/.bin/vitest run test/e2e/record.test.ts --hookTimeout=900000 --testTimeout=900000`: +**9 passed in 488.01 s**. Together with the fixed-image full run's other passing files (offline TTS 2, renderer 7, +CLI 1), **all 19 unique media checks pass**. The earlier full command's resource failure remains documented above; +it was not hidden or fixed by weakening a guard. No source/test changes occurred between that full run and retry. + +- Both final demos have full AAC duration, fitted steps, unchanged source WAV hashes and valid native captions. +- Host `node dist/cli.js compare .../a/demo.transcript.json .../b/demo.transcript.json`: exit 0, transcripts match. +- Step start differences: 5 ms, 40.333 ms, 9 ms (maximum 40.333 ms < 150 ms). +- Clip tails, run a: 203.667 / 208.333 / 242.667 ms; run b: 232 / 201.333 / 233.667 ms (all >= 200 ms). +- First final demo: -16.42 LUFS / -1.98 dBTP. Magenta click discrepancy: + -12.493 ms (limit ±66.667 ms / two frames). Retimed master and delivery frame checks pass. +- Final TTS synthesis: a 198.801 s / 5,783,284 KiB; b 180.446 s / 5,694,716 KiB peak process memory. +- Person approved the corrected narrated MP4: "Playback sounds good". Same voice/source clips in final pair. +- Final unit/type/build/package/whitespace checks and both review axes passed as recorded above. + +No implementation steps left undone. Small implementation notes (fractional caption timestamps, explicit AAC peak +margin, normalization resampling/duration guards and clearing stale buses) retain the approved design. The +odd-viewport encoder and optional 4K follow-ups remain open and untouched. Ready for commit approval. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/001-single-replay-postprocessing-holds.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/001-single-replay-postprocessing-holds.md new file mode 100644 index 0000000..ea28c14 --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/001-single-replay-postprocessing-holds.md @@ -0,0 +1,22 @@ +# 001: Add voice-fit holds after a single replay + +**Date:** 2026-11-11 (ticket pass, Q1; agreed: "ok") + +**Question:** how can the runner fit speech when narration becomes known only as the script calls `step()`? + +**Decision:** **capture the script once, synthesize narration after capture, then extend the recorded video with +frame holds where speech needs more time.** Keep the existing imperative script API and execute each action once. +The screen is frozen during added holds. Shift transcript times, captions and events onto the final video timeline +together; the master consumed by later polish must agree with that timeline too. + +This replaces the current GH-5 draft's proposal to synthesize all narration before recording. Voice/audio metadata +and the version 1 schema remain as shipped by GH-4. CPU synthesis stays outside real-time capture. + +- **Why:** narration is discovered at runtime, and Qwen synthesis can take minutes. Postprocessing keeps capture + free of that work and avoids another execution of application actions. +- **Not chosen:** a preparation replay, which repeats side effects and needs application resets; pausing capture + to synthesize each step, which lets application state change during long unrecorded gaps. +- **To settle next:** narration placement, hold boundaries/padding, audio mix, and sync/comparison checks. + +Feasibility reference: [FFmpeg tpad source](https://ffmpeg.org/doxygen/trunk/vf__tpad_8c.html) supports cloning +frames for holds. This decision does not prescribe the final filter graph. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/002-narration-starts-with-action.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/002-narration-starts-with-action.md new file mode 100644 index 0000000..38c9a47 --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/002-narration-starts-with-action.md @@ -0,0 +1,14 @@ +# 002: Start narration with its action + +**Date:** 2026-11-11 (ticket pass, Q2; agreed: "ok") + +**Question:** does narration begin at the step's start or after the action finishes? + +**Decision:** **start each clip at the step's final `startMs`, alongside the action.** Preserve the action's +normal playback speed. Add a frame hold when speech needs more time than the captured action window. When the +action lasts longer than speech, it continues after the clip ends. Padding and exact frame boundaries are settled +separately. + +- **Why:** a natural walkthrough pace with the explanation accompanying its action. +- **Not chosen:** completing the action first and then playing its entire narration over a held screen, which + lengthens every narrated step. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/003-capture-settled-result-before-hold.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/003-capture-settled-result-before-hold.md new file mode 100644 index 0000000..fbebad9 --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/003-capture-settled-result-before-hold.md @@ -0,0 +1,17 @@ +# 003: Capture a settling window before choosing the held frame + +**Date:** 2026-11-11 (ticket pass, Q3; agreed: "ok") + +**Question:** freeze the frame at action callback completion, or give the resulting UI time to settle? + +**Decision:** **after the action returns, use the existing bounded network-idle check, then capture another +100 ms before ending the step's visual window.** The idle check requires 500 ms of quiet and waits at most +10 seconds. Warn and continue if the page remains busy, matching the existing before-step timeout behavior. +An added postprocessing hold uses the final frame of this captured visual window. + +Keep `actionEndMs` as the callback's actual finish time. The settling window and any added hold contribute to +the step's final `endMs`; they do not redefine when the action completed. Scripts must still explicitly await +the UI state they intend to demonstrate; network quiet is not proof that a business operation succeeded. + +- **Why:** avoid freezing a loading indicator or unfinished transition when the callback returns promptly. +- **Not chosen:** freezing immediately at callback return, which is quicker but can hold an incomplete view. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/004-speech-tail-padding.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/004-speech-tail-padding.md new file mode 100644 index 0000000..89d23f7 --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/004-speech-tail-padding.md @@ -0,0 +1,14 @@ +# 004: Leave at least 200 ms after speech + +**Date:** 2026-11-11 (ticket pass, Q4; agreed: "ok") + +**Question:** fit speech exactly into its step, or leave a short pause before the next step? + +**Decision:** **each narrated step lasts at least through its clip plus 200 ms**, measured from its final +`startMs`. Existing captured time counts toward this minimum; insert only the extra hold needed. The step must +also include its captured action and settling window. Round added holds upward to whole video frames, so speech +and its minimum tail are never truncated by rounding. Blank narration gets no speech-tail padding. + +- **Why:** give successive narration lines breathing room while retaining longer captured actions. +- **Not chosen:** fitting clips exactly to step boundaries with no extra pause, which is more compact but can + sound rushed. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/005-normalize-assembled-voice-track.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/005-normalize-assembled-voice-track.md new file mode 100644 index 0000000..8618148 --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/005-normalize-assembled-voice-track.md @@ -0,0 +1,18 @@ +# 005: Normalize the assembled voice track and preserve source clips + +**Date:** 2026-11-11 (ticket pass, Q5; agreed: "ok") + +**Question:** normalize delivered narration volume or retain Qwen's original output level? + +**Decision:** **normalize the assembled voice track to a target of -16 LUFS with a -1.5 dBTP peak ceiling.** +Keep original generated WAV clips unchanged, including their recorded durations and complete-file SHA-256 hashes. +Normalization belongs to the assembled delivery audio, after its placement on the final timeline. Source clip +hashes remain the basis for the inherited same-audio comparison. + +- **Why:** a consistent delivery level across demos while keeping the reproducible source clips available. +- **Not chosen:** using the original Qwen volume for delivery, which can vary between demos. +- **Verification:** measure the rendered audio rather than assuming filter options prove the result. Exact + encoding settings, measurement tolerances and treatment of unmeasurable input belong in the implementation plan. + +Source: [FFmpeg loudnorm documentation](https://ffmpeg.org/ffmpeg-filters.html#loudnorm), checked during the +ticket pass; supports measured two-pass loudness normalization. The numerical targets are this product's choice. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/006-host-compare-command.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/006-host-compare-command.md new file mode 100644 index 0000000..12673fa --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/decisions/006-host-compare-command.md @@ -0,0 +1,18 @@ +# 006: Expose transcript comparison as a host command + +**Date:** 2026-11-11 (ticket pass, Q6; agreed: "ok") + +**Question:** let users compare their own recordings from the CLI, or keep comparison inside tests? + +**Decision:** **add `flow-reel compare a.json b.json`, running on the host without Docker.** Read the two +transcripts, check their schema and sync, and compare step order, original narration, voice identity, audio hashes +and step starts within the inherited 150 ms tolerance. Report differences and return a nonzero exit status for a +failed check. Reuse the same checks in tests. Recording also automatically checks that narration fits its steps. + +This is a narrow exception to story decision 008's host-only-launcher rule: the read-only comparison command +processes local JSON on the host. Recording, browser capture, TTS and rendering continue inside the image. + +- **Why:** users and CI can verify their own saved runs without downloading or starting the recording image. +- **Not chosen:** a test-only comparison helper, which gives users no direct way to check their recordings. +- **Boundary:** comparison uses transcript data, as required by story decision 009. Exact exit codes, diagnostics + and validation details are specified in the implementation plan; file paths alone must not count as audio changes. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/draft.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/draft.md index a733865..79a4dc9 100644 --- a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/draft.md +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/draft.md @@ -2,27 +2,52 @@ ## Problem -Voice and video are produced separately; nothing yet makes the voice fit each step, or proves two runs are the same. +GH-4 records a silent video and generates separate narration clips. Speech can outlast its action window, and +users have no command to verify that recordings are synchronized and reproducible. + +The current script API reveals narration when `step()` executes. Synthesizing everything before capture would +require another action pass or an API change. Qwen's CPU work must remain outside real-time capture. ## Change -- Narration is synthesised before recording, so the runner knows each clip's length and holds each step until its - clip ends; `endMs` records the held end (decision 011). -- The clips are placed at each step's `startMs` and muxed into `demo.mp4`. -- A sync check: no clip runs into the next step. -- A check comparing two `transcript.json` files by decision 009's rule (e.g. `flow-reel compare a.json b.json`). +- Execute actions once with the existing API. Capture first, synthesize afterward, then insert frame holds to fit + narration into the recorded video (ticket decision 001). The UI is frozen during added holds. +- Start each clip with its step's action. Preserve action speed and actual callback completion in `actionEndMs` + (decision 002). +- After each callback, use the existing bounded network-idle check (500 ms quiet, 10 s maximum), warn on timeout, + and capture another 100 ms before choosing the held frame. Scripts still explicitly await their intended UI + state (decision 003). +- Each step includes its captured action/settling window and at least its narration duration plus 200 ms from + `startMs`. Existing captured time counts; add only necessary whole-frame holds, rounding upward. Blank narration + gets no speech padding; all-blank demos retain silent-video behavior (decision 004). +- Shift transcript times, captions, events and the master used by later polish onto one final timeline. Retain + schema version 1; original narration and source WAV bytes, durations and hashes stay intact. +- Assemble the clips at final step starts, normalize delivered voice to -16 LUFS with a -1.5 dBTP peak ceiling, + and mux it into the MP4 (decision 005). Measure rendered output to verify the audio level and placement. +- Automatically check final recording sync. Add `flow-reel compare a.json b.json`, reading transcripts on the host + without Docker, printing differences and returning nonzero on failure (decision 006). +- Compare step order, original narration, voice identity and audio metadata, with step starts within ±150 ms + (story decision 009). Output/script paths do not establish a content difference. Comparison uses transcript + data; it does not claim byte-identical MP4s or perceptual frame equality. + +Decision 001 replaces this draft's earlier pre-capture synthesis proposal. Decision 006 makes a narrow exception +for host-side comparison to story decision 008; recording and rendering stay inside the image. ## Out of scope -Cursor, zoom and SFX (the next tickets); PR delivery (a later story). +Cursor/zoom polish, SFX, cloud adapters, voice changes, release publishing, PR delivery, a new script API, repeated +application actions, perceptual comparison and new rendering options. The existing odd-viewport/encoder-exit +follow-up and optional 4K master stay separate; this ticket takes up neither. ## Done when -- `flow-reel record demo.ts` writes a narrated MP4, captions and transcript in one command (decision 003). -- The sync check passes on the sample demo, and two runs compare as the same under decision 009. - -## For the ticket pass - -- Hold strategy: wait after the action, or start the clip with the action and wait for whichever ends last. -- Loudness normalisation of the voice track. -- Whether the comparison is a CLI command, a test helper, or both. +- One `flow-reel record demo.ts` command writes a playable narrated MP4, SRT/VTT, transcript and matching events. + Each application action runs once. Holds show the chosen settled frame at unchanged action playback speed. +- No narration overlaps another step; every voiced step has its 200 ms speech tail. Transcript, captions, events + and the polish master agree with the final video timeline. Original source WAV hashes remain unchanged. +- The delivered voice meets the agreed loudness/peak targets under explicit measurement tolerances. Blank-only + demos remain valid and silent. Invalid rendering/sync fails the command before a new success transcript is written. +- The comparison command reports valid matching runs as equal, identifies content/timing differences, rejects + invalid input, and works without Docker. The sample pair matches under the ±150 ms rule. +- Automated media checks verify frame holds, decoded audio placement and real output streams. Sample narrated + playback is inspected/listened to; unit tests, type checking, build and the full Docker E2E suite pass. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0842_draft.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0842_draft.md new file mode 100644 index 0000000..a733865 --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0842_draft.md @@ -0,0 +1,28 @@ +# Hold each step for its voice clip, mux the audio, and check sync and reproducibility + +## Problem + +Voice and video are produced separately; nothing yet makes the voice fit each step, or proves two runs are the same. + +## Change + +- Narration is synthesised before recording, so the runner knows each clip's length and holds each step until its + clip ends; `endMs` records the held end (decision 011). +- The clips are placed at each step's `startMs` and muxed into `demo.mp4`. +- A sync check: no clip runs into the next step. +- A check comparing two `transcript.json` files by decision 009's rule (e.g. `flow-reel compare a.json b.json`). + +## Out of scope + +Cursor, zoom and SFX (the next tickets); PR delivery (a later story). + +## Done when + +- `flow-reel record demo.ts` writes a narrated MP4, captions and transcript in one command (decision 003). +- The sync check passes on the sample demo, and two runs compare as the same under decision 009. + +## For the ticket pass + +- Hold strategy: wait after the action, or start the clip with the action and wait for whichever ends last. +- Loudness normalisation of the voice track. +- Whether the comparison is a CLI command, a test helper, or both. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0842_meta.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0842_meta.md new file mode 100644 index 0000000..8ac33bf --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0842_meta.md @@ -0,0 +1,12 @@ +--- +kind: ticket +title: Hold each step for its voice clip, mux the audio, and check sync and reproducibility +key: GH-5 +parent: GH-1 +status: tracked +branch: +tracker-url: https://github.com/codeonym-oss/flow-reel/issues/5 +mr: +created: 2026-11-02 +updated: 2026-11-02 +--- diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0925_meta.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0925_meta.md new file mode 100644 index 0000000..bf519b1 --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_0925_meta.md @@ -0,0 +1,12 @@ +--- +kind: ticket +title: Hold each step for its voice clip, mux the audio, and check sync and reproducibility +key: GH-5 +parent: GH-1 +status: planned +branch: +tracker-url: https://github.com/codeonym-oss/flow-reel/issues/5 +mr: +created: 2026-11-02 +updated: 2026-11-11 +--- diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_1018_impl.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_1018_impl.md new file mode 100644 index 0000000..f1e6366 --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/2026-11-11_1018_impl.md @@ -0,0 +1,148 @@ +# Implementation: Voice-fit holds, narration muxing and transcript comparison + +Planned 2026-11-11 against `main` at `76a31531ae34cec8ae238c1188016423579ee025`, after GH-4 merged in PR #12. +Follow inherited story decisions 002, 004, 008-013 (including supersession notes), GH-4's shipped voice/clip contract, +and this ticket's decisions 001-006. Shared understanding confirmed with "ok". No follow-ups are taken up. + +## Current code and baseline + +- `src/runner/run.ts` captures once, calibrates times against a green marker, writes a silent delivery MP4, closes + browser/display, and calls `voiceSteps`. Narration is only discovered as `step()` executes. +- `actionEndMs` and `endMs` currently coincide. Network idle is checked before a step, not after its callback. + `trackNetwork().idle()` already supplies 500 ms quiet / 10 s timeout behavior. +- GH-4 writes portable `.audio/NNN.wav` clips and version 1 voice/audio metadata. The selected Qwen/Ryan/ + English CPU runtime, seed, original narration and WAV hashes are settled; do not change the worker or its pins. +- `.flow-reel//master.mp4` currently contains calibration pre-roll. `master.json.startMs` identifies where + delivery time zero starts in it. The current events and captions use calibrated delivery-relative times. +- `src/cli.ts` only routes `record`; the host launcher starts Docker/Podman. No comparison or sync checker exists. +- Baseline: all 50 unit tests and type checking pass. The script-loader subprocess failed inside the sandbox and + passed on rerun outside. Docker E2E was not repeated for planning; merged GH-4 CI passed its full media suite. + +## Read first + +- This ticket's decisions 001-006 and inherited story decisions 008-013, especially the comparison/schema/event rules. +- GH-4 decisions 001, 005-007; GH-2 decisions 005-007 (master size, intermediates and capture-overload failure). +- `src/runner/run.ts`, `recorder.ts`, `network.ts`, `tts.ts`, `captions.ts`, `wrap.ts`. +- `src/schema/transcript.ts`, `events.ts`, `check.ts`; `src/index.ts`, `src/cli.ts`, `src/host/launch.ts`. +- `test/unit/tts.test.ts`, `schema.test.ts`, `network.test.ts`, `captions.test.ts`. +- `test/e2e/record.test.ts`, `tts.test.ts`, `player.ts`, `app/index.html`; `README.md`, `.github/workflows/ci.yml`. +- Official FFmpeg documentation for trim/setpts, tpad, concat, audio delay/padding, resampling and loudnorm. + Verify behavior against the FFmpeg version actually in the recording image. + +## Implementation shape + +Keep the reusable boundaries small: a pure timeline planner (`src/runner/timeline.ts`), media rendering +(`src/runner/render.ts`), semantic sync validation (`src/schema/sync.ts`) and pure comparison (`src/compare.ts`). +The CLI only reads files, calls comparison and maps its result to an exit code. Reuse existing capture, synthesis, +caption serialization and schema validation. No generic editing engine, provider framework or new public options. + +## Steps + +1. **Capture a settled visual window without changing action identity.** In `run.ts`, retain `actionEndMs` at + callback return, then perform the bounded idle wait plus 100 ms capture from decision 003. Save that visual + endpoint for the planner. Warn on idle timeout, as the existing before-step wait does. Keep capture's overload + guard, calibration and single execution of the body. Expensive synthesis/rendering occurs after capture and + browser/display shutdown. + → check: add a focused real-browser fixture with an action that returns before a delayed response updates the + UI. Observe failure before this change, then verify the captured endpoint includes the settled result while + actionEndMs still denotes callback return. A blank-narration fixture can exercise this without running Qwen. + Use an even-scaled viewport and bounded container cleanup, following GH-4's harness pattern. + +2. **Plan frame-exact holds and one timestamp mapping.** Add a pure planner taking calibrated captured steps, + source video frame count/FPS and the delivery cut offset. Quantize each captured visual endpoint upward to a + source frame boundary. Preserve the original calibration pre-roll. A hold is inserted after the final frame + of that step's captured window, before subsequent source frames. Compute the deficit from clip duration plus + 200 ms versus that window; add only enough whole frames, with no speech padding for blank steps. + Return hold boundaries/counts, final step times and a source-to-final timestamp mapping. Apply prior holds to + each start/actionEnd; endMs includes its visual window and own hold. Preserve source WAV metadata. Map every + event, including events in gaps, exactly once; preserve coordinates, action durations and typing cadence. + Derive durations from integer frame/sample counts rather than cumulative rounded millisecond additions. + → check: failing unit slices in `test/unit/timeline.test.ts` before implementation. At 30 fps, a step starting + at 500 ms with visual end 1000 ms and a 1000 ms clip needs 21 added frames: final end 1700 ms, and an original + next start at 1200 ms becomes 1900 ms. Cover fractional-frame boundaries, multiple holds, long actions, blank/ + empty steps, zero-duration callbacks with a visual settle window, final-step holds, nonzero pre-roll, events + before/after each boundary, unchanged event durations and nonmutation. Expected frame counts must be worked + examples, not recomputed copies of the planner's own algorithm. + +3. **Add shared sync and comparison checks.** Preserve the existing version 1 validator's compatibility; add + semantic checks at the new boundary. Require positive video dimensions/FPS, ordered nonoverlapping step + windows within the video, and startMs <= actionEndMs <= endMs. Audio must have usable hash/duration metadata, + fit within its step with the 200 ms tail, and never reach the next step. Voiced recordings must have clips for + nonblank narration; legacy silent transcripts without a voice remain comparable. Blank-only configured voice + is valid without clips. Comparison reads metadata, not referenced audio/caption files. + Compare width/height/FPS, step count/index/order, exact optional narration, optional voice identity (adapter, + voice, seed), clip presence/hash/duration, and each start within 150 ms inclusive. Ignore script/output paths; + do not compare raw caption files whose timestamps legitimately differ. End/actionEnd/video duration must be + internally valid but are not additional equality tolerances beyond story decision 009. State this limit. + Return separate validation errors and content differences so CLI can use 0 = valid/equal, 1 = valid/different, + 2 = invalid input/usage (including unreadable JSON or failed sync). + → check: `test/unit/sync.test.ts` and `compare.test.ts` cover overlap, truncated clips, missing voiced metadata, + reversed/out-of-video times, exact 150 ms acceptance and 151 ms rejection, text/order/voice/hash/duration changes, + relocated paths, blank-only and legacy silent inputs, and malformed input. Build one behavior at a time. + +4. **Render the held master and placed narration with existing FFmpeg.** Record the original capture to + `.flow-reel//capture.mp4`. Render `master.mp4` at the same resolution and 30 fps using the planned frame + spans/holds. Preserve its leading calibration frames and `master.json.startMs` offset, so delivery-relative + final events remain usable by GH-7. Check rendered frame counts against the plan. Use frame-index trim/PTS + operations for cuts and holds (e.g. trim + tpad clone + concat), rather than accumulating imprecise seeks. + Produce the delivery video with the existing Lanczos downscale, H.264 quality and fast-start behavior. + Assemble original clips at final step starts as a 48 kHz stereo PCM voice bus, pad/trim to video length, then + perform measured two-pass loudnorm (-16 LUFS, LRA 11, peak ceiling -1.5 dBTP). Keep normalized `voice.wav` under + the work folder for later rendering/SFX; public audio metadata still references original source WAVs. Encode + delivery as AAC-LC, 48 kHz stereo, 192 kb/s, with no new flags. For no clips, skip the audio bus/normalizer and + retain a video-only MP4. Reject nonempty voice input with no measurable signal instead of reporting successful + normalization. All-blank input is not a normalization failure. + Measure decoded delivery loudness and true peak. Use an explicit pre-encode peak margin if needed for AAC + overshoot; delivered integrated loudness must be within 1 LU of -16 and measured true peak <= -1.5 dBTP + (allow at most 0.1 dB measurement tolerance). If fixtures cannot meet this, report evidence before relaxing the + agreed targets. Keep frame/sample counts explicit and prevent encoding padding from truncating speech/video. + → check: a small synthetic-media test in `test/e2e/render.test.ts`, without Qwen or a browser. Use known color + changes around a hold boundary and distinct tone clips: inspect the held color, later color/event shifts, + preserved pre-roll offset and exact planned frame count. Decode AAC and verify tone onsets/endings within one + video frame of planned placement, no overlap/truncation, stereo/sample rate and measured output level. Cover + no-hold, multiple/final holds, blank-only output and render failure. Inspect original WAV hashes before/after. + +5. **Wire the record and compare commands.** Replace the early silent delivery write with the post-capture + pipeline: synthesize original clips, plan holds, render/measure final media, build/validate the final transcript + and sync, then generate captions and events on that timeline. Stage delivery media in the work folder and + publish a new success transcript only after required render/sync checks pass. Do not add a general multi-file + transaction system. Remove imports/helpers made unused by replacing the old downscale path; leave unrelated + capture behavior alone. Keep source capture and retimed master roles explicit in comments/artifact paths. + Add `compare` dispatch before CLI container routing. Lazy-load only pure comparison/schema code on that path; + recording still launches the image. Print useful file/step-specific errors or differences, preserving exit + statuses above, and extend usage text without introducing report formats or other commands. + → check: new record output assertions must first fail on the old silent MP4/short step windows, then pass. + CLI tests invoke the built entry point for equal, different, invalid JSON, failed-sync and bad-argument cases. + Prove comparison works with external executables unavailable to the child, so it cannot accidentally start + Docker/ffmpeg. Validate final transcript and captions against each actual delivery file. + +6. **Verify the complete demo and document the contract.** Extend the existing pair of recording E2Es rather + than adding more full Qwen recordings: require real narration streams, preserved source hashes, fitted windows, + final captions, retimed master/events, native playback and a successful host compare invocation. Retain the + ±150 ms and two-frame click checks; keep physical capture checks on capture.mp4 and planned frame counts on the + retimed master. The existing missing-assets test and real offline TTS tests remain meaningful. Run expensive + Docker files serially and retain their timeout cleanup and artifacts. + Update README for the narrated MP4, frozen holds, settling/padding, normalization, compare syntax/statuses, + transcript-only equality limits and work-folder roles. Replace obsolete silent-video/gap claims only where GH-5 + changes them. Existing known odd-viewport and 4K work stays in its follow-ups. No Docker/package dependency + change is planned; CI's 45-minute budget is changed only if new measurements justify it. + → check: `npm run typecheck`, `npm test`, `npm run build`, `npm run test:e2e`, `git diff --check`, and an npm pack + dry run for new compiled modules. Inspect/listen to a concrete narrated MP4 in a normal player; if listening + is unavailable to the agent, request human review of the actual artifact while other checks continue. Record + results, frame/sync evidence and resource measurements in build notes before commit approval. + +## Risks + +- Repeating actions to discover text would repeat side effects: keep synthesis after the only replay. +- Callback return/network quiet is not proof of business success. Scripts await intended state; capture the + bounded settle window before its held frame, and retain the timeout warning policy. +- Off-by-one frame cuts, AAC priming/padding and double-applied timestamp shifts can desynchronize output. Use + integer frame/sample planning, synthetic decoded-output checks, and the existing real click/frame assertion. +- A retimed delivery file with an unretimed master/events would break GH-7/GH-8. Produce those artifacts together, + preserving the master's cut-offset meaning and original source clips. +- Normalization can change level after lossy encoding. Measure decoded delivery, keep raw hashes intact and + explicitly handle silence. Source-clip equality does not claim identical encoded MP4/audio bytes. +- CPU TTS remains costly and capture still needs spare CPU. Render after capture/model execution; measure render + memory/runtime and avoid running expensive E2E files concurrently. Do not weaken capture-overload checks. +- Transcript comparison cannot prove visual/semantic UI equality or audit referenced file bytes. Document the + inherited structural/audio-metadata/step-start rule without broadening it into perceptual comparison. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/CHANGES.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/CHANGES.md index 22b6933..c3694e5 100644 --- a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/CHANGES.md +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/history/CHANGES.md @@ -1,2 +1,14 @@ - 2026-11-02: tracked as GH-5 (https://github.com/codeonym-oss/flow-reel/issues/5). - 2026-11-02: out-of-scope line updated after story decision 012. +- 2026-11-11: began the ticket pass against main at 76a3153; type checking and all 50 unit tests pass (unit suite rerun outside the sandbox after its script-loader subprocess failed inside). Agreed to single-replay capture followed by synthesis and postprocessing holds (decision 001). +- 2026-11-11: agreed to start each narration clip at its step's start alongside the action, extending the visual step where needed (decision 002). +- 2026-11-11: agreed to capture a bounded network-idle wait plus 100 ms after each callback before choosing the held frame, preserving the actual actionEndMs and warning on timeout (decision 003). +- 2026-11-11: agreed to at least 200 ms after each narration clip, counting existing captured time and rounding added holds upward to whole frames; blank narration has no speech padding (decision 004). +- 2026-11-11: agreed to normalize assembled delivery narration to -16 LUFS and a -1.5 dBTP peak ceiling, preserving original WAV bytes and transcript hashes (decision 005). +- 2026-11-11: agreed to a Docker-free host compare command with shared schema/sync/reproducibility checks and automatic recording sync validation; documented the narrow exception to story decision 008 (decision 006). +- 2026-11-11: shared understanding confirmed ("ok"); archived the prior draft/meta, wrote impl.md against 76a3153, updated the draft for postprocessing holds/normalization/host comparison, and marked planned. Tracker description update awaits approval. +- 2026-11-11: approved tracker description update applied to GH-5 and verified against draft.md; implementation invoked, awaiting the required branch/working-tree and board approval. +- 2026-11-11: approved implementation branch created from latest origin/main, retaining the agreed planning changes; implementation started with the planned test seams. +- 2026-11-11: implemented settled capture, frame holds, shared sync/compare, audio assembly/normalization and CLI routing; 63 unit tests, type checking/build and both host/image synthetic media checks pass. Full serial media suite and separate Standards/Spec reviews are running. Archived impl.md before noting fractional caption timestamps and AAC peak margin. +- 2026-11-11: implementation and both review axes complete; 63 unit tests plus all 19 media checks verified (recording file rerun: 9 passed after user freed CPU for an overload-guard failure). Type/build/package checks pass; human playback approved. Ready for commit approval; no commit or push yet. +- 2026-11-11: person approved the proposed implementation commit ("ok"); preparing feat: fit narration, mux audio and compare recordings (#5). Push/open-PR approval remains pending. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/impl.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/impl.md new file mode 100644 index 0000000..75e3fee --- /dev/null +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/impl.md @@ -0,0 +1,157 @@ +# Implementation: Voice-fit holds, narration muxing and transcript comparison + +Planned 2026-11-11 against `main` at `76a31531ae34cec8ae238c1188016423579ee025`, after GH-4 merged in PR #12. +Follow inherited story decisions 002, 004, 008-013 (including supersession notes), GH-4's shipped voice/clip contract, +and this ticket's decisions 001-006. Shared understanding confirmed with "ok". No follow-ups are taken up. + +## Current code and baseline + +- `src/runner/run.ts` captures once, calibrates times against a green marker, writes a silent delivery MP4, closes + browser/display, and calls `voiceSteps`. Narration is only discovered as `step()` executes. +- `actionEndMs` and `endMs` currently coincide. Network idle is checked before a step, not after its callback. + `trackNetwork().idle()` already supplies 500 ms quiet / 10 s timeout behavior. +- GH-4 writes portable `.audio/NNN.wav` clips and version 1 voice/audio metadata. The selected Qwen/Ryan/ + English CPU runtime, seed, original narration and WAV hashes are settled; do not change the worker or its pins. +- `.flow-reel//master.mp4` currently contains calibration pre-roll. `master.json.startMs` identifies where + delivery time zero starts in it. The current events and captions use calibrated delivery-relative times. +- `src/cli.ts` only routes `record`; the host launcher starts Docker/Podman. No comparison or sync checker exists. +- Baseline: all 50 unit tests and type checking pass. The script-loader subprocess failed inside the sandbox and + passed on rerun outside. Docker E2E was not repeated for planning; merged GH-4 CI passed its full media suite. + +## Read first + +- This ticket's decisions 001-006 and inherited story decisions 008-013, especially the comparison/schema/event rules. +- GH-4 decisions 001, 005-007; GH-2 decisions 005-007 (master size, intermediates and capture-overload failure). +- `src/runner/run.ts`, `recorder.ts`, `network.ts`, `tts.ts`, `captions.ts`, `wrap.ts`. +- `src/schema/transcript.ts`, `events.ts`, `check.ts`; `src/index.ts`, `src/cli.ts`, `src/host/launch.ts`. +- `test/unit/tts.test.ts`, `schema.test.ts`, `network.test.ts`, `captions.test.ts`. +- `test/e2e/record.test.ts`, `tts.test.ts`, `player.ts`, `app/index.html`; `README.md`, `.github/workflows/ci.yml`. +- Official FFmpeg documentation for trim/setpts, tpad, concat, audio delay/padding, resampling and loudnorm. + Verify behavior against the FFmpeg version actually in the recording image. + +## Implementation shape + +Keep the reusable boundaries small: a pure timeline planner (`src/runner/timeline.ts`), media rendering +(`src/runner/render.ts`), semantic sync validation (`src/schema/sync.ts`) and pure comparison (`src/compare.ts`). +The CLI only reads files, calls comparison and maps its result to an exit code. Reuse existing capture, synthesis, +caption serialization and schema validation. No generic editing engine, provider framework or new public options. + +## Steps + +1. **Capture a settled visual window without changing action identity.** In `run.ts`, retain `actionEndMs` at + callback return, then perform the bounded idle wait plus 100 ms capture from decision 003. Save that visual + endpoint for the planner. Warn on idle timeout, as the existing before-step wait does. Keep capture's overload + guard, calibration and single execution of the body. Expensive synthesis/rendering occurs after capture and + browser/display shutdown. + → check: add a focused real-browser fixture with an action that returns before a delayed response updates the + UI. Observe failure before this change, then verify the captured endpoint includes the settled result while + actionEndMs still denotes callback return. A blank-narration fixture can exercise this without running Qwen. + Use an even-scaled viewport and bounded container cleanup, following GH-4's harness pattern. + +2. **Plan frame-exact holds and one timestamp mapping.** Add a pure planner taking calibrated captured steps, + source video frame count/FPS and the delivery cut offset. Quantize each captured visual endpoint upward to a + source frame boundary. Preserve the original calibration pre-roll. A hold is inserted after the final frame + of that step's captured window, before subsequent source frames. Compute the deficit from clip duration plus + 200 ms versus that window; add only enough whole frames, with no speech padding for blank steps. + Return hold boundaries/counts, final step times and a source-to-final timestamp mapping. Apply prior holds to + each start/actionEnd; endMs includes its visual window and own hold. Preserve source WAV metadata. Map every + event, including events in gaps, exactly once; preserve coordinates, action durations and typing cadence. + Derive durations from integer frame/sample counts rather than cumulative rounded millisecond additions. + → check: failing unit slices in `test/unit/timeline.test.ts` before implementation. At 30 fps, a step starting + at 500 ms with visual end 1000 ms and a 1000 ms clip needs 21 added frames: final end 1700 ms, and an original + next start at 1200 ms becomes 1900 ms. Cover fractional-frame boundaries, multiple holds, long actions, blank/ + empty steps, zero-duration callbacks with a visual settle window, final-step holds, nonzero pre-roll, events + before/after each boundary, unchanged event durations and nonmutation. Expected frame counts must be worked + examples, not recomputed copies of the planner's own algorithm. + +3. **Add shared sync and comparison checks.** Preserve the existing version 1 validator's compatibility; add + semantic checks at the new boundary. Require positive video dimensions/FPS, ordered nonoverlapping step + windows within the video, and startMs <= actionEndMs <= endMs. Audio must have usable hash/duration metadata, + fit within its step with the 200 ms tail, and never reach the next step. Voiced recordings must have clips for + nonblank narration; legacy silent transcripts without a voice remain comparable. Blank-only configured voice + is valid without clips. Comparison reads metadata, not referenced audio/caption files. + Compare width/height/FPS, step count/index/order, exact optional narration, optional voice identity (adapter, + voice, seed), clip presence/hash/duration, and each start within 150 ms inclusive. Ignore script/output paths; + do not compare raw caption files whose timestamps legitimately differ. End/actionEnd/video duration must be + internally valid but are not additional equality tolerances beyond story decision 009. State this limit. + Return separate validation errors and content differences so CLI can use 0 = valid/equal, 1 = valid/different, + 2 = invalid input/usage (including unreadable JSON or failed sync). + → check: `test/unit/sync.test.ts` and `compare.test.ts` cover overlap, truncated clips, missing voiced metadata, + reversed/out-of-video times, exact 150 ms acceptance and 151 ms rejection, text/order/voice/hash/duration changes, + relocated paths, blank-only and legacy silent inputs, and malformed input. Build one behavior at a time. + +4. **Render the held master and placed narration with existing FFmpeg.** Record the original capture to + `.flow-reel//capture.mp4`. Render `master.mp4` at the same resolution and 30 fps using the planned frame + spans/holds. Preserve its leading calibration frames and `master.json.startMs` offset, so delivery-relative + final events remain usable by GH-7. Check rendered frame counts against the plan. Use frame-index trim/PTS + operations for cuts and holds (e.g. trim + tpad clone + concat), rather than accumulating imprecise seeks. + Produce the delivery video with the existing Lanczos downscale, H.264 quality and fast-start behavior. + Assemble original clips at final step starts as a 48 kHz stereo PCM voice bus, pad/trim to video length, then + perform measured two-pass loudnorm (-16 LUFS, LRA 11, peak ceiling -1.5 dBTP). Keep normalized `voice.wav` under + the work folder for later rendering/SFX; public audio metadata still references original source WAVs. Encode + delivery as AAC-LC, 48 kHz stereo, 192 kb/s, with no new flags. For no clips, skip the audio bus/normalizer and + retain a video-only MP4. Reject nonempty voice input with no measurable signal instead of reporting successful + normalization. All-blank input is not a normalization failure. + Measure decoded delivery loudness and true peak. Use an explicit pre-encode peak margin if needed for AAC + overshoot; delivered integrated loudness must be within 1 LU of -16 and measured true peak <= -1.5 dBTP + (allow at most 0.1 dB measurement tolerance). If fixtures cannot meet this, report evidence before relaxing the + agreed targets. Keep frame/sample counts explicit and prevent encoding padding from truncating speech/video. + → check: a small synthetic-media test in `test/e2e/render.test.ts`, without Qwen or a browser. Use known color + changes around a hold boundary and distinct tone clips: inspect the held color, later color/event shifts, + preserved pre-roll offset and exact planned frame count. Decode AAC and verify tone onsets/endings within one + video frame of planned placement, no overlap/truncation, stereo/sample rate and measured output level. Cover + no-hold, multiple/final holds, blank-only output and render failure. Inspect original WAV hashes before/after. + +5. **Wire the record and compare commands.** Replace the early silent delivery write with the post-capture + pipeline: synthesize original clips, plan holds, render/measure final media, build/validate the final transcript + and sync, then generate captions and events on that timeline. Stage delivery media in the work folder and + publish a new success transcript only after required render/sync checks pass. Do not add a general multi-file + transaction system. Remove imports/helpers made unused by replacing the old downscale path; leave unrelated + capture behavior alone. Keep source capture and retimed master roles explicit in comments/artifact paths. + Add `compare` dispatch before CLI container routing. Lazy-load only pure comparison/schema code on that path; + recording still launches the image. Print useful file/step-specific errors or differences, preserving exit + statuses above, and extend usage text without introducing report formats or other commands. + → check: new record output assertions must first fail on the old silent MP4/short step windows, then pass. + CLI tests invoke the built entry point for equal, different, invalid JSON, failed-sync and bad-argument cases. + Prove comparison works with external executables unavailable to the child, so it cannot accidentally start + Docker/ffmpeg. Validate final transcript and captions against each actual delivery file. + +6. **Verify the complete demo and document the contract.** Extend the existing pair of recording E2Es rather + than adding more full Qwen recordings: require real narration streams, preserved source hashes, fitted windows, + final captions, retimed master/events, native playback and a successful host compare invocation. Retain the + ±150 ms and two-frame click checks; keep physical capture checks on capture.mp4 and planned frame counts on the + retimed master. The existing missing-assets test and real offline TTS tests remain meaningful. Run expensive + Docker files serially and retain their timeout cleanup and artifacts. + Update README for the narrated MP4, frozen holds, settling/padding, normalization, compare syntax/statuses, + transcript-only equality limits and work-folder roles. Replace obsolete silent-video/gap claims only where GH-5 + changes them. Existing known odd-viewport and 4K work stays in its follow-ups. No Docker/package dependency + change is planned; CI's 45-minute budget is changed only if new measurements justify it. + → check: `npm run typecheck`, `npm test`, `npm run build`, `npm run test:e2e`, `git diff --check`, and an npm pack + dry run for new compiled modules. Inspect/listen to a concrete narrated MP4 in a normal player; if listening + is unavailable to the agent, request human review of the actual artifact while other checks continue. Record + results, frame/sync evidence and resource measurements in build notes before commit approval. + +## Risks + +- Repeating actions to discover text would repeat side effects: keep synthesis after the only replay. +- Callback return/network quiet is not proof of business success. Scripts await intended state; capture the + bounded settle window before its held frame, and retain the timeout warning policy. +- Off-by-one frame cuts, AAC priming/padding and double-applied timestamp shifts can desynchronize output. Use + integer frame/sample planning, synthetic decoded-output checks, and the existing real click/frame assertion. +- A retimed delivery file with an unretimed master/events would break GH-7/GH-8. Produce those artifacts together, + preserving the master's cut-offset meaning and original source clips. +- Normalization can change level after lossy encoding. Measure decoded delivery, keep raw hashes intact and + explicitly handle silence. Source-clip equality does not claim identical encoded MP4/audio bytes. +- CPU TTS remains costly and capture still needs spare CPU. Render after capture/model execution; measure render + memory/runtime and avoid running expensive E2E files concurrently. Do not weaken capture-overload checks. +- Transcript comparison cannot prove visual/semantic UI equality or audit referenced file bytes. Document the + inherited structural/audio-metadata/step-start rule without broadening it into perceptual comparison. + +## Implementation notes + +- Frame-derived timestamps retain fractional milliseconds in JSON. Caption serialization rounds to whole + milliseconds (with a regression test across a second boundary), because SRT/WebVTT timestamp fields require it. +- The renderer uses a -2 dBTP pre-encode normalization ceiling (0.5 dB margin) and checks decoded AAC against the + agreed -1.5 dBTP ceiling with 0.1 dB measurement tolerance. Synthetic tone fixtures pass on host and image FFmpeg. +- The review baseline is 76a3153 plus the uncommitted implementation, including new untracked modules/tests. + groundwork.md supplies tracker configuration; docs/agents/issue-tracker.md is absent. diff --git a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/meta.md b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/meta.md index 8ac33bf..da48030 100644 --- a/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/meta.md +++ b/planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-5-voice-fit-mux-sync-check/meta.md @@ -3,10 +3,10 @@ kind: ticket title: Hold each step for its voice clip, mux the audio, and check sync and reproducibility key: GH-5 parent: GH-1 -status: tracked -branch: +status: building +branch: feat/5-voice-fit-mux-sync-check tracker-url: https://github.com/codeonym-oss/flow-reel/issues/5 mr: created: 2026-11-02 -updated: 2026-11-02 +updated: 2026-11-11 --- diff --git a/src/cli.ts b/src/cli.ts index 48ef956..03f7e90 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1,12 +1,31 @@ #!/usr/bin/env node // `flow-reel record