feat: preserve narration and write SRT/VTT captions (#3)

README.md

@@ -4,8 +4,8 @@ Record reproducible feature demos from a short Playwright script. A coding agent
 writes the script, and `flow-reel` replays it in a sandbox on a virtual display while recording the screen. The same
 script gives the same video, step for step.
 
-This release records **silent** video with step timings. Narration, captions, voice, cursor and zoom polish, and sound
-effects come in later releases.
+This release records **silent** video with narration text, captions and step timings. Voice, cursor and zoom polish,
+and sound effects come in later releases.
 
 ## Requirements
 
@@ -58,8 +58,18 @@ Next to the script:
 | File | What it is |
 |---|---|
 | `settings.mp4` | The silent demo, 1920×1080, 30 fps. |
-| `settings.transcript.json` | Each step's `startMs`, `actionEndMs` and `endMs` on the video's timeline (schema version 1). |
+| `settings.transcript.json` | Each step's original narration and `startMs`, `actionEndMs` and `endMs` on the video's timeline (schema version 1). |
 | `settings.events.json` | Every move, click, keystroke run and scroll the runner made, timed the same way (schema version 1). |
+| `settings.srt` | Captions for a subtitle-capable video player. |
+| `settings.vtt` | WebVTT captions for a browser's `<video>` / `<track>`. |
+
+Each caption shows one step's narration from its `startMs` to its `endMs`. Caption whitespace is normalized and the
+player wraps the text; the transcript keeps the original narration. Blank narration and steps lasting zero
+milliseconds stay in the transcript but produce no caption. Short actions can be too brief to read: step holds for
+voice clips come in a later release.
+
+Captions escape `&`, `<` and `>` so narration is emitted without raw tags. WebVTT displays the literal text; some
+SRT players display escape sequences such as `&lt;` instead. Use WebVTT in a browser when literal symbols matter.
 
 In `.flow-reel/settings/`: the 2560×1440 master the polish steps will render from. The `.flow-reel/` folder ignores
 itself in git; the files next to the script are yours to commit or not.
@@ -76,7 +86,7 @@ a script have the same steps, each starting within ±150 ms.
 
 ## Known gaps
 
-- No narration, captions or audio yet.
+- No synthesized voice or audio yet; narration is text only.
 - No cursor drawn in the video yet: the runner logs the motion in `events.json` for the polish step.
 - Playwright calls the runner doesn't cover run without any mouse motion (the warning names them).
 

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/decisions/001-one-caption-per-step.md

@@ -0,0 +1,15 @@
+# 001: One caption cue per step, using its recorded boundaries
+
+**Date:** 2026-11-10 (GH-3 ticket pass, Q1; recommended option 1, agreed: "ok")
+
+**Question:** should long narration stay in one caption or be split into shorter cues?
+
+**Decision:** emit one cue per captioned step, containing its complete narration, from `startMs` to `endMs`.
+Do not split narration into separately timed chunks or extend the step for readability. GH-5 owns voice holds;
+captions use the resulting boundaries when those holds exist.
+
+- **Why:** the runner knows step boundaries but has no timing for individual words or sentences. The same rule
+  works before and after voice is added.
+- **Not chosen:** splitting long narration into shorter cues (would introduce estimated timing before voice exists).
+- **Cost accepted:** long narration can occupy several lines, and short actions can give little reading time until
+  GH-5 introduces voice holds.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/decisions/002-player-wraps-plain-text.md

@@ -0,0 +1,16 @@
+# 002: Players wrap plain caption text; the transcript preserves narration
+
+**Partly superseded by 004** (SRT players may display escape sequences instead of the literal characters).
+
+**Date:** 2026-11-10 (GH-3 ticket pass, Q2; recommended option 1, agreed: "ok")
+
+**Question:** should captions use player-controlled wrapping or fixed-length line breaks?
+
+**Decision:** normalize caption whitespace into a single line and let the player wrap it for its display. Preserve
+the original narration unchanged in the transcript. Emit narration as literal caption text, using the format's
+escaping rules so markup-like input is not interpreted as styling or cue structure.
+
+- **Why:** wrapping adapts to the player's available width without inventing fixed character limits or altering the
+  narration stored for later voice synthesis.
+- **Not chosen:** inserting line breaks at a fixed length (extra formatting rules, awkward on smaller displays).
+- **Cost accepted:** visual line breaks can differ between players and display sizes.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/decisions/003-skip-empty-or-zero-duration-cues.md

@@ -0,0 +1,14 @@
+# 003: Skip captions with blank text or no duration
+
+**Date:** 2026-11-10 (GH-3 ticket pass, Q3; recommended option 1, agreed: "ok")
+
+**Question:** should blank narration or a zero-duration step skip its caption or fail the recording?
+
+**Decision:** preserve every step and its original narration in the transcript. Emit a caption only when its
+normalized narration is nonblank and `endMs > startMs`. Skipping a cue never removes a transcript step, changes
+timing, extends a step, or fails an otherwise successful recording. Number emitted SRT cues consecutively.
+
+- **Why:** intentionally silent and instantaneous actions remain valid, while subtitle files contain only usable
+  cues on the recorded timeline.
+- **Not chosen:** failing recordings that include these steps (unnecessarily forbids silent steps).
+- **Cost accepted:** not every transcript step has a corresponding subtitle cue.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/decisions/004-escape-text-with-srt-compatibility-limit.md

@@ -0,0 +1,26 @@
+# 004: Escape caption text and document SRT player differences
+
+**Date:** 2026-11-10 (GH-3 ticket pass, Q4; recommended option 1, agreed: "ok")
+
+**Partly supersedes 002:** literal text is guaranteed by WebVTT escaping; SRT players may display the escape sequences.
+
+**Question:** how should markup-like narration be handled when SRT players do not consistently decode escapes?
+
+**Decision:** after normalizing caption whitespace, escape `&`, `<` and `>` as `&amp;`, `&lt;` and `&gt;` in both
+formats. Escape original characters once, preserving existing entity-looking input as literal text in WebVTT.
+Keep original narration unchanged in the transcript. Document that some SRT players display these sequences rather
+than decoding them; do not promise identical literal rendering across all players.
+
+- **Why:** captions do not emit raw tags; WebVTT can represent the original text, and SRT has a clear, documented
+  compatibility limit without changing the script API or rejecting narration.
+- **Not chosen:** raw SRT text (players may interpret tags as styling).
+- **Cost accepted:** SRT captions containing special characters may show escape sequences in some players.
+
+## Sources checked
+
+- [WebVTT syntax and cue text](https://www.w3.org/TR/webvtt1/#syntax): cue text supports character references;
+  encoding `>` also prevents a raw `-->` inside a cue payload.
+- [Library of Congress SRT description](https://www.loc.gov/preservation/digital/formats/fdd/fdd000569.shtml):
+  describes the format's structure and player-dependent formatting.
+- [FFmpeg subtitle decoder](https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/htmlsubtitles.c): handles tags
+  and otherwise copies text; HTML entity decoding must not be assumed for SRT playback.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/draft.md

@@ -6,19 +6,30 @@ After ticket 1 the video has timings but no words: a viewer can't tell what each
 
 ## Change
 
-- Each step's `narration` (the first argument of `step()`) goes into `transcript.json` (decision 011).
-- `demo.srt` and `demo.vtt` are written from the steps' `narration`, `startMs` and `endMs`.
+- Each step's original `narration` (the first argument of `step()`) goes into `demo.transcript.json` unchanged
+  (story decision 011). Version 1 transcripts without narration remain valid.
+- `demo.srt` and `demo.vtt` are written beside the script and MP4, from the final calibrated `startMs` and `endMs`.
+- One cue per captioned step, with no splitting or added reading time (ticket decision 001).
+- Caption whitespace is normalized and players wrap the text; the transcript retains original whitespace
+  (decision 002). Blank narration and zero-duration steps produce no cue but stay in the transcript (decision 003).
+- Caption text escapes `&`, `<` and `>`. WebVTT displays literal text; some SRT players display escape sequences.
+  This compatibility limit is documented (decision 004).
 
 ## Out of scope
 
-Voice (tickets 3-4); burned-in captions (polish, a later story).
+Voice, step holds and audio muxing (GH-4/GH-5); burned-in captions; cursor/zoom polish; caption styling options.
 
 ## Done when
 
 - The sample demo writes captions that play in sync in a standard player (mpv, VLC, a `<video>` with `<track>`).
 - Two runs give identical caption text and step order.
+- New recordings preserve original narration on every transcript step, and each emitted cue uses that run's
+  final step boundaries. Recording timing stays unchanged.
+- Tests cover whitespace, escaping, Unicode, timestamp boundaries, skipped cues, old transcripts and empty output.
+- Generated WebVTT loads in a real browser with correct decoded text and times; generated SRT parses with correct
+  times, and its player-dependent escaping limitation is documented.
 
-## For the ticket pass
+## Decisions and implementation
 
-- Long narration: one cue per step, or split into cues by length?
-- Escaping and line breaking in SRT/VTT.
+Ticket decisions 001-004 settle cue boundaries, wrapping, skipped cues and escaping. See `impl.md` for the build
+steps, checks and compatibility risks.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/history/2026-11-10_1231_draft.md

@@ -0,0 +1,24 @@
+# Narrate steps in the transcript and write SRT/VTT captions
+
+## Problem
+
+After ticket 1 the video has timings but no words: a viewer can't tell what each step shows.
+
+## Change
+
+- Each step's `narration` (the first argument of `step()`) goes into `transcript.json` (decision 011).
+- `demo.srt` and `demo.vtt` are written from the steps' `narration`, `startMs` and `endMs`.
+
+## Out of scope
+
+Voice (tickets 3-4); burned-in captions (polish, a later story).
+
+## Done when
+
+- The sample demo writes captions that play in sync in a standard player (mpv, VLC, a `<video>` with `<track>`).
+- Two runs give identical caption text and step order.
+
+## For the ticket pass
+
+- Long narration: one cue per step, or split into cues by length?
+- Escaping and line breaking in SRT/VTT.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/history/2026-11-10_1231_meta.md

@@ -0,0 +1,12 @@
+---
+kind: ticket
+title: Narrate steps in the transcript and write SRT/VTT captions
+key: GH-3
+parent: GH-1
+status: tracked
+branch:
+tracker-url: https://github.com/codeonym-oss/flow-reel/issues/3
+mr:
+created: 2026-11-02
+updated: 2026-11-02
+---

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/history/2026-11-10_1246_meta.md

@@ -0,0 +1,12 @@
+---
+kind: ticket
+title: Narrate steps in the transcript and write SRT/VTT captions
+key: GH-3
+parent: GH-1
+status: planned
+branch:
+tracker-url: https://github.com/codeonym-oss/flow-reel/issues/3
+mr:
+created: 2026-11-02
+updated: 2026-11-10
+---

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/history/2026-11-10_1324_impl.md

@@ -0,0 +1,88 @@
+# Implementation: Narration text and SRT/VTT captions
+
+Planned 2026-11-10 against `main` at `a5e83798cdebfd817b7404a70868575236b55503`, after GH-2 merged as PR #10.
+Scope: preserve narration and write caption sidecars; the MP4 stays silent. Settled by story decisions 009-011,
+GH-2 decision 006 (output locations), and this ticket's decisions 001-004. No follow-ups are taken up.
+
+## Current code and baseline
+
+- `src/index.ts` already defines `step(narration, action)`; `src/runner/run.ts` receives the text but omits it from
+  `steps.push()`. The runner calibrates all step times before writing the final transcript.
+- `src/schema/transcript.ts` already accepts optional `narration` in version 1. Keep that optional field so old
+  GH-2 transcripts remain valid; new recordings write narration on every step.
+- No caption module or output exists. `test/e2e/record.test.ts` already records the same fixture twice and checks
+  output validity, timing reproducibility and click/frame alignment.
+- Baseline: all 7 schema tests and `npm run typecheck` passed during the ticket pass. The previous project check
+  passed all 36 unit tests. Docker E2E was not rerun for planning; GH-2's merged PR had passing E2E CI.
+
+## Read first
+
+- `../../decisions/009-what-counts-as-the-same-video.md`, `010-demo-script-api.md`, `011-transcript-json-schema.md`.
+- `../GH-2-silent-replay-recording-sandbox/decisions/006-intermediates-in-flow-reel-folder.md`.
+- This ticket's `decisions/001`-`004`.
+- `src/runner/run.ts`, `src/schema/transcript.ts`, `src/index.ts`.
+- `test/unit/schema.test.ts`, `test/e2e/record.test.ts`, `README.md`, `.github/workflows/ci.yml`.
+- [WebVTT file syntax](https://www.w3.org/TR/webvtt1/#syntax) and the source notes in decision 004.
+
+## Steps
+
+1. **Add a pure caption serializer**, `src/runner/captions.ts`, with
+   `renderCaptions(steps: readonly TranscriptStep[]): { srt: string; vtt: string }`. Keep timestamp formatting and
+   text escaping private to that module; no dependencies, public API additions, CLI flags or generic subtitle
+   framework. Use the final step times as supplied, without changing or mutating the steps.
+   - One cue per eligible step, in input order, from `startMs` to `endMs`. Do not split text or extend boundaries.
+   - Normalize whitespace runs to a single space and trim for captions only. Skip absent narration (old schema),
+     blank normalized text and `endMs <= startMs`. Number emitted SRT cues consecutively starting at 1.
+   - Escape original `&`, `<`, `>` exactly once as agreed in decision 004. Preserve Unicode; do not interpret
+     markup or pre-existing character references.
+   - SRT uses `HH:MM:SS,mmm`; WebVTT uses `HH:MM:SS.mmm` and starts with `WEBVTT` plus a blank line. Use LF line
+     endings, blank lines between cues and a terminating newline. Empty input produces an empty SRT string and a
+     header-only WebVTT string (`WEBVTT\n\n`). Hours have at least two digits and do not wrap after 24.
+   → check: write failing unit tests in `test/unit/captions.test.ts`, then implement. Cover exact output in both
+   formats, millisecond precision and minute/hour carries, long text staying one cue, Unicode, tabs/CRLF/blank
+   lines, `<b>`, `&`, entity-looking text and `-->`, skipped cues with consecutive numbering, empty input, old
+   steps without narration, deterministic repeated serialization and unchanged input steps.
+
+2. **Preserve narration and write captions in the recording path.** In `src/runner/run.ts`, include the original
+   `narration` in each completed step. Add `<name>.srt` and `<name>.vtt` beside the script, using the same basename
+   as its MP4 and JSON files. Generate from the validated transcript after timing calibration; write UTF-8.
+   Keep schema version 1, `endMs = actionEndMs`, recording timing, master files and audio behavior as they are.
+   → check: extend the existing E2E output assertions so they fail until recording writes narration and both
+   sidecars. Keep the schema test accepting old GH-2 transcripts; verify invalid non-string narration is rejected
+   by the existing validator, without making narration mandatory globally.
+
+3. **Verify actual output and playback.** Extend `test/e2e/record.test.ts` rather than creating another recording
+   pipeline. Assert exact narration in both transcripts, caption text/order matching the script, and cue boundaries
+   matching each run's own final transcript. Compare text/order between runs, not whole files: timestamps can
+   differ by the allowed ±150 ms. Keep the existing frame and reproducibility checks.
+   → check: parse a generated VTT through Chromium's native `<track>` loader using Playwright already in the
+   recording image (serve HTML/video/VTT from a local fixture or intercepted HTTP routes). Assert parsed cue times
+   and decoded `getCueAsHTML().textContent`; check a markup-like text fixture preserves literal text. Load generated
+   SRT with host ffprobe/ffmpeg and assert cue count and timestamps; do not mistake parsing for visual validation
+   of SRT escapes. Inspect sample caption playback in a standard player/browser and report what was verified.
+
+4. **Update documentation.** In `README.md`, describe narration text and SRT/VTT sidecars, their paths, timing,
+   skipped cues, player-controlled wrapping and silent MP4 behavior. Explain the SRT escaping limit from decision
+   004 and that short steps can be hard to read until GH-5 adds voice holds.
+   → check: README output names match a recorded fixture, and no statement claims voice/audio is implemented.
+
+5. **Validate the completed change.** Run `npm run typecheck`, `npm test`, `npm run build` and `npm run test:e2e`.
+   The existing CI runs these checks; no workflow or image change is planned.
+   → check: checks pass, and recorded outputs preserve GH-2's timing and resolution guarantees. If a required
+   playback check cannot run, report it explicitly rather than treating serializer tests as playback proof.
+
+## Tests
+
+- Unit: caption serialization and edge cases; existing version 1 schema compatibility.
+- E2E: the existing pair of fixture recordings, narration persistence, caption sidecars, a real WebVTT parser,
+  SRT parsing and timestamps, and the retained ±150 ms/two-frame checks.
+- Playback: observe captions on the recorded sample in a standard player/browser; document SRT escape differences.
+
+## Risks
+
+- Captions generated before calibration would be offset from the MP4: serialize only final transcript times.
+- SRT escaping is player-dependent; exact literal rendering is guaranteed for WebVTT, not all SRT players
+  (decision 004). Do not strip narration or add player-specific output modes.
+- A short action may be too brief to read; GH-5 owns holds. Keep empty/instantaneous steps in the transcript.
+- Reproducible text does not imply byte-identical caption files across recordings: timestamps may vary.
+- Docker E2E needs enough free CPU; capture overload remains a recording failure under GH-2 decision 007.

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/history/2026-11-10_1327_meta.md

@@ -0,0 +1,12 @@
+---
+kind: ticket
+title: Narrate steps in the transcript and write SRT/VTT captions
+key: GH-3
+parent: GH-1
+status: building
+branch: feat/3-narrated-transcript-and-captions
+tracker-url: https://github.com/codeonym-oss/flow-reel/issues/3
+mr:
+created: 2026-11-02
+updated: 2026-11-10
+---

planning/GH-1-narrated-reproducible-feature-demos/tickets/GH-3-narrated-transcript-and-captions/history/CHANGES.md

@@ -1 +1,6 @@
 - 2026-11-02: tracked as GH-3 (https://github.com/codeonym-oss/flow-reel/issues/3).
+- 2026-11-10: ticket pass against merged GH-2: decisions 001-004 agreed, impl.md written, draft clarified and status set to planned. Seven schema tests and typecheck passed; GitHub issue description not yet updated.
+- 2026-11-10: with approval, updated GH-3's GitHub issue description to match draft.md and verified an exact match.
+- 2026-11-10: building on feat/3-narrated-transcript-and-captions; planning changes carried into the branch.
+- 2026-11-10: implementation complete: original narration, SRT/VTT sidecars, serializer and playback tests, README. Typecheck/build, 45 unit tests and 6 E2E tests passed; screenshot inspected; standards/spec reviews found no issues. Awaiting commit approval.
+- 2026-11-10: committed as 8aa061f; in review at https://github.com/codeonym-oss/flow-reel/pull/11.