|
| 1 | +# Rule: Captions Workflow (audio → word timings → caption component) |
| 2 | + |
| 3 | +Word-synced captions are a two-step pipeline: generate word timings with |
| 4 | +`rustmotion captions`, then inject them into a scenario through the variables |
| 5 | +system. Never hand-write word timings for real voiceovers. |
| 6 | + |
| 7 | +## Step 1 — Generate word timings |
| 8 | + |
| 9 | +```bash |
| 10 | +# From audio (requires whisper.cpp: brew install whisper-cpp) |
| 11 | +rustmotion captions voice.mp3 -o words.json |
| 12 | + |
| 13 | +# Pick a model / force the language |
| 14 | +rustmotion captions voice.mp3 -o words.json --model small --lang fr |
| 15 | + |
| 16 | +# Offline import from existing subtitles (no whisper needed) |
| 17 | +rustmotion captions --from-srt subs.srt -o words.json |
| 18 | +rustmotion captions --from-vtt subs.vtt -o words.json |
| 19 | +``` |
| 20 | + |
| 21 | +Output shape (also valid as a `--props` file, on purpose): |
| 22 | + |
| 23 | +```json |
| 24 | +{ |
| 25 | + "words": [ |
| 26 | + { "text": "Hello", "start": 0.0, "end": 0.32 }, |
| 27 | + { "text": "world", "start": 0.32, "end": 0.7 } |
| 28 | + ] |
| 29 | +} |
| 30 | +``` |
| 31 | + |
| 32 | +- Transcription uses a whisper.cpp binary found in PATH (`whisper-cli`, |
| 33 | + `whisper-cpp`, `main`). Models are resolved from |
| 34 | + `~/.cache/whisper/ggml-<name>.bin` or a direct `.bin` path. |
| 35 | +- **SRT/VTT approximation**: subtitle cues carry sentence-level timing only, |
| 36 | + so each cue's duration is spread **uniformly** across its words. Good |
| 37 | + enough for karaoke modes; for precise word pops prefer real transcription. |
| 38 | + |
| 39 | +## Step 2 — Inject into the scenario |
| 40 | + |
| 41 | +Declare a `words` variable in `config` and reference it from the caption |
| 42 | +component: |
| 43 | + |
| 44 | +```json |
| 45 | +{ |
| 46 | + "config": { "words": { "type": "array", "default": [] } }, |
| 47 | + "scenes": [ |
| 48 | + { |
| 49 | + "duration": 5, |
| 50 | + "children": [ |
| 51 | + { |
| 52 | + "type": "caption", |
| 53 | + "mode": "word_pop", |
| 54 | + "words": "$words", |
| 55 | + "active_color": "#FFFFFF", |
| 56 | + "pill_color": "#FF3366", |
| 57 | + "position": { "x": 0, "y": 1600 }, |
| 58 | + "style": { "width": "100%", "font-size": 72 } |
| 59 | + } |
| 60 | + ] |
| 61 | + } |
| 62 | + ] |
| 63 | +} |
| 64 | +``` |
| 65 | + |
| 66 | +```bash |
| 67 | +rustmotion render -f scenario.json --props words.json |
| 68 | +``` |
| 69 | + |
| 70 | +## Caption modes |
| 71 | + |
| 72 | +| Mode | Behavior | |
| 73 | +|---|---| |
| 74 | +| `highlight` (default) | All words visible, active word takes `active_color` | |
| 75 | +| `karaoke` | Same as highlight (karaoke-style progression) | |
| 76 | +| `word_by_word` | Only the active word, centered, no pill | |
| 77 | +| `word_pop` | TikTok style: only the active word, spring scale-in, pill background | |
| 78 | +| `karaoke_pop` | Full line visible + active word scales 1.15x with `active_color` and pill | |
| 79 | + |
| 80 | +- `pill_color` (word_pop / karaoke_pop): pill background behind the active |
| 81 | + word. Defaults to black at 70% (`#000000B3`). |
| 82 | +- `max_width`: wraps the line in highlight/karaoke/karaoke_pop modes — always |
| 83 | + set it on vertical formats to avoid viewport overflow. |
| 84 | +- The caption paints from its baseline: position it with enough headroom |
| 85 | + (font-size + pill padding above the anchor point). |
0 commit comments