Skip to content

Commit 039ec0c

Browse files
authored
feat(captions): whisper transcription, SRT/VTT import and TikTok presets (#58)
Closes #45. Subprocess whisper transcription with actionable hints, offline SRT/VTT importers with uniform word spread, word_pop/karaoke_pop caption presets, props-system integration verified end-to-end.
1 parent c191df4 commit 039ec0c

8 files changed

Lines changed: 1126 additions & 1 deletion

File tree

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Rule: Captions Workflow (audio → word timings → caption component)
2+
3+
Word-synced captions are a two-step pipeline: generate word timings with
4+
`rustmotion captions`, then inject them into a scenario through the variables
5+
system. Never hand-write word timings for real voiceovers.
6+
7+
## Step 1 — Generate word timings
8+
9+
```bash
10+
# From audio (requires whisper.cpp: brew install whisper-cpp)
11+
rustmotion captions voice.mp3 -o words.json
12+
13+
# Pick a model / force the language
14+
rustmotion captions voice.mp3 -o words.json --model small --lang fr
15+
16+
# Offline import from existing subtitles (no whisper needed)
17+
rustmotion captions --from-srt subs.srt -o words.json
18+
rustmotion captions --from-vtt subs.vtt -o words.json
19+
```
20+
21+
Output shape (also valid as a `--props` file, on purpose):
22+
23+
```json
24+
{
25+
"words": [
26+
{ "text": "Hello", "start": 0.0, "end": 0.32 },
27+
{ "text": "world", "start": 0.32, "end": 0.7 }
28+
]
29+
}
30+
```
31+
32+
- Transcription uses a whisper.cpp binary found in PATH (`whisper-cli`,
33+
`whisper-cpp`, `main`). Models are resolved from
34+
`~/.cache/whisper/ggml-<name>.bin` or a direct `.bin` path.
35+
- **SRT/VTT approximation**: subtitle cues carry sentence-level timing only,
36+
so each cue's duration is spread **uniformly** across its words. Good
37+
enough for karaoke modes; for precise word pops prefer real transcription.
38+
39+
## Step 2 — Inject into the scenario
40+
41+
Declare a `words` variable in `config` and reference it from the caption
42+
component:
43+
44+
```json
45+
{
46+
"config": { "words": { "type": "array", "default": [] } },
47+
"scenes": [
48+
{
49+
"duration": 5,
50+
"children": [
51+
{
52+
"type": "caption",
53+
"mode": "word_pop",
54+
"words": "$words",
55+
"active_color": "#FFFFFF",
56+
"pill_color": "#FF3366",
57+
"position": { "x": 0, "y": 1600 },
58+
"style": { "width": "100%", "font-size": 72 }
59+
}
60+
]
61+
}
62+
]
63+
}
64+
```
65+
66+
```bash
67+
rustmotion render -f scenario.json --props words.json
68+
```
69+
70+
## Caption modes
71+
72+
| Mode | Behavior |
73+
|---|---|
74+
| `highlight` (default) | All words visible, active word takes `active_color` |
75+
| `karaoke` | Same as highlight (karaoke-style progression) |
76+
| `word_by_word` | Only the active word, centered, no pill |
77+
| `word_pop` | TikTok style: only the active word, spring scale-in, pill background |
78+
| `karaoke_pop` | Full line visible + active word scales 1.15x with `active_color` and pill |
79+
80+
- `pill_color` (word_pop / karaoke_pop): pill background behind the active
81+
word. Defaults to black at 70% (`#000000B3`).
82+
- `max_width`: wraps the line in highlight/karaoke/karaoke_pop modes — always
83+
set it on vertical formats to avoid viewport overflow.
84+
- The caption paints from its baseline: position it with enough headroom
85+
(font-size + pill padding above the anchor point).

0 commit comments

Comments
 (0)