diff --git a/CHANGELOG.md b/CHANGELOG.md index f21b763..3e4a908 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,15 @@ ## Unreleased +**`bin/vh pace`: fixes from the review of #72** +- Why: an independent review of #72 (merged before its findings were in) found that an ASR boundary off by more than about 0.1 s could drop syllables or move one behind a pause, a crash when a line has no audio, a missing key found only after the take was written, and some smaller gaps. A second independent review, of this PR, found that the breath rule had no loudness test, that a line with no audio added silence and got zero-width times, that swapped spans repeated audio, and that CI could not catch a revert of the word-time alignment; those are fixed here too. +- Line edges: a line's voice is everything voiced between the silences that part it from its neighbours, each found within 0.2 s of the ASR boundary (at least 0.12 s long when there is one; shorter dips are between syllables); the ASR start and end only locate those silences. Before, on the synthetic take, starts 0.2 s late or ends 0.3 s early lost syllables in every trial, and a boundary 0.1 s inside a syllable moved it behind a 0.39 s pause; now nothing is lost in 40 trials each (a start or an end off by up to 0.5 s either way), and a boundary 0.05–0.2 s inside a syllable leaves it where it was. A whole boundary shifted 0.4 s or more one way can still put a few syllables in the neighbouring line (nothing is lost). A blip at a line's edge under 0.12 s, set off by 0.06 s of quiet and 15 dB under the line's speech (one per side: a breath), stays in the audio but is not counted as speech in the pace or the pauses; a line opening with three short loud syllables measures 5.17 against a true 5.07 (6.36 with a length-only rule). Timelines whose spans go back in time (a hand edit) are refused: cutting at their silences repeated up to 3 s of speech. +- Word times follow the audio: `atempo` stretches a line unevenly (pauses and speech by different amounts), so the loudness envelopes before and after are aligned (dynamic time warping within ±0.15 s of the even stretch). Mapped word starts on the synthetic take are within 8 ms of the real onsets (48 kHz; 27 ms at 22.05 kHz in a line slowed to ×0.92), against 40–60 ms with the average stretch; a line's first word starts and its last word ends where the speech does. On the 2-minute take, mapped line starts sit on the voice onsets (median 0, max 0.03 s). +- A line nothing was heard for (missing from the audio, or under the voice threshold, like a whisper) is named and left alone: its stretch of the take is copied as it is, unpaced, and the line gets a 0.1 s span with its words spread across it. Before, a skipped last line crashed, a skipped middle line got zero-width times and an extra clamped pause, and a whispered line was replaced by silence. A "……" line keeps the pause it stands for, with that stretch's own audio. The analysis hop is a whole number of samples at every rate (at 22.05 kHz it drifted 0.23%), and the alignment keeps only its band (a 120 s line took 1.2 GB, now 0.23 GB). +- `--align gemini` without `GEMINI_API_KEY`, or `--align whisper` without mlx-whisper, stops before anything is written (it wrote the take and skipped the captions). `--restore --dry-run` is refused (it restored). `--ref` on a line without words is an error instead of a ZeroDivisionError. The language may follow the flags. A missing `script.txt`, or line ids not in it, is said (those lines count as one block, and their pause tags are not known). A failed first transcription exits with a message instead of a traceback. The `--restore` refusal no longer suggests deleting the `.raw` files, which may be the only copy of the original. +- A take without word times is transcribed once and kept in `audio/pace..asr.json` under its sha256, so the `.raw` timeline stays byte-identical (it was rewritten with the transcription); `--dry-run` transcribes but writes nothing. +- tools/ci.sh: the synthetic take is 22.05 kHz with a "……" line, and mapped words must be within 35 ms of their syllables (27 ms now, 61 ms without the alignment). New cases: ASR starts 0.2 s late and ends 0.3 s early, a line break 0.1 s inside a syllable, the last line and a middle line with no audio, swapped spans, a quiet breath and three short loud syllables, word times only in the cache (and none at all), no `script.txt`, `--target` / `--extra` / the language after the flags, `--restore --dry-run`, `--align gemini` without a key, `--ref` on a line without words, and `voiceover.wav` / `timeline.json` restored too. Run against #72's `pace.py`, and against this PR's first commit, they fail where the two reviews said. + **`bin/vh pace`: bring the pace of the lines in one narration take closer, without synthesizing again** - Why: a one-request Gemini take ran audibly fast and slow from line to line, and a project script evened it out on the take itself (the steps went into playbook/04 in #70). The maintainer asked for a command, and for it to stay gentle ("别太严格这个语速"): it is optional, moves each line part of the way, leaves the lines that are already close, and its numbers are a reference for the ear. - Per line: lines split at the silence nearest the ASR boundary (an aligner had put one line's end and the next one's start at the same instant, inside the first one's last syllable), then cut points refined from the waveform (an ASR start placed late by a misheard first syllable no longer clips it); pauses inside a line longer than `--pause` 0.26 s shortened to it, unless the script asked for them (``, ``, `<#s#>`); pace = spoken units per voiced second (CJK characters with digits as they are read, or English syllables); tempo `(reference / pace) ^ --strength` (0.8) within `--clamp` (0.92–1.18), through ffmpeg `atempo`; lines within `--tolerance` (5%) of the reference, or under 4 units or 0.5 s, keep their pace. The reference is the median of the lines, `--ref id` or `--ref first-last` (a passage a person approved), or `--target N`. diff --git a/playbook/04-audio.md b/playbook/04-audio.md index fa803ab..a126c44 100644 --- a/playbook/04-audio.md +++ b/playbook/04-audio.md @@ -219,7 +219,7 @@ bin/vh qa mix $A/stems --beats $A/music.beats.json # 只看混音报告（词 **语速的依据和用法**：表里是句内语速，不算句间停顿。普通话正常朗读、去掉 150 ms 以上的停顿后约 5.2 字/秒：七种语言的朗读语料里，汉语是 5.18 音节/秒，一个汉字一个音节（Pellegrino, Coupé & Marsico 2011，*Language* 87(3)，[doi:10.1353/lan.2011.0057](https://doi.org/10.1353/lan.2011.0057)）。电话闲聊按总时长算是每分钟 228–247 字，约 3.8–4.1 字/秒（Yuan, Liberman & Cieri 2006，Interspeech，[doi:10.21437/interspeech.2006-204](https://doi.org/10.21437/interspeech.2006-204)）。知识科普的 4.5–5.5 围着正常朗读，原理讲解慢一档。倒推稿子字数时乘 0.85，留出句间停顿【综合】：45 s 的知识短视频约 170–210 字，3 分钟的讲解约 540–690 字。TTS 的实际语速看音色：本节第 4 小节比较 `duck_ratio` 用的那段测试旁白（0.25 s 间距，本地 Qwen3-TTS 0.6B 的 Serena，7 句 107 字）句内约 4.1 字/秒。所以字数只用来起稿，时长以 `timeline.json` 的实测为准（硬规则 2）。 **一条 take 里快慢差得明显时，可以逐句拉近**：整稿一次合成、不写逐句指示，语气能统一，句与句的语速却可能差不少。句子之间本来就该有快慢，人听着不别扭就不用动；听得出忽快忽慢时，不用重合成，用 `bin/vh pace` 在这条 take 上逐句处理。先加 `--dry-run` 看每句的语速和要拉的倍数，再写： -1. 起止按波形定：从对齐给的句首往回走，走到声音静下来为止（ASR 给的句首不一定准，第一个字听错时常常偏晚，照着剪会切掉它），句尾同样往后走； +1. 起止按波形定：ASR 给的边界只用来找两句之间的那段静音，一句的人声就是两段静音之间有声的部分（ASR 给的句首句尾不一定准，第一个字听错时常常偏晚，照着剪会切掉它）； 2. 句内过长的停顿压到约 0.25 s，切口两侧各淡几毫秒；稿子里写了停顿标签的句子不压； 3. 每句用 `atempo`（不变调）往同一个语速拉近一些，不必拉齐。参照默认取各句的中位数；人听过、认可了某一段（比如开头），用 `--ref` 指定它。只校正一部分：倍数取（参照 ÷ 这句的语速）的 0.8 次方，限在 0.92–1.18，差得不多的句子不动，自然的起伏留着； 4. 句间停顿用 take 自己的，只把太短、太长的按句尾标点和段落收进一个范围，个别地方用 `--extra` 加减；拼接别用 `anullsrc` 加 `concat`（见上文"旁白晚一点进来"）。写回 `voiceover..wav` 和 `timeline..json`，逐字时间跟着换算过去，原件留成 `.raw`，字幕跟着重做。 diff --git a/tools/audio/README.md b/tools/audio/README.md index bcd259d..19ea23d 100644 --- a/tools/audio/README.md +++ b/tools/audio/README.md @@ -91,16 +91,17 @@ bin/vh pace projects/

--ref o1-o2 # 参照人听过、认可的几句（ bin/vh pace projects/

--restore # 放回原来的 take ``` -- **输入**：`voiceover..wav` 和带 `words` 的 `timeline..json`（`--align gemini` 或 `minimax` 给的）；没有逐字时间的，加 `--align gemini|whisper` 先把原 take 转写一遍。`script.txt` 提供段落（空行）和停顿标签。卡拍（`--beats`）的 timeline 和双人对话不处理：重排会让句子离开拍点，两个声音本来就各有各的语速。 -- **每句**：先在 ASR 给的句间边界附近找到两句之间的那段静音，在那里分开（对齐有时把上一句的结尾和下一句的开头放在同一个时刻，还落在上一句最后一个字里）；再按波形定起止，从对齐给的句首往回走到声音静下来为止（ASR 把句首放晚了也不会切掉第一个字），句尾同样往后走。句内长于 `--pause`（0.26 s）的停顿压到 0.26 s，切口两侧各淡 6 ms；稿子里这句写了 ``、`` 或 `<#秒#>` 的，停顿是要求过的，保留。语速 = 字数 ÷ 去掉这些多余停顿后的发声时长；数字按念法算（75 是三个字，1953年是五个），英文按音节。倍数 = （参照 ÷ 这句的语速）的 `--strength` 次方（默认 0.8），限在 `--clamp`（0.92–1.18），用 `atempo` 拉，不变调。和参照差不到 5%（`--tolerance`）的句子、不到 4 个字或不到 0.5 s 的短句不动。 -- **句间停顿**：用 take 自己的（上一句人声结束到下一句人声开始），只把太短、太长的收进按句尾定的范围：逗号或没有标点 0.12–0.40 s，句号 0.25–0.70 s，一段的最后一句 0.45–1.20 s。`--extra id=秒` 给某句之后加（负数是减）。第一句之前（`--lead`）和最后一句之后的部分原样留着。拼接在 numpy 里做，不经过 `anullsrc` 和 `concat`（见下文"手动的响度和合成命令"）。 -- **写回**：原件留成 `voiceover..raw.wav` 和 `timeline..raw.json`。再跑一次总是从原件出发，参数不会叠加；`bin/vh tts` 重新合成后，新的 take 就是原件（timeline 的 `pace` 里记着写出那个文件的 sha256，用来分辨）。`timeline.json` 和 `voiceover.wav` 是这个语言的拷贝时一起更新。`pace` 里还有参照、参数和每句的语速、倍数、处理后的语速；各句的 `file` 去掉了（`vo//` 里是原 take 的切片）。最后用默认参数跑 `bin/vh captions

`；原来的字幕折得更窄（竖屏的 `zh-max 11`）时会提示用原来的宽度再跑一次。 -- **逐字时间**：默认 `--align map`，把每个字的时间按切口、压掉的停顿和倍数换算到新 take 上，句首、句尾对到波形上的人声。离线、不花钱、每次结果一样。`--align gemini` 把新 take 整条转写一遍（9 分钟以内一次调用），`--align whisper` 用本地 mlx-whisper（只能在 Apple Silicon 上，模型用 `WHISPER_MODEL`，默认 `mlx-community/whisper-large-v3-turbo`）；两种都交给 `tts.py` 的 `align_lines`，更新 `asr` 的相似度和标记，并报出和换算的时间差了多少。转写失败（比如当天额度用完）时，新 take 照样写好、保留换算的时间，命令以 1 退出，之后同一条命令再跑一次就行。 +- **输入**：`voiceover..wav` 和带 `words` 的 `timeline..json`（`--align gemini` 或 `minimax` 给的）；没有逐字时间的，加 `--align gemini|whisper` 先把原 take 转写一遍，结果按这条 take 的 sha256 存在 `pace..asr.json` 里，再跑不用重转（`--dry-run` 也会转写，但什么都不写）。`script.txt` 提供段落（空行）和停顿标签；没有它、或者有的句子 id 在里面找不到时，命令会说出来：这些句子按同一段算（不算段落末），停顿标签也不算。卡拍（`--beats`）的 timeline 和双人对话不处理：重排会让句子离开拍点，两个声音本来就各有各的语速。 +- **每句**：ASR 给的句首句尾只用来找两句之间的那段静音：在边界前后 0.2 s 里找离它最近、至少 0.12 s 长的一段静音（没有就找短的，字与字之间的小空隙比它短），在那里分开。一句的人声就是两段静音之间所有有声的部分，所以 ASR 把句首放晚了、句尾放早了、两句的边界落在一个字里，都不会切掉字（对齐有时把上一句的结尾和下一句的开头放在同一个时刻，还落在上一句最后一个字里）。句子边上一小段有声的（不到 0.12 s，和正文隔着 0.06 s 以上的静音，又比这句的人声低 15 dB 以上），当成换气，每边最多一段：跟着音频留着，但不算进语速和停顿；响的短字不算换气。一句里什么都没听到的（音频里缺了这句，或者轻到低于人声阈值），不处理：它那一段 take 原样留着，这句给 0.1 s 的时长，命令会点名，要人去听。ASR 的句子顺序乱了（后一句的开头早于前一句，多半是手改过 timeline）时拒绝处理，否则会重复一段音频。两句连着念、中间不到 0.12 s，ASR 的边界又差出一个字，或者整条边界往同一边偏 0.4 s 以上时，个别字可能分到相邻那句（音频不丢，只是跟着那句的倍数走）。句内长于 `--pause`（0.26 s）的停顿压到 0.26 s，切口两侧各淡 6 ms；稿子里这句写了 ``、`` 或 `<#秒#>` 的，停顿是要求过的，保留。语速 = 字数 ÷ 去掉这些多余停顿后的发声时长；数字按念法算（75 是三个字，1953年是五个），英文按音节。倍数 = （参照 ÷ 这句的语速）的 `--strength` 次方（默认 0.8），限在 `--clamp`（0.92–1.18），用 `atempo` 拉，不变调。和参照差不到 5%（`--tolerance`）的句子、不到 4 个字或不到 0.5 s 的短句不动。 +- **句间停顿**：用 take 自己的（上一句说完到下一句开口），只把太短、太长的收进按句尾定的范围：逗号或没有标点 0.12–0.40 s，句号 0.25–0.70 s，一段的最后一句 0.45–1.20 s。中间夹着一句没有字的（"……"）时，这段停顿就是它要的，原样保留。`--extra id=秒` 给某句之后加（负数是减）。第一句之前（`--lead`）和最后一句之后的部分原样留着。拼接在 numpy 里做，不经过 `anullsrc` 和 `concat`（见下文"手动的响度和合成命令"）。 +- **写回**：原件留成 `voiceover..raw.wav` 和 `timeline..raw.json`。再跑一次总是从原件出发，参数不会叠加；`bin/vh tts` 重新合成后，新的 take 就是原件（timeline 的 `pace` 里记着写出那个文件的 sha256，用来分辨）。`timeline.json` 和 `voiceover.wav` 是这个语言的拷贝时一起更新。`pace` 里还有参照、参数和每句的语速、倍数、处理后的语速；各句的 `file` 去掉了（`vo//` 里是原 take 的切片）。写回的是单声道（`bin/vh tts` 的输出本来就是）。处理后的 timeline 别手改：再跑一次会从 `.raw` 重新生成，手改的会丢。最后用默认参数跑 `bin/vh captions

`；原来的字幕折得更窄（竖屏的 `zh-max 11`）时会提示用原来的宽度再跑一次。`--restore` 只在当前的 take 正是 `pace` 写出的那条时才放回去；不是的话（又跑过 `bin/vh tts`，或者文件被手动改过）什么都不动，`.raw` 可能是原件仅有的一份，先留着。`--restore` 和 `--dry-run` 不能一起用。 +- **逐字时间**：默认 `--align map`，把每个字的时间换算到新 take 上：先过切口和压掉的停顿，再过 `atempo` 的伸缩。`atempo` 在一句之内伸缩得不均匀（停顿和人声拉得不一样多），所以按处理前后的响度包络对齐（动态时间规整，限在平均伸缩的 ±0.15 s 以内），字跟着实际的音频走；每句第一个字的开头、最后一个字的结尾对到这句人声的起止。离线、不花钱、每次结果一样。`--align gemini` 把新 take 整条转写一遍（9 分钟以内一次调用），`--align whisper` 用本地 mlx-whisper（只能在 Apple Silicon 上，模型用 `WHISPER_MODEL`，默认 `mlx-community/whisper-large-v3-turbo`）；两种都交给 `tts.py` 的 `align_lines`，更新 `asr` 的相似度和标记，并报出和换算的时间差了多少。缺 key 或没装 mlx-whisper 时，在写任何文件之前就停下；转写失败（比如当天额度用完）时，新 take 照样写好、保留换算的时间，命令以 1 退出，之后同一条命令再跑一次就行。 实测（2026-10-07）： -- 合成的测试 take（6 句、42 个已知起点的"音节"，快的 7.3、慢的 4.1 个/秒）：音节一个不少；换算的逐字起点和真实起点相差中位数 2 ms，最多 40 ms（`atempo` 在一句之内伸缩得不完全均匀），在 30 fps 的一帧以内； -- 一条 2 分钟、27 句的 Gemini 整稿：句内语速 5.18–7.38 字/秒（标准差 0.50）拉到 5.88–6.83（0.21），13 句的速度没动，总长 129.5 s → 121.4 s。这里的量法只算人声的起止之间、数字按念法算，所以和 `playbook/04` 里手工那次的数字不一样。换算出的句首都在波形上人声起点的 0.04 s 以内（多半早 0.03 s，是句首留的余量）；同一条用 Gemini 重转写，句首和换算的差中位数 0.03 s、最多 0.13 s；用 whisper 重转写，句首中位数早 0.19 s，最多早 0.27 s。所以默认用换算，重转写主要用来查有没有字丢了、念错了。Gemini 那次把第一句里几个字的时间排到了后面两句，这两句被标出来，音频本身没有问题（whisper 听这两句是 0.94 和 1.00）； -- 同一条用 `bin/vh qa` 查 click：原 take 57 处，处理后 53 处，切口没有带进新的。 +- 合成的测试 take（6 句、42 个已知起点的"音节"，快的 7.3、慢的 4.1 个/秒）：音节一个不少；换算的逐字起点和真实起点相差中位数 2 ms、最多 8 ms（48 kHz；CI 用的 22.05 kHz 版本里最多 27 ms，不做包络对齐时 61 ms，CI 的上限是 35 ms）。一句 120 s 的长句，对齐用 0.23 GB 内存； +- 同一条 take 把 ASR 弄偏：每句句首晚或早 0.1–0.5 s、句尾早或晚 0.1–0.5 s（每种 40 次），句间边界放进上一句最后一个字里 0.05–0.2 s，音节都一个不少，也没有哪个字被挪到停顿后面（从 ASR 句首往回走的第一版，句首晚 0.2 s 或句尾早 0.3 s 时每次都丢字）。整条边界往同一边偏 0.4 s 以上时，42 个字里有 3–19 个分到了相邻那句；一句开头是三个 90 ms 的短字时，语速量得准（5.17，实际 5.07；只看长短、不看音量的换气规则会量成 6.36）； +- 一条 2 分钟、27 句的 Gemini 整稿：句内语速 5.18–7.38 字/秒（标准差 0.50）拉到 5.88–6.83（0.20），13 句的速度没动，总长 129.5 s → 121.5 s，处理 1.4 s。这里的量法只算人声的起止之间、数字按念法算，所以和 `playbook/04` 里手工那次的数字不一样。换算出的句首和波形上人声的起点差中位数 0、最多 0.03 s；用 whisper 重转写，句首中位数早 0.20 s，最多早 0.26 s；用 Gemini 重转写（前一版上量的），句首和换算的差中位数 0.03 s、最多 0.13 s。所以默认用换算，重转写主要用来查有没有字丢了、念错了。Gemini 那次把第一句里几个字的时间排到了后面两句，这两句被标出来，音频本身没有问题（whisper 听这两句是 0.94 和 1.00）； +- 同一条用 `bin/vh qa` 查 click：原 take 57 处，处理后 54 处，切口没有带进新的。 ### 音色（`bin/vh voices`） diff --git a/tools/audio/pace.py b/tools/audio/pace.py index 54073b1..7f0d21f 100644 --- a/tools/audio/pace.py +++ b/tools/audio/pace.py @@ -14,35 +14,43 @@ transcribes the take first. A timeline on a beat grid (--beats) or a dialogue is refused: re-pacing would move lines off the grid, and two voices keep their own pace. Per line: - 1. cut points from the waveform: two lines split at the silence nearest their ASR boundary (an aligner can put one - line's end and the next one's start at the same instant, inside the first one's last syllable); from the aligned - start, step back while the signal is within 32 dB of the take's speech level (an ASR start placed late by a - misheard first syllable would clip it), and forward from the end; + 1. cut points from the waveform: two lines split at the silence nearest their ASR boundary (within 0.2 s of it, at + least 0.12 s long when there is one: shorter dips are between syllables). A line's voice is everything voiced + (within 32 dB of the take's speech level) between the silences on either side, so an ASR start placed late by a + misheard first syllable, an end placed early, or a boundary inside a syllable cannot cut anything off. A blip under + 0.12 s at a line's edge, set off by 0.06 s of quiet and 15 dB under the line's speech, is a breath (one per side): + kept in the audio, not counted as speech. A line nothing was heard for is named and left alone: its stretch of the + take is copied as it is. Spans that go back in time (a hand-edited timeline) are refused; 2. silences inside the line longer than --pause (0.26 s) are shortened to it, 6 ms fades on both sides of each cut. A line whose script has , or a <#s#> mark keeps its pauses (they were asked for); - 3. pace = spoken units / voiced seconds after step 2: CJK characters (digits read out: 75 = 七十五, 3; 1953年 = 4), or + 3. pace = spoken units / voiced seconds after step 2: CJK characters (digits read out: 75 = 七十五, 3; 1953年 = 5), or syllables in English. Reference: the median of the lines (default), --ref the pooled pace of lines a person approved (--ref hook, --ref s1-s3), or --target N. Each line moves by tempo (ffmpeg atempo, pitch kept) x = (reference / pace) ^ --strength, within --clamp; lines within --tolerance of it, and lines under 4 units or 0.5 s, are left as they are; 4. pauses between lines are the take's own, measured voice to voice, kept inside a range by the line's end: comma (or no punctuation) 0.12–0.40 s, full stop 0.25–0.70 s, end of a block 0.45–1.20 s; --extra id=s adds - (or, negative, takes away) after a line. The silence before the first line (--lead) and after the last is kept. + (or, negative, takes away) after a line. A pause with a "……" line (no words) in it is the pause that line asks for: + kept as it is. The silence before the first line (--lead) and after the last is kept. Lines are joined in numpy, never with anullsrc + concat (playbook/04-audio.md, "旁白晚一点进来"). Output voiceover..wav and timeline..json written back (timeline.json / voiceover.wav too when they are - copies of this language), the originals kept as voiceover..raw.wav / timeline..raw.json, then + copies of this language; mono, like bin/vh tts), the originals kept as voiceover..raw.wav / .raw.json, then bin/vh captions . A second run starts again from the .raw take, so settings never compound; after a new bin/vh tts run the new take is the original (the timeline's "pace" block records a sha256 of the - paced file to tell them apart). --restore puts the original back and removes the .raw files. - Word times: --align map (default) carries each word through the cuts and the tempo change: offline, free, - deterministic. --align gemini transcribes the new take (one call per ≤ 9 min, GEMINI_API_KEY) and --align whisper - with a local mlx-whisper (Apple Silicon; WHISPER_MODEL, default mlx-community/whisper-large-v3-turbo); both go + paced file to tell them apart). --restore puts the original back and removes the .raw files, only when the take + is the one bin/vh pace wrote (otherwise the .raw files may be the only copy of the original: nothing is touched). + Word times: --align map (default) carries each word through the cuts and the tempo change, which atempo does not + apply evenly, so the loudness envelopes before and after are aligned and the words follow the audio; offline, + free, deterministic. A take without word times is transcribed once (--align gemini|whisper) and kept in + audio/pace..asr.json under its sha256. --align gemini transcribes the new take (one call per ≤ 9 min, + GEMINI_API_KEY; checked before anything is written) and --align whisper with a local mlx-whisper (Apple Silicon; + WHISPER_MODEL, default mlx-community/whisper-large-v3-turbo); both go through tts.py's align_lines like bin/vh tts --align, refresh asr {similarity, flag} and print how far the mapped times were (a failed transcription keeps the mapped times, exit 1). A whisper "1953" against a script's 一九五三 is spread over the heard word. The timeline's "pace" block: reference, settings, and per line its pace before, the tempo, and after. """ -import argparse, hashlib, json, math, os, re, shutil, subprocess, sys, tempfile +import argparse, hashlib, importlib.util, json, math, os, re, shutil, subprocess, sys, tempfile from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) @@ -59,6 +67,10 @@ EDGE_FADE, CUT_FADE = 0.010, 0.006 MIN_UNITS, MIN_VOICED = 4, 0.5 # shorter lines keep their pace: too few units to measure one SPLIT_NEAR = 0.2 # look this far around the ASR boundary between two lines for the silence between them +GAP_MIN = 0.12 # … a silence at least this long when there is one (shorter dips are between syllables) +BREATH, BREATH_GAP, BREATH_DB = 0.12, 0.06, 15.0 # a voiced blip this short at a line's edge, set off by this much quiet and + # this many dB under the line's speech: a breath, not speech (one per side) +MIN_SPAN = 0.1 # a line nothing was heard for still gets this much (no zero-width cue; tts.align_lines does the same) GAPS = {"comma": (0.12, 0.40), "stop": (0.25, 0.70), "block": (0.45, 1.20)} # voice-to-voice silence after a line (s) ASR_PIECE = 540.0 # longest piece per gemini transcription (16 kHz inline audio: ≈ 10 min fits 20 MB) STOPS = tuple("。！？!?….") @@ -138,6 +150,37 @@ def atempo(x, sr, f): "-f", "f32le", "pipe:1"], input=x.astype(" lo else INF, + at(i - 1, j - 1) if i and j else INF) + D[i, j - lo] = cost[j - lo] + best + i, j, path = n - 1, m - 1, {} + while i > 0 or j > 0: + path.setdefault(i, []).append(j) + steps = [(at(i - 1, j - 1), i - 1, j - 1)] if i and j else [] + steps += [(at(i - 1, j), i - 1, j)] if i else [] + steps += [(at(i, j - 1), i, j - 1)] if j else [] + _, i, j = min(steps) + path.setdefault(0, []).append(0) + out = np.array([np.mean(path[k]) for k in range(n)], dtype=float) + return np.maximum.accumulate(out) + def sha256(p): h = hashlib.sha256() with open(p, "rb") as f: @@ -184,91 +227,105 @@ def align_take(segs, wav, engine, lang, tmp, dur): # ---------- the edit ---------- def gap_kind(seg, nxt, block): - if nxt is not None and block.get(nxt["id"]) != block.get(seg["id"]): - return "block" + if nxt is not None and seg["id"] in block and nxt["id"] in block and block[nxt["id"]] != block[seg["id"]]: + return "block" # a line the script does not have: no block break claimed return "stop" if seg["text"].rstrip(CLOSERS).endswith(STOPS) else "comma" def plan(x, sr, segs, lang, block, asked, a): - """Cut points, inner pauses, pace and tempo of every line (nothing written).""" - n = len(x) // int(HOP * sr) - env = x[: n * int(HOP * sr)].reshape(n, -1) - db = 10 * np.log10(np.mean(env ** 2, axis=1) + 1e-18) + """Cut points, inner pauses, pace and tempo of every line with words (nothing written). + The voice of a line is everything voiced between the silences that part it from its neighbours, each found near the + ASR boundary between the two; the ASR start and end are not cut points, so an ASR that put them late or early (or a + line's end and the next one's start at the same instant) cannot cut a syllable off.""" + hop = max(1, round(HOP * sr)); H = hop / sr; n = len(x) // hop + if n < 2: + sys.exit("the take is too short to measure") + db = 10 * np.log10(np.mean(x[: n * hop].reshape(n, hop) ** 2, axis=1) + 1e-18) voiced = db > max(FLOOR_DB, float(np.percentile(db, 95)) - BELOW) - frame = lambda t: int(t / HOP + 1e-6) - loud = lambda t: 0 <= frame(t) < n and bool(voiced[frame(t)]) + fr = lambda t: min(max(int(t / H + 1e-6), 0), n) dur = len(x) / sr - # where one line ends and the next begins: the silence nearest the ASR boundary (an aligner can put a line's end and the - # next one's start at the same instant, at the last voiced frame of the first); lines that touch split at the boundary - splits = [] - for p, q in zip(segs, segs[1:]): + U = [units(s["text"], lang) for s in segs] + spoken = [i for i in range(len(segs)) if U[i]] + if not spoken: + sys.exit("no line with words to pace") + + def split(p, q): + """The silence between lines p and q: the quiet run nearest their ASR boundary, at least GAP_MIN long when there + is one (shorter dips are between syllables); lines that touch split at the boundary itself.""" m = (p["end"] + q["start"]) / 2 - k, stop, runs = max(frame(min(p["end"], q["start"]) - SPLIT_NEAR), frame(p["start"])), min(frame(max(p["end"], q["start"]) + SPLIT_NEAR), frame(q["end"]), n), [] + k, stop = max(fr(min(p["end"], q["start"]) - SPLIT_NEAR), fr(p["start"])), fr(max(p["end"], q["start"]) + SPLIT_NEAR) + runs = [] while k < stop: if voiced[k]: k += 1; continue - j = k - while j < n and not voiced[j]: - j += 1 - k0 = k + k0, j = k, k while k0 > 0 and not voiced[k0 - 1]: k0 -= 1 - if j - k0 >= 3: - runs.append((k0 * HOP, j * HOP)) - k = j - splits.append(min(runs, key=lambda r: max(r[0] - m, m - r[1], 0.0)) if runs else (m, m)) - lines, lo = [], 0.0 - for i, s in enumerate(segs): - wlo = splits[i - 1][1] if i else 0.0 # the voice of this line lies between the silences around it - hi = splits[i][0] if i + 1 < len(segs) else dur - hi = max(hi, wlo + HOP) - u = units(s["text"], lang) - a0 = min(max(s["start"], wlo), hi) - if not u: # "……": nothing to measure, the span as it is - v0, v1 = a0, min(max(s["end"], a0), hi) - else: - v0 = a0 - if loud(v0): - while v0 - HOP >= wlo and loud(v0 - HOP): - v0 -= HOP - else: # an early ASR start: move on to the voice - while v0 + HOP < min(s["end"], hi) and not loud(v0): - v0 += HOP - v1 = min(max(s["end"], v0 + HOP), hi) - if loud(v1): - while v1 + HOP <= hi and loud(v1): - v1 += HOP - else: - while v1 - HOP > v0 and not loud(v1 - HOP): - v1 -= HOP - nxt = sum(splits[i]) / 2 if i + 1 < len(segs) else dur # margins reach into the silence, up to its middle - c0, c1 = max(lo, v0 - HEAD * bool(u)), min(max(nxt, v1), v1 + TAIL * bool(u)) - # quiet runs strictly inside the voiced extent: cut the middle of the long ones (kept when the script asks for them) - keep, runs, k, end, seen = np.ones(int(round(c1 * sr)) - int(round(c0 * sr)), bool), [], frame(v0), min(frame(v1), n), False - while u and k < end: - if voiced[k]: - seen = True; k += 1; continue - j = k - while j < end and not voiced[j]: + while j < n and not voiced[j]: j += 1 - if seen and j < end and (j - k) * HOP > a.pause > 0: - runs.append((k * HOP, j * HOP)) - k = j + runs.append((k0 * H, j * H)); k = j + near = lambda r: max(r[0] - m, m - r[1], 0.0) + for floor in (GAP_MIN, 3 * H): + c = [r for r in runs if r[1] - r[0] >= floor - 1e-9] + if c: + return min(c, key=near) + return (min(max(m, 0.0), dur),) * 2 + + back = [j for i, j in zip(spoken, spoken[1:]) if segs[j]["start"] < segs[i]["start"] - 0.05] + sp = [split(segs[i], segs[j]) for i, j in zip(spoken, spoken[1:])] + back += [spoken[k + 1] for k in range(1, len(sp)) if sp[k][0] < sp[k - 1][0] - 1e-6] + if back: # spans out of order: cutting at their silences would repeat audio + sys.exit(f"the timeline's lines are out of order at @{segs[back[0]]['id']} (its ASR span starts before the line before it):" + f" re-align the take (bin/vh tts … --align gemini --resume) instead of editing the timeline by hand") + lines = [] + for k, i in enumerate(spoken): + s, u = segs[i], U[i] + wlo, whi = sp[k - 1][1] if k else 0.0, sp[k][0] if k + 1 < len(spoken) else dur + mlo, mhi = sum(sp[k - 1]) / 2 if k else 0.0, sum(sp[k]) / 2 if k + 1 < len(spoken) else dur # margins stop mid-silence + f0 = fr(wlo); idx = np.flatnonzero(voiced[f0:max(fr(whi), f0)]) + if len(idx): + v0, v1 = (f0 + idx[0]) * H, min((f0 + idx[-1] + 1) * H, dur) + c0, c1 = max(mlo, v0 - HEAD, 0.0), min(mhi, v1 + TAIL, dur) + vr = np.split(idx, np.flatnonzero(np.diff(idx) > 1) + 1) # voiced runs; the speech is between the breaths at its edges + level = float(np.percentile(db[f0 + idx], 90)) # the line's speech level + breath = lambda r, gap: len(r) * H < BREATH and gap * H >= BREATH_GAP and db[f0 + r].max() < level - BREATH_DB + if len(vr) > 1 and breath(vr[0], vr[1][0] - vr[0][-1] - 1): + vr = vr[1:] + if len(vr) > 1 and breath(vr[-1], vr[-1][0] - vr[-2][-1] - 1): + vr = vr[:-1] + s0, s1 = (f0 + vr[0][0]) * H, min((f0 + vr[-1][-1] + 1) * H, dur) + else: # nothing heard for this line: a point, untouched + v0 = v1 = c0 = c1 = s0 = s1 = min(max(s["start"], wlo), max(whi, wlo), dur) + sa, sb = int(round(c0 * sr)), min(len(x), int(round(c1 * sr))) + keep, runs = np.ones(max(sb - sa, 0), bool), [] + q = voiced[fr(s0):fr(s1)] if len(idx) else voiced[:0] # the speech, first to last voiced frame + j = 0 + while j < len(q): # quiet runs inside the line (q starts and ends voiced) + if q[j]: + j += 1; continue + e = j + while e < len(q) and not q[e]: + e += 1 + if (e - j) * H > a.pause > 0: + runs.append((s0 + j * H, s0 + e * H)) + j = e excess = sum(e - b - a.pause for b, e in runs) - if runs and not asked.get(s["id"]): + if runs and not asked.get(s["id"]): # cut the middle of the long ones (kept when the script asks for them) for b, e in runs: - keep[int(round((b + a.pause / 2 - c0) * sr)): int(round((e - a.pause / 2 - c0) * sr))] = False - voiced_s = max(v1 - v0 - excess, HOP) - lines.append({"seg": s, "u": u, "c0": c0, "c1": c1, "v0": v0, "v1": v1, "keep": keep, "runs": runs, - "rate": u / voiced_s if u else None, "voiced": voiced_s, "kept_pauses": bool(runs and asked.get(s["id"]))}) - lo = c1 + keep[max(0, int(round((b + a.pause / 2) * sr)) - sa): max(0, int(round((e - a.pause / 2) * sr)) - sa)] = False + voiced_s = max(s1 - s0 - excess, H) + lines.append({"i": i, "seg": s, "u": u, "c0": c0, "c1": c1, "v0": v0, "v1": v1, "s0": s0, "s1": s1, "sa": sa, "sb": sb, "keep": keep, + "runs": runs, "heard": bool(len(idx)), "rate": u / voiced_s if len(idx) else None, "voiced": voiced_s, + "kept_pauses": bool(runs and asked.get(s["id"]))}) for L in lines: - L["ok"] = L["u"] >= MIN_UNITS and L["voiced"] >= MIN_VOICED + L["ok"] = L["heard"] and L["u"] >= MIN_UNITS and L["voiced"] >= MIN_VOICED measurable = [L for L in lines if L["ok"]] if a.target: ref, how = a.target, f"--target {a.target:g}" elif a.ref: ids = ref_ids(a.ref, segs) - pool = [L for L in lines if L["seg"]["id"] in ids and L["u"]] + pool = [L for L in lines if L["seg"]["id"] in ids and L["heard"]] + if not pool: + sys.exit(f"--ref {a.ref}: no line there has words that were heard; pick lines with speech") ref, how = sum(L["u"] for L in pool) / sum(L["voiced"] for L in pool), f"--ref {a.ref} ({len(pool)} line(s))" elif measurable: ref, how = float(np.median([L["rate"] for L in measurable])), f"median of {len(measurable)} line(s)" @@ -277,20 +334,23 @@ def plan(x, sr, segs, lang, block, asked, a): for L in lines: f, why = 1.0, "" if not L["ok"]: - why = "short" if L["u"] else "no words" + why = "short" if L["heard"] else "nothing heard" elif ref and abs(math.log(ref / L["rate"])) > math.log(1 + a.tolerance): f = min(a.clamp[1], max(a.clamp[0], (ref / L["rate"]) ** a.strength)) elif ref: why = "close enough" L["f"], L["why"] = f, why - for i, L in enumerate(lines): # voice-to-voice silence after the line, in the take and in the new one - nxt = lines[i + 1] if i + 1 < len(lines) else None - if nxt: - kind = gap_kind(L["seg"], nxt["seg"], block) - g = nxt["v0"] - L["v1"] - lo_k, hi_k = GAPS[kind] - L["gap"], L["kind"] = max(0.04, min(max(g, lo_k), hi_k) + a.extra.get(L["seg"]["id"], 0.0)), kind - L["gap0"] = g + chain = [L for L in lines if L["heard"]] # a line nothing was heard for stays inside the pause around it + if not chain: + sys.exit("nothing was heard for any line with words: is audio/voiceover..wav the take this timeline describes?") + for k, L in enumerate(chain[:-1]): # voice-to-voice silence after the line, in the take and in the new one + nxt = chain[k + 1] + between = [segs[i]["id"] for i in range(L["i"] + 1, nxt["i"])] # "……" lines, unheard lines: that stretch is kept as it is + g = nxt["s0"] - L["s1"] # speech to speech: a breath before a line is part of the pause + kind = "kept" if between else gap_kind(L["seg"], nxt["seg"], block) + lo_k, hi_k = GAPS.get(kind, (g, g)) + extra = sum(a.extra.get(x_, 0.0) for x_ in [L["seg"]["id"]] + between) + L["gap"], L["kind"], L["gap0"] = max(0.04, min(max(g, lo_k), hi_k) + extra), kind, g return lines, ref, how def ref_ids(spec, segs): @@ -304,12 +364,12 @@ def ref_ids(spec, segs): sys.exit(f"--ref {spec}: not a line id or a range of them (first-last) in this timeline: {' '.join(ids)}") def render(x, sr, lines): - """→ (new audio, per line a function: time in the take → time in the new audio); sets each line's real tempo.""" - out, maps = [x[: int(round(lines[0]["c0"] * sr))]], [] + """→ (new audio, a function: time in the take → time in the new audio); sets each line's real tempo and new span.""" + lines = [L for L in lines if L["heard"]] + out = [x[: lines[0]["sa"]]] t = len(out[0]) / sr - for i, L in enumerate(lines): - a, b = int(round(L["c0"] * sr)), int(round(L["c1"] * sr)) - chunk, keep = x[a:b].copy(), L["keep"][: b - a] + for k, L in enumerate(lines): + chunk, keep = x[L["sa"]:L["sb"]].copy(), L["keep"][: L["sb"] - L["sa"]] fz = int(CUT_FADE * sr) for e in np.flatnonzero(np.diff(keep.astype(np.int8))): # fade out before each cut, in after it (length unchanged) if keep[e]: @@ -321,32 +381,68 @@ def render(x, sr, lines): fe = min(int(EDGE_FADE * sr), len(y) // 2) if fe: y[:fe] *= np.linspace(0, 1, fe); y[-fe:] *= np.linspace(1, 0, fe) - ck = np.concatenate([[0], np.cumsum(keep)]) - scale = len(y) / max(len(kept), 1) # the real stretch, so a line's last word ends where its audio does - maps.append(lambda r, ck=ck, a=a, t=t, scale=scale: t + ck[min(max(int(round(r * sr)) - a, 0), len(ck) - 1)] * scale / sr) - L["f_real"] = 1 / scale if len(kept) else 1.0 # atempo's length is off by up to ~1% (silence stretches unevenly) - out.append(y); t += len(y) / sr - if i + 1 < len(lines): # the pause: voice-to-voice gap minus the margins already in both chunks - nxt = lines[i + 1] - margins = (L["c1"] - L["v1"]) / L["f_real"] + (nxt["v0"] - nxt["c0"]) / nxt["f"] - pad = np.zeros(int(round(max(0.0, L["gap"] - margins) * sr))) + L["ck"], L["scale"] = np.concatenate([[0], np.cumsum(keep)]), len(y) / len(kept) if len(kept) else 1.0 + L["warp"] = warp(kept, y, sr) if len(y) != len(kept) else None + L["f_real"] = 1 / L["scale"] if L["scale"] else 1.0 # atempo's length is off by up to ~1% (silence stretches unevenly) + L["n0"] = t; out.append(y); t += len(y) / sr; L["n1"] = t + if k + 1 < len(lines): # the pause: voice-to-voice gap minus the margins already in both chunks + nxt = lines[k + 1] + margins = (L["c1"] - L["s1"]) / L["f_real"] + (nxt["s0"] - nxt["c0"]) / nxt["f"] + n = int(round(max(0.0, L["gap"] - margins) * sr)) + if L["kind"] == "kept": # the take's own audio there (a "……" line, a line too soft to be + pad = x[L["sb"]:nxt["sa"]].copy() # heard), fitted at its end, which is silence before the next line + fz = min(int(CUT_FADE * sr), n // 2) + pad = np.concatenate([pad, np.zeros(max(0, n - len(pad)))])[:n] + if fz and n < nxt["sa"] - L["sb"]: + pad[n - fz:] *= np.linspace(1, 0, fz) + else: + pad = np.zeros(n) out.append(pad); t += len(pad) / sr - out.append(x[int(round(lines[-1]["c1"] * sr)):]) # the take's own tail - return np.concatenate(out), maps - -def paced_segments(lines, maps): - segs = [] - for L, m in zip(lines, maps): - s = {k: v for k, v in L["seg"].items() if k != "file"} # vo//NN.wav are cuts of the original take - if s.get("words"): - s["words"] = [dict(w, start=round(m(w["start"]), 3), end=round(max(m(w["end"]), m(w["start"])), 3)) for w in s["words"]] - w0, w1 = s["words"][0], s["words"][-1] # the voice's own edges where the ASR put a word late (or ended early) - w0["start"] = min(w0["start"], round(m(L["v0"]), 3)); w1["end"] = max(w1["end"], round(m(L["v1"]), 3)) + out.append(x[lines[-1]["sb"]:]) # the take's own tail + + def remap(r): + """Inside a line: through its cuts and tempo; between lines: linearly across the new pause; lead and tail as they are.""" + prev = None + for L in lines: + if r < L["c0"]: + if prev is None: + return min(r, L["n0"]) + w = (r - prev["c1"]) / max(L["c0"] - prev["c1"], 1e-9) + return prev["n1"] + (L["n0"] - prev["n1"]) * min(max(w, 0.0), 1.0) + if r < L["c1"]: + k = min(max(int(round(r * sr)) - L["sa"], 0), len(L["ck"]) - 1) + if L["warp"] is None or len(L["warp"]) < 2: + return L["n0"] + L["ck"][k] * L["scale"] / sr + hop = max(1, round(0.01 * sr)) # through the aligned envelopes, to the sample within the frame + return L["n0"] + float(np.interp(L["ck"][k] / hop, np.arange(len(L["warp"])), L["warp"])) * hop / sr + prev = L + return prev["n1"] + (r - prev["c1"]) + return np.concatenate(out), remap + +def paced_segments(segs, lines, remap): + """Every line of the timeline on the new take: words through remap, kept inside their own line's new span; the first + word starts and the last one ends where the speech does (an ASR puts these edges late or early).""" + by = {L["i"]: L for L in lines} + out = [] + for i, s0 in enumerate(segs): + s, L = {k: v for k, v in s0.items() if k != "file"}, by.get(i) # vo//NN.wav are cuts of the original take + if L and L["heard"] and s.get("words"): + lo, hi = L["n0"], max(L["n1"], L["n0"]) + at = lambda r: round(min(max(remap(r), lo), hi), 3) + s["words"] = [dict(w, start=at(w["start"]), end=max(at(w["end"]), at(w["start"]))) for w in s["words"]] + if L["heard"]: + w0, w1 = s["words"][0], s["words"][-1] + w0["start"] = min(at(L["s0"]), w0["end"]); w1["end"] = max(at(L["s1"]), w1["start"]) s["start"], s["end"] = s["words"][0]["start"], s["words"][-1]["end"] - else: - s["start"], s["end"] = round(m(L["c0"]), 3), round(m(L["c1"]), 3) - segs.append(s) - return segs + else: # no words, or nothing heard: where its span went + s["start"] = round(remap(s0["start"]), 3); s["end"] = round(max(remap(s0["end"]), s["start"]), 3) + if s.get("words"): # nothing heard: after the line before, MIN_SPAN at least, its words + s["start"] = max(s["start"], out[-1]["end"] if out else 0.0) # spread across it + s["end"] = round(max(s["end"], s["start"] + MIN_SPAN), 3); n, d = len(s["words"]), s["end"] - s["start"] + s["words"] = [dict(w, start=round(s["start"] + d * k / n, 3), end=round(s["start"] + d * (k + 1) / n, 3)) + for k, w in enumerate(s["words"])] + out.append(s) + return out # ---------- main ---------- @@ -364,7 +460,9 @@ def main(): ap.add_argument("--align", default="map", choices=["map", "gemini", "whisper"], help="word times of the new take") ap.add_argument("--dry-run", action="store_true", help="measure and print, write nothing") ap.add_argument("--restore", action="store_true", help="put the original take and timeline back, remove the .raw files") - a = ap.parse_args() + a = ap.parse_intermixed_args() # the language may come after the flags too + if a.restore and a.dry_run: + ap.error("--restore puts files back; --dry-run only measures: use one of them") try: a.clamp = tuple(float(v) for v in a.clamp.split(",")) assert len(a.clamp) == 2 and 0.5 <= a.clamp[0] <= 1 <= a.clamp[1] <= 2 @@ -404,8 +502,9 @@ def captions(old): sys.exit(f"--restore: no {rel(raw_wav)} / {rel(raw_tl)} to put back") cur = json.loads(tlp.read_text(encoding="utf-8")) if tlp.exists() else {} if not (wav.exists() and cur.get("pace") and sha256(wav) == cur["pace"].get("sha256")): - sys.exit(f"--restore: {rel(wav)} is not the take bin/vh pace wrote (a new bin/vh tts run?), so the .raw files are from an" - f" older take; nothing changed (delete them by hand if they are not needed)") + sys.exit(f"--restore: {rel(wav)} is not the take bin/vh pace wrote (a new bin/vh tts run, or a file changed by hand)," + f" so nothing was put back. The .raw files may be the only copy of the original take and timeline: keep them" + f" until you know which take you want") shutil.move(str(raw_wav), str(wav)); shutil.move(str(raw_tl), str(tlp)) if copies: shutil.copyfile(tlp, latest); shutil.copyfile(wav, audio / "voiceover.wav") @@ -439,6 +538,11 @@ def captions(old): for s in tts.parse_script(script)[0]: block[s["id"]] = s["_block"] asked[s["id"]] = bool(ASKED_PAUSE.search((s["zh"] if a.lang == "zh" else s["en"]) or s["zh"] or s["en"])) + unknown = [s["id"] for s in segs if s["id"] not in block] + if unknown: # the result changes without it: say so + print(f" note: {'no ' + str(rel(script)) if not block else str(len(unknown)) + ' line(s) not in ' + str(rel(script))}" + f" ({', '.join(unknown[:6])}{' …' if len(unknown) > 6 else ''}): no blocks there (a pause after a full stop gets the" + f" full-stop range) and no pause tags (their pauses are cut like any other)") for i in a.extra: if i not in {s["id"] for s in segs}: sys.exit(f"--extra {i}: no line with that id") @@ -449,22 +553,45 @@ def captions(old): with tempfile.TemporaryDirectory() as td: tmp = Path(td) missing = [s["id"] for s in segs if units(s["text"], a.lang) and not s.get("words")] - if missing: - if a.align == "map": - sys.exit(f"no word times for {', '.join(missing[:6])}{' …' if len(missing) > 6 else ''}: run bin/vh tts … --align gemini" - f" --resume first, or pass --align gemini|whisper here to transcribe the take") - print(f" {len(missing)} line(s) without word times: transcribing the take ({a.align}) …", flush=True) + cache, src_sha = audio / f"pace.{a.lang}.asr.json", sha256(src_wav) + old = json.loads(cache.read_text(encoding="utf-8")) if missing and cache.exists() else {} + found = old.get("sha256") == src_sha and len(old.get("segments", [])) == len(segs) + if missing and not found and a.align == "map": + sys.exit(f"no word times for {', '.join(missing[:6])}{' …' if len(missing) > 6 else ''}: run bin/vh tts … --align gemini" + f" --resume first, or pass --align gemini|whisper here to transcribe the take") + if (missing and not found) or (a.align != "map" and not a.dry_run): # before anything is written if a.align == "gemini" and not os.environ.get("GEMINI_API_KEY"): - sys.exit("set GEMINI_API_KEY (Google AI Studio → Get API key) for --align gemini") - for s, r in zip(segs, align_take(segs, src_wav, a.align, a.lang, tmp, dur)): + sys.exit("set GEMINI_API_KEY (Google AI Studio → Get API key) for --align gemini; nothing was written") + if a.align == "whisper" and importlib.util.find_spec("mlx_whisper") is None: + sys.exit("--align whisper needs mlx-whisper (Apple Silicon; bin/vh pace adds it with --align whisper); nothing was written") + if missing: # the take's own word times, transcribed once and kept beside it + if found: + got = old["segments"]; print(f" word times of the take from {rel(cache)} ({old.get('engine')}, transcribed before)") + else: + print(f" {len(missing)} line(s) without word times: transcribing the take ({a.align}) …", flush=True) + try: + got = align_take(segs, src_wav, a.align, a.lang, tmp, dur) + except Exception as e: # quota, network: nothing is written yet + sys.exit(f"transcribing the take ({a.align}) failed: {e}. Nothing was written") + if not a.dry_run: + cache.write_text(json.dumps({"sha256": src_sha, "engine": a.align, "segments": got}, ensure_ascii=False), encoding="utf-8") + for s, r in zip(segs, got): s.update(r); s.pop("file", None) - tl["align"] = {"model": a.align, "by": "bin/vh pace"} + tl["align"] = {"model": old.get("engine") if found else a.align, "by": "bin/vh pace"} lines, ref, how = plan(x, sr, segs, a.lang, block, asked, a) unit = "chars/s" if a.lang == "zh" else "syll/s" - y, maps = render(x, sr, lines) - new = paced_segments(lines, maps) + y, remap = render(x, sr, lines) + unheard = [L["seg"]["id"] for L in lines if not L["heard"]] + if unheard: + print(f" note: nothing heard for {', '.join(unheard)} (missing from the audio, or under the voice threshold): that stretch" + f" of the take is kept as it is, unpaced; listen to it", flush=True) + new = paced_segments(segs, lines, remap) print(f"{'line':<8} {'units':>5} {'pace':>5} tempo after pause after (take → now)") - for L in lines: + by = {L["i"]: L for L in lines} + for i, s in enumerate(segs): + L = by.get(i) + if not L: + print(f" {s['id']:<6} {'':5} (no words: the pause it stands for is kept)"); continue r = L["rate"] print(f" {L['seg']['id']:<6} {L['u']:5d} " + (f"{r:5.2f} → ×{L['f']:.3f} → {r * L['f_real']:5.2f}" if r else f"{'':5} {'':6} {'':5}") + (f" {L['kind']:<5} {L['gap0']:.2f} → {L['gap']:.2f} s" if "gap" in L else "") @@ -473,15 +600,13 @@ def captions(old): m = [L for L in lines if L["ok"]] if m: before = np.array([L["rate"] for L in m]); after = np.array([L["rate"] * L["f_real"] for L in m]) - print(f"{len(lines)} line(s) · pace {before.min():.2f}–{before.max():.2f} {unit} (sd {before.std():.2f}) → {after.min():.2f}–{after.max():.2f}" + print(f"{len(segs)} line(s) · pace {before.min():.2f}–{before.max():.2f} {unit} (sd {before.std():.2f}) → {after.min():.2f}–{after.max():.2f}" f" (sd {after.std():.2f}) · reference {ref:.2f} ({how}) · {dur:.2f} s → {len(y) / sr:.2f} s") if a.dry_run: print("dry run: nothing written. The numbers are a reference; listen before keeping it.") return if src_wav == wav: # the original files as they are shutil.copyfile(wav, raw_wav); shutil.copyfile(tlp, raw_tl) - if missing: # word times found for the original: keep them with it - raw_tl.write_text(json.dumps(tl, ensure_ascii=False, indent=2), encoding="utf-8") part = wav.with_name(wav.name + ".part") write_wav(y, sr, codec, part) out = {k: v for k, v in tl.items() if k not in ("segments", "duration")} @@ -491,7 +616,8 @@ def captions(old): "gaps": GAPS, "align": "map", "sha256": sha256(part), # the engine once its transcription is in "lines": [{"id": L["seg"]["id"], "units": L["u"], "pace": round(L["rate"], 3) if L["rate"] else None, "tempo": round(L["f"], 4), "after": round(L["rate"] * L["f_real"], 3) if L["rate"] else None, - **({"note": L["why"]} if L["why"] else {})} for L in lines]} + **({"note": L["why"]} if L["why"] else {})} if L else {"id": s["id"], "units": 0, "note": "no words"} + for s, L in ((s, by.get(i)) for i, s in enumerate(segs))]} def save(): body = json.dumps(out, ensure_ascii=False, indent=2) tlp.with_name(tlp.name + ".part").write_text(body, encoding="utf-8"); os.replace(tlp.with_name(tlp.name + ".part"), tlp) diff --git a/tools/ci.sh b/tools/ci.sh index 0ba06ef..198e342 100755 --- a/tools/ci.sh +++ b/tools/ci.sh @@ -821,23 +821,24 @@ PY rm -rf "$t" } -# bin/vh pace on a synthetic take: tone-burst "syllables" with known onsets, lines at 4 to 7 a second, a long pause inside a -# line and one the script asks for, gaps too short and too long, an ASR start placed 0.12 s late, and one line's end and the -# next one's start put at the same instant inside the first one's last syllable (an aligner did that). Checked against the -# known syllables, not against the tool's own numbers +# bin/vh pace on a synthetic take (22.05 kHz): tone-burst "syllables" with known onsets, lines at 4 to 7 a second, a long +# pause inside a line and one the script asks for, gaps too short and too long, an ASR start placed 0.12 s late, one line's +# end and the next one's start put at the same instant inside the first one's last syllable (an aligner did that), and a +# "……" line. Then ASRs that are further off, a line with no audio, and the refusals. Checked against the known syllables, +# not against the tool's own numbers pace_checks() { - local t p a rc + local t p v a rc if ! command -v uv >/dev/null || ! command -v ffmpeg >/dev/null; then skip "pace" "needs uv and ffmpeg"; return; fi t=$(mktemp -d "${TMPDIR:-/tmp}/vh-ci.XXXXXX"); p="$t/p" python3 - "$p/audio" <<'PY' import json, math, struct, sys, wave from pathlib import Path -a = Path(sys.argv[1]); a.mkdir(parents=True); SR = 24000 +a = Path(sys.argv[1]); a.mkdir(parents=True); SR = 22050 lines = [("a1", "一二三四五六七八，", 4.0, None, 0.05), ("a2", "九十百千万亿兆京。", 7.0, None, 1.6), # id, text, per second, ("b1", "甲乙丙丁戊己庚辛壬癸。", 5.5, (5, 0.7), 0.4), ("b2", "好。", 3.0, None, 0.3), # (after unit k, pause), ("b3", "子丑寅卯辰巳午未。", 5.0, (4, 0.6), 0.5), ("b4", "东南西北中发白。", 5.6, None, 0.0)] # gap after the line (a / "script.txt").write_text("@a1 一二三四五六七八，\n@a2 九十百千万亿兆京。\n\n@b1 甲乙丙丁戊己庚辛壬癸。\n@b2 好。\n" - "@b3 子丑寅卯辰巳午未。\n@b4 东南西北中发白。\n", encoding="utf-8") + "@b3 子丑寅卯辰巳午未。\n@b3x ……\n@b4 东南西北中发白。\n", encoding="utf-8") x, segs = [0.0] * int(0.4 * SR), [] for k, (sid, text, rate, pause, gap) in enumerate(lines): chars = [c for c in text if c not in "，。"]; burst = 0.75 / rate; words = [] @@ -851,6 +852,8 @@ for k, (sid, text, rate, pause, gap) in enumerate(lines): words[0]["start"] += 0.12 # the ASR heard the first syllable late if sid == "b2": # b1 ends and b2 starts at the same instant, 20 ms before b1's voice ends segs[-1]["words"][-1]["end"] = words[0]["start"] = segs[-1]["end"] = round(segs[-1]["end"] - 0.02, 3) + if sid == "b4": # "……": no words, its span is the pause + segs.append({"id": "b3x", "zh": "……", "en": "", "text": "……", "start": segs[-1]["end"], "end": words[0]["start"]}) segs.append({"id": sid, "zh": text, "en": "", "text": text, "start": words[0]["start"], "end": words[-1]["end"], "words": words}) x += [0.0] * int(gap * SR) x += [0.0] * int(0.5 * SR) @@ -862,31 +865,40 @@ for f in ("timeline.zh.json", "timeline.json"): (a / "voiceover.wav").write_bytes((a / "voiceover.zh.wav").read_bytes()) PY cp -R "$p/audio" "$t/orig" + # the onsets of the syllables in a take (10 ms frames above −40 dBFS) and the quiet before each: what every check below counts + cat > "$t/onsets.py" <<'PY' +import subprocess, numpy as np +def onsets(wav, sr=22050): + y = np.frombuffer(subprocess.run(["ffmpeg", "-v", "error", "-i", wav, "-f", "f32le", "-ac", "1", "-ar", str(sr), "-"], + check=True, capture_output=True).stdout, " -40 + up, down = (np.flatnonzero(np.diff(on.astype(int)) == 1) + 1) * 0.01, (np.flatnonzero(np.diff(on.astype(int)) == -1) + 1) * 0.01 + return up, np.array([up[0]] + [u - d for u, d in zip(up[1:], down)]), on +PY vh pace "$p" --dry-run >/dev/null && diff -r "$p/audio" "$t/orig" >/dev/null && ok "pace --dry-run writes nothing" || bad "pace --dry-run changed the project" - if a=$(vh pace "$p" 2>&1) && a=$(uv run -q --no-project --with numpy python - "$p/audio" "$t/orig" 2>&1 <<'PY' -import json, subprocess, sys + if a=$(vh pace "$p" 2>&1) && a=$(PYTHONPATH="$t" uv run -q --no-project --with numpy python - "$p/audio" "$t/orig" 2>&1 <<'PY' +import json, sys import numpy as np -A, O = sys.argv[1], sys.argv[2]; sr = 24000; bad = [] -y = np.frombuffer(subprocess.run(["ffmpeg", "-v", "error", "-i", f"{A}/voiceover.zh.wav", "-f", "f32le", "-ac", "1", "-ar", str(sr), "-"], - check=True, capture_output=True).stdout, " -40 -ons = (np.flatnonzero(np.diff(on.astype(int)) == 1) + 1) * 0.01 -words = [w for s in tl["segments"] for w in s["words"]] +ons, quiet, on = onsets(f"{A}/voiceover.zh.wav") +words = [w for s in tl["segments"] for w in s.get("words", [])] if len(ons) != 42 or len(words) != 42: bad.append(f"{len(ons)} syllables in the paced take, {len(words)} words (42 each)") else: err = np.abs(ons - [w["start"] for w in words]) - if err.max() > 0.06: bad.append(f"a mapped word {err.max():.3f} s from its syllable ({words[int(err.argmax())]['w']})") + if err.max() > 0.035: bad.append(f"a mapped word {err.max():.3f} s from its syllable ({words[int(err.argmax())]['w']})") L = {l["id"]: l for l in P["lines"]} -paced = [l for l in P["lines"] if l.get("note") != "short"] +paced = [l for l in P["lines"] if l.get("note") not in ("short", "no words")] sd = lambda k: float(np.std([l[k] for l in paced])) if not sd("after") < 0.8 * sd("pace"): bad.append(f"spread of paces {sd('pace'):.2f} → {sd('after'):.2f}") -if not all(0.92 - 1e-9 <= l["tempo"] <= 1.18 + 1e-9 for l in P["lines"]) or L["b2"]["tempo"] != 1.0 or L["a1"]["tempo"] <= 1 or L["a2"]["tempo"] >= 1: - bad.append(f"tempos {[(l['id'], l['tempo']) for l in P['lines']]}") +if not all(0.92 - 1e-9 <= l["tempo"] <= 1.18 + 1e-9 for l in P["lines"] if "tempo" in l) or L["b2"]["tempo"] != 1.0 or L["a1"]["tempo"] <= 1 \ + or L["a2"]["tempo"] >= 1 or L["b3x"].get("note") != "no words": + bad.append(f"tempos {[(l['id'], l.get('tempo'), l.get('note')) for l in P['lines']]}") def longest_quiet(s): q = on[int(s["start"] / 0.01): int(s["end"] / 0.01)]; best = cur = 0 for v in q: @@ -894,20 +906,149 @@ def longest_quiet(s): return best * 0.01 if longest_quiet(S["b1"]) > 0.3 or longest_quiet(S["b3"]) < 0.5: bad.append(f"inner pauses: b1 {longest_quiet(S['b1']):.2f} (cap 0.26), b3 {longest_quiet(S['b3']):.2f} (asked for)") gap = lambda i, j: S[j]["start"] - S[i]["end"] -if not (0.1 <= gap("a1", "a2") <= 0.2 and 1.1 <= gap("a2", "b1") <= 1.25 and 0.35 <= gap("b1", "b2") <= 0.45): - bad.append(f"gaps a1-a2 {gap('a1', 'a2'):.2f} (0.12), a2-b1 {gap('a2', 'b1'):.2f} (1.2), b1-b2 {gap('b1', 'b2'):.2f} (the take's 0.4)") +if not (0.1 <= gap("a1", "a2") <= 0.2 and 1.1 <= gap("a2", "b1") <= 1.25 and 0.35 <= gap("b1", "b2") <= 0.45 and 0.45 <= gap("b3", "b4") <= 0.56): + bad.append(f"gaps a1-a2 {gap('a1', 'a2'):.2f} (0.12), a2-b1 {gap('a2', 'b1'):.2f} (1.2), b1-b2 {gap('b1', 'b2'):.2f} (the take's 0.4)," + f" b3-b4 {gap('b3', 'b4'):.2f} (the 0.5 a \"……\" line stands for)") +if not S["b3"]["end"] - 0.005 <= S["b3x"]["start"] <= S["b3x"]["end"] <= S["b4"]["start"] + 0.005: bad.append(f"the \"……\" line is not between b3 and b4: {S['b3x']}") caps = json.load(open(f"{A}/captions.json", encoding="utf-8")) if [c["start"] for c in caps] != [s["start"] for s in tl["segments"]]: bad.append("captions.json is not from the paced timeline") print("\n".join(bad)); sys.exit(1 if bad else 0) PY - ); then ok "pace: every syllable kept and every mapped word on it, paces closer within the clamp, an inner pause capped and an asked-for one kept, gaps in range, originals kept, captions redone" + ); then ok "pace: every syllable kept and every mapped word on it, paces closer within the clamp, an inner pause capped and an asked-for one kept, gaps in range, a \"……\" line's pause kept, originals kept, captions redone" else bad "pace on a synthetic take: $a"; fi cp "$p/audio/voiceover.zh.wav" "$t/once.wav" vh pace "$p" >/dev/null && cmp -s "$t/once.wav" "$p/audio/voiceover.zh.wav" && ok "pace runs again from the original take: the same bytes" || bad "pace: a second run gave a different take" + vh pace "$p" --restore --dry-run >/dev/null 2>&1; rc=$? + [ $rc = 2 ] && cmp -s "$t/once.wav" "$p/audio/voiceover.zh.wav" && ok "pace --restore --dry-run is refused" || bad "pace --restore --dry-run: exit $rc" vh pace "$p" --restore >/dev/null && cmp -s "$p/audio/voiceover.zh.wav" "$t/orig/voiceover.zh.wav" && cmp -s "$p/audio/timeline.zh.json" "$t/orig/timeline.zh.json" \ + && cmp -s "$p/audio/voiceover.wav" "$t/orig/voiceover.zh.wav" && cmp -s "$p/audio/timeline.json" "$t/orig/timeline.json" \ && [ ! -e "$p/audio/voiceover.zh.raw.wav" ] && ok "pace --restore puts the original back" || bad "pace --restore" - python3 -c "import json, sys; t = json.load(open(sys.argv[1])); t['grid'] = {'beats': 'm.json'}; json.dump(t, open(sys.argv[1], 'w'))" "$p/audio/timeline.zh.json" - a=$(vh pace "$p" 2>&1); rc=$? + # ASRs further off: every start 0.2 s late and end 0.3 s early (f), a line break placed 0.1 s inside a2's last syllable (m), + # the last line with no audio at all (s): no syllable lost or moved behind a pause, no crash + for v in f m s; do mkdir "$t/$v" && cp -R "$t/orig" "$t/$v/audio"; done + python3 - "$t" <<'PY' +import json, subprocess, sys +t = sys.argv[1]; F = "audio/timeline.zh.json" +def edit(v, fn): + tl = json.load(open(f"{t}/{v}/{F}", encoding="utf-8")); fn(tl["segments"]); json.dump(tl, open(f"{t}/{v}/{F}", "w", encoding="utf-8"), ensure_ascii=False) +def late(segs): + for s in segs: + if s.get("words"): + s["start"] = s["words"][0]["start"] = round(min(s["start"] + 0.2, s["words"][0]["end"]), 3) + s["end"] = s["words"][-1]["end"] = round(max(s["end"] - 0.3, s["words"][-1]["start"]), 3) +def inside(segs): + p, q = segs[1], segs[2] # a2 | b1 + p["end"] = p["words"][-1]["end"] = q["start"] = q["words"][0]["start"] = round(p["end"] - 0.1, 3) +edit("f", late); edit("m", inside) +b4 = json.load(open(f"{t}/s/{F}", encoding="utf-8"))["segments"][-1]["start"] +subprocess.run(["ffmpeg", "-v", "error", "-y", "-i", f"{t}/orig/voiceover.zh.wav", "-t", f"{b4 - 0.05:.3f}", f"{t}/s/audio/voiceover.zh.wav"], check=True) +PY + a=""; for v in f m s; do vh pace "$t/$v" >/dev/null 2>"$t/$v.err" || a="$a $v: $(tail -c 300 "$t/$v.err")"; done + if [ -z "$a" ] && a=$(PYTHONPATH="$t" uv run -q --no-project --with numpy python - "$t" 2>&1 <<'PY' +import json, sys +from onsets import onsets +t = sys.argv[1]; bad = [] +for v, want in (("f", 42), ("m", 42), ("s", 35)): + ons, quiet, _ = onsets(f"{t}/{v}/audio/voiceover.zh.wav") + if len(ons) != want: bad.append(f"{v}: {len(ons)} syllables, want {want}") + elif v == "m" and quiet[15] > 0.1: bad.append(f"m: a2's last syllable moved behind a {quiet[15]:.2f} s pause") +L = {l["id"]: l for l in json.load(open(f"{t}/s/audio/timeline.zh.json", encoding="utf-8"))["pace"]["lines"]} +if L["b4"].get("note") != "nothing heard": bad.append(f"s: b4 {L['b4']}") +print("\n".join(bad)); sys.exit(1 if bad else 0) +PY + ); then ok "pace: ASR starts 0.2 s late and ends 0.3 s early, a line break inside a syllable, a line with no audio: every syllable kept, none moved" + else bad "pace with an ASR further off:$a"; fi + # more cases, each on its own copy: b1 with no audio (u), b1 and b2 spans swapped (o), word times only in the cache (c, and c2 + # without it), no script.txt (n), and a take of its own (t): a quiet breath before one line, three short loud syllables + # opening the next + for v in u o c c2 n; do mkdir "$t/$v" && cp -R "$t/orig" "$t/$v/audio"; done + python3 - "$t" <<'PY' +import hashlib, json, math, struct, sys, wave +from pathlib import Path +t = sys.argv[1]; F = "audio/timeline.zh.json" +tl = json.load(open(f"{t}/orig/timeline.zh.json", encoding="utf-8")); S = {s["id"]: s for s in tl["segments"]} +b1 = S["b1"] # b1's syllables zeroed (its timeline end sits 0.02 s before its voice ends) +with wave.open(f"{t}/orig/voiceover.zh.wav") as r: + sr, pcm = r.getframerate(), bytearray(r.readframes(r.getnframes())) +z0, z1 = int((b1["words"][0]["start"] - 0.005) * sr), int((b1["words"][-1]["end"] + 0.025) * sr) +pcm[2 * z0: 2 * z1] = bytes(2 * (z1 - z0)) +with wave.open(f"{t}/u/audio/voiceover.zh.wav", "wb") as w: + w.setnchannels(1); w.setsampwidth(2); w.setframerate(sr); w.writeframes(bytes(pcm)) +o = json.loads(json.dumps(tl)); a, b = o["segments"][2], o["segments"][3] +a["start"], a["end"], b["start"], b["end"] = b["start"], b["end"], a["start"], a["end"] +json.dump(o, open(f"{t}/o/{F}", "w", encoding="utf-8"), ensure_ascii=False) +for v in ("c", "c2"): # word times removed; for c they are in the cache under the take's sha256 + c = json.loads(json.dumps(tl)) + for s in c["segments"]: + s.pop("words", None) + Path(f"{t}/{v}/{F}").write_text(json.dumps(c, ensure_ascii=False), encoding="utf-8") +sha = hashlib.sha256(open(f"{t}/orig/voiceover.zh.wav", "rb").read()).hexdigest() +Path(f"{t}/c/audio/pace.zh.asr.json").write_text(json.dumps({"sha256": sha, "engine": "whisper", "segments": [ + {"start": s["start"], "end": s["end"], **({"words": s["words"]} if s.get("words") else {})} for s in tl["segments"]]}), encoding="utf-8") +Path(f"{t}/n/audio/script.txt").unlink() +SR, x, segs, A = 22050, [0.0] * int(0.4 * 22050), [], Path(f"{t}/t/audio") +A.mkdir(parents=True) +def burst(dur, amp, f0): + n = int(dur * SR); return [amp * math.sin(math.pi * i / n) ** 0.5 * (math.sin(2 * math.pi * f0 * i / SR) + 0.5 * math.sin(4 * math.pi * f0 * i / SR)) for i in range(n)] +for sid, text, plan in (("t1", "甲乙丙丁戊己庚辛。", [("breath",)] + [(0.15, 0.05)] * 8), ("t2", "子丑寅卯辰巳午未申酉戌亥。", [(0.09, 0.09)] * 3 + [(0.15, 0.05)] * 9), + ("t3", "东南西北中发白甲。", [(0.15, 0.05)] * 8)): + words, chars = [], iter(text[:-1]) + for k, step in enumerate(plan): + if step == ("breath",): + x += burst(0.08, 0.0095, 300) + [0.0] * int(0.12 * SR); continue + t0 = len(x) / SR; x += burst(step[0], 0.25, 150 + 10 * (k % 4)) + words.append({"w": next(chars), "start": round(t0, 3), "end": round(len(x) / SR, 3)}); x += [0.0] * int(step[1] * SR) + words[-1]["w"] += "。"; segs.append({"id": sid, "zh": text, "en": "", "text": text, "start": words[0]["start"], "end": words[-1]["end"], "words": words}) + x += [0.0] * int(0.45 * SR) +with wave.open(str(A / "voiceover.zh.wav"), "wb") as w: + w.setnchannels(1); w.setsampwidth(2); w.setframerate(SR); w.writeframes(struct.pack(f"<{len(x)}h", *(int(round(v * 32767)) for v in x))) +(A / "timeline.zh.json").write_text(json.dumps({"provider": "say", "lang": "zh", "segments": segs}, ensure_ascii=False), encoding="utf-8") +(A / "script.txt").write_text("".join(f"@{s['id']} {s['text']}\n" for s in segs), encoding="utf-8") +PY + cp -R "$t/c/audio" "$t/c.before" + a=""; rc=0 + vh pace "$t/u" >"$t/u.out" 2>&1 || a="$a u: $(tail -c 300 "$t/u.out")" + vh pace "$t/t" >"$t/t.out" 2>&1 || a="$a t: $(tail -c 300 "$t/t.out")" + b=$(vh pace "$t/c" --dry-run 2>&1) && diff -r "$t/c/audio" "$t/c.before" >/dev/null || a="$a c --dry-run wrote or failed: $b" + case "$b" in *"transcribed before"*) ;; *) a="$a c --dry-run did not use the cache: $b" ;; esac + vh pace "$t/c" >/dev/null 2>&1 && cmp -s "$t/c/audio/timeline.zh.raw.json" "$t/c.before/timeline.zh.json" || a="$a c: the .raw timeline is not the original" + b=$(vh pace "$t/c2" 2>&1); [ $? = 1 ] && case "$b" in *"no word times"*) ;; *) false ;; esac || a="$a c2 (no words, no cache): $b" + b=$(vh pace "$t/o" 2>&1); [ $? = 1 ] && case "$b" in *"out of order"*) ;; *) false ;; esac || a="$a o (spans swapped): $b" + b=$(vh pace "$t/n" --dry-run 2>&1); case "$b" in *"no audio/script.txt"*) ;; *) a="$a n (no script.txt): $b" ;; esac + b=$(PYTHONPATH="$t" uv run -q --no-project --with numpy python - "$t" 2>&1 <<'PY' +import json, sys +from onsets import onsets +t = sys.argv[1]; bad = [] +raw = {s["id"]: s for s in json.load(open(f"{t}/orig/timeline.zh.json", encoding="utf-8"))["segments"]} +tl = json.load(open(f"{t}/u/audio/timeline.zh.json", encoding="utf-8")); S = {s["id"]: s for s in tl["segments"]} +L = {l["id"]: l for l in tl["pace"]["lines"]} +ons, _, _ = onsets(f"{t}/u/audio/voiceover.zh.wav") +g0 = raw["b1"]["end"] + 0.02 + 0.4 - raw["a2"]["end"] # a2's voice to b2's, in the take (b2's timeline start is the shared one) +g1 = S["b2"]["start"] - S["a2"]["end"] +if len(ons) != 32 or L["b1"].get("note") != "nothing heard" or abs(g1 - g0) > 0.05 or S["b1"]["end"] - S["b1"]["start"] < 0.1 - 1e-9 \ + or S["b1"]["start"] < S["a2"]["end"]: + bad.append(f"u: {len(ons)} syllables (32), b1 {L['b1']} at {S['b1']['start']}–{S['b1']['end']}, a2→b2 {g1:.2f} s (the take's {g0:.2f})") +P = {l["id"]: l for l in json.load(open(f"{t}/t/audio/timeline.zh.json", encoding="utf-8"))["pace"]["lines"]} +true = {"t1": 8 / 1.55, "t2": 12 / 2.29} # syllables over first onset → last end; the breath is not speech +for k, v in true.items(): + if abs(P[k]["pace"] / v - 1) > 0.06: bad.append(f"t: {k} measured {P[k]['pace']:.2f}, true {v:.2f}") +c = json.load(open(f"{t}/c/audio/timeline.zh.json", encoding="utf-8")) +if sum(len(s.get("words", [])) for s in c["segments"]) != 42: bad.append("c: the cached word times did not reach the paced timeline") +print("\n".join(bad)); sys.exit(1 if bad else 0) +PY + ) || a="$a $b" + if [ -z "$a" ]; then ok "pace: a line with no audio keeps its stretch of the take, a quiet breath is not speech and short loud syllables are, cached word times are used and the .raw stays the original, swapped spans and a missing script.txt are said" + else bad "pace on the other cases:$a"; fi + mkdir "$t/k" && cp -R "$t/orig" "$t/k/audio" + a=$(vh pace "$t/k" --dry-run --target 6 --extra b2=0.3 zh 2>&1); rc=$? + case "$rc:$a" in 0:*"(--target 6)"*) printf '%s\n' "$a" | grep -q '^ b2 .*→ 0\.59 s' && ok "pace --target, --extra and the language after the flags" || bad "pace --extra b2=0.3: $a" ;; + *) bad "pace --target 6 … zh: exit $rc: $a" ;; esac + a=$(env -u GEMINI_API_KEY "$VH_BASH" "$ROOT/bin/vh" pace "$t/k" --align gemini 2>&1); rc=$? + [ $rc = 1 ] && diff -r "$t/k/audio" "$t/orig" >/dev/null && ok "pace --align gemini without a key stops before writing" || bad "pace --align gemini without a key: exit $rc, $a" + a=$(vh pace "$t/k" --ref b3x --dry-run 2>&1); rc=$? + case "$rc:$a" in 1:*"--ref b3x"*) ok "pace --ref on a line without words is refused" ;; *) bad "pace --ref b3x: exit $rc: $a" ;; esac + python3 -c "import json, sys; t = json.load(open(sys.argv[1])); t['grid'] = {'beats': 'm.json'}; json.dump(t, open(sys.argv[1], 'w'))" "$t/k/audio/timeline.zh.json" + a=$(vh pace "$t/k" 2>&1); rc=$? case "$rc:$a" in 1:*"beat grid"*) ok "pace refuses a timeline on a beat grid" ;; *) bad "pace on a beat grid: exit $rc: $a" ;; esac rm -rf "$t" }