DubPipe

English · Русский

Automatic English → Russian video dubbing. Feed it a video file or a YouTube URL and get the same picture back with a Russian voice track instead of the English speech, while the music and effects stay where they were. It can also produce subtitles only — two SRT files, the original transcript and the translation.

Everything except the translation step runs on this computer, or on your own GPU server on the local network. No GPU required: without one, everything runs on the CPU.

⚠️ Personal use only

DubPipe exists for private copying — watching a video at home in a language you understand. Publishing or distributing the resulting track infringes the rights of the copyright holder and violates platform rules. The tool deliberately has no upload feature of any kind, and it never will.

---

Table of contents

• What it does
• How it works
• Requirements
• Installation
• Quick start
• Desktop app
• Subtitles
• Execution profiles
• Commands
• Character voices
• Review and fix the result
• Configuration
• A GPU server
• Performance
• Downloaded components and their licenses
• Privacy and security
• Development
• Known limitations
• License

---

What it does

• Dubbing. Speech recognition with word-level timing, translation written to fit the time slot, Russian speech synthesis, tempo fitting and mixing back into the video.
• Subtitles. Two SRT files — <name>.en.srt and <name>.ru.srt — laid out by readability rules rather than dumped from the recognizer.
• Character voices. Speaker diarization assigns a separate voice per speaker, and the voice gender is estimated from the pitch of the original speaker, so a man is not dubbed by a female voice by default.
• Review mode. Watch the finished file, mark what is wrong (voice, speaker, translation, volume levels) and re-mix with one button — only the affected replicas are re-synthesized.
• Resumable cache. Every stage is cached by input content and stage settings; a repeated run recomputes only what actually changed.
• No system installation. ffmpeg, yt-dlp, whisper.cpp and piper are downloaded into a local folder, not into the system, and no administrator rights are needed.

How it works

Stage	What happens	Engine
S1	Input handling, audio extraction	ffmpeg (+ yt-dlp for URLs)
S2	Recognition, word timings, speaker diarization	whisper.cpp or faster-whisper, silero VAD, pyannote
S3	Translation into Russian, written to fit the slot	Kilo Gateway (cloud) or Ollama (local)
S4	Voice/music separation (optional)	MDX-Net through a Python sidecar
S5	Russian speech synthesis	piper or silero
S6	Fitting each line into its slot (tempo, shortening, pulling up sentence continuations); actors' voices (optional)	ffmpeg + LLM; X-Voice on the GPU server
S7	Mixing and muxing back into the video	ffmpeg

Any of S2, S4, S5 and the actors' voices can run on your own GPU server instead of this computer, and local translation can use that server's Ollama.

Word timings come from whisper.cpp DTW alignment rather than the heuristic offsets: the heuristic puts the first words of a phrase at the start of the whole segment, which on real material lands 0.5–1.3 s before the speech actually begins.

Recognition also invents text. Whisper learned subtitle files during training, so on music and in silence it writes out the boilerplate it saw there — credits for the captions, ad notices, "thanks for watching" — and sometimes loops a single phrase for minutes. On a Korean episode that was 68 replicas out of 412, which the pipeline then dutifully translated and voiced. S2 drops them: known captioning formulas by pattern, and stuck decoding by the one property real speech never has — identical lines that overlap in time. Repeated lines that do not overlap are kept, because a character really can say "seriously?" twelve times in a row.

Requirements

• Node.js 20+ (tested on 25.9)
• About 2 GB of free space for models and binaries
• No GPU needed, though one helps: see Using the GPU
• Python is optional and only for the parts that are optional themselves: voice separation (S4, pip install numpy onnxruntime), speaker diarization (S2, pip install pyannote.audio, which pulls in PyTorch, ~1 GB), the silero voices (S5, torch and soundfile) and recognition with asr.engine: faster-whisper (S2, pip install faster-whisper). Without Python, separation falls back to ducking the original, every line gets a single voice, synthesis uses piper and recognition whisper.cpp; the rest of the pipeline never touches Python. A stage given to the GPU server needs no Python here at all. An optional pip install onnxruntime-directml moves the speaker embeddings onto the GPU and makes diarization four times faster.

The code targets Windows, Linux and macOS; the packaged desktop build and every measurement in this README come from Windows.

Installation

npm install
npm run build                       # or run everything through: npx tsx src/cli.ts
npm link                            # get the `dub` command instead of the long npx tsx path
npx tsx src/cli.ts doctor --fetch   # download the missing components (~165 MB)

Quick start

# 1. Create a configuration file
npx tsx src/cli.ts config init

# 2. Check the environment
npx tsx src/cli.ts doctor

# 3. Produce a dubbed video
npx tsx src/cli.ts process video.mp4 --out video.ru.mp4

# 4. Or subtitles only, both languages
npx tsx src/cli.ts process video.mp4 --to-stage s3 --subtitles

By default the result is written next to the source file with a .ru suffix. Use --out-dir to choose a folder, or --out for an exact path.

A run can be interrupted at any moment — the Stop button in the app or Ctrl+C in the terminal (exit code 130). Child processes are killed immediately and model requests are aborted; finished stages stay in the cache and the interrupted one is recomputed next time.

Desktop app

npm run ui          # opens in the browser
npm run app:dev     # the same thing in an application window
npm run app:build   # build the .exe (installer + portable)

The interface uses a graphite background and mint accents, independent of the system's light/dark preference. The same design covers the browser and Electron app, including downloads, settings, environment checks, editors and dialogs.

The main screen is a folder of videos, with real frame thumbnails, name search and filters for source files, finished dubs and unprocessed files. Opening a file prepares the studio; it does not start processing. The studio keeps the video beside the replica editor and the character/voice and volume controls. A seven-stage rail shows progress; detailed stages, subtitles, model comparison and the log remain available on their own tabs.

Downloads separates link preparation from the active queue and completed history. Paste one or several URLs, inspect the first link, choose quality and start downloading. The destination stays visible. Settings has nine categories: Compute, Translation, Voice, Recognition, Sound, Fitting, Downloads, System and YAML. Draft settings survive navigation; switching between the form and YAML requires saving or discarding changes first.

The folder is re-read with the Refresh button, and by itself when the window gets focus back: videos get downloaded into it and finished dubs moved out of it behind the program's back. Delete moves a video to the Recycle Bin and removes the work files of its dub, which are useless without the source. Only a media file from the chosen folder can be deleted this way, and not the one being processed right now. When the interface runs in a browser (npm run ui) on Linux or macOS without gio, there is no Recycle Bin to reach, and the file is deleted permanently; the message says so.

If something is missing — a program, model weights, a voice or an API key — a panel at the top states exactly what will not work and how to fix it. One button downloads everything in parallel, with progress bars.

The packaged app needs no separate Node.js installation. Python is needed only for optional local engines described under Requirements. Build artifacts are written to release/; the Windows x64 portable built on 2026-09-30 is about 155 MiB. Both installer and portable use %APPDATA%/DubPipe for configuration and working data. To build only portable:

npm run build
npx electron-builder --win portable --x64

The interface supports Russian and English. The language selector in the sidebar remembers the choice, including the Electron menus. Pipeline diagnostic messages remain in Russian.

What happens to the original track

The Russian speech goes over the original, so something has to be done with it. There are three ways, chosen by the settings.

Without separation — the default — the original is ducked by 18 dB under our speech. Simple and reliable, but the music dips along with the foreign voice.

With separation on, the extracted voice is subtracted from the original, and only where our own speech plays. The music stays at full level, and between replicas the original is untouched, so a song we do not dub keeps its vocals. That is the default, separation.apply: under_speech.

everywhere replaces the original with the rebuilt background for the whole film. It removes the original voice everywhere, and strips songs of their vocals.

The speech rate is measured, not guessed

How many characters the synthesizer speaks per second decides how long a translation to ask for, and one number cannot describe it. Every replica carries a fixed cost — the approach to the phrase and the tail after it. Measured on 255 replicas of one voice: lines under 15 characters come out at 10.2 characters per second, lines over 80 at 16.6, with the voice unchanged.

So a straight line is fitted — duration = overhead + characters / rate — and on that material it gave 0.51 seconds and 17.8 characters per second. The real speaking rate is nearly twice the apparent one, and long lines used to be asked for a third less text than they could hold.

Both numbers are measured on every run and remembered, for the recording and for the voice itself, so the first run of the next video aims correctly. A measurement taken on a different voice is never used: it is worse than an honest default.

Fitting to the timing

A replica that does not fit its slot used to be rewritten shorter by the model, losing meaning. Fitting now first borrows the silence that follows the replica, up to 1.2 seconds, never crossing into the next one. On a Korean episode that took the number of rewritten replicas from 52 down to 1, and sped-up ones from 114 down to 9.

Recognition cuts a long sentence into replicas wherever it happens to, and the translation keeps the cut. When the Russian first half is shorter than the English one, the second half used to wait for its place in the original — on a FRANCE 24 clip that left 2.7 seconds of silence in the middle of a sentence. So a continuation is pulled up: when the same speaker goes on, the previous Russian line does not end a sentence, and the gap in the original is at most 0.6 seconds, the line starts earlier, so that the pause inside the sentence is no longer than 0.3 seconds. It moves only earlier and at most 3 seconds, and this is the last step of fitting, done on the finished clips, actors' voices included.

Subtitles

The Subtitles button next to Dub (or --subtitles on the command line) runs recognition and translation and writes two files next to the video.

Subtitles are not pipeline replicas dumped to disk: replicas are cut for voicing and can be one word long or four lines wide. Cues are rebuilt under readability rules — at most two lines of 42 characters, duration between 1 and 7 seconds, a gap of at least 84 ms (two frames at 24 fps) between neighbours, and a reading speed of at most 17 characters per second. A long replica is split on sentence boundaries with the time divided proportionally; a line break prefers punctuation and never leaves a preposition or conjunction dangling at the end of a line. Files are UTF-8 with a BOM, otherwise Windows players mangle Cyrillic.

The Subtitles tab shows the cues of both languages with per-cue warnings (too fast to read, overlap, line too long), lets you edit text and timecodes, and renders the cues over the video. All rules are configurable under subtitles: in config.yaml.

A run without voicing — subtitles, or anything stopping at translation — skips speaker diarization, since speakers are only used to hand out voices. On a 15-minute episode that brings recognition down to about 1.5 minutes; with diarization the same stage takes five to six. The recognition cache keeps the two apart, so a later dubbing run does not pick up a transcript that has no speakers in it.

Languages

The source language is a parameter, not an assumption. Set asr.language and the pipeline adapts what actually depends on the writing system: how much longer a translation may be than its source (×2 for English, ×3 for Korean, ×5 for Chinese and Japanese — a character carries more meaning than a letter), whether recognized words are joined with spaces, the subtitle line width and reading speed, and which characters end a sentence. A language without a profile gets Latin rules. Subtitle files are named by language code, so a Korean episode yields episode.ko.srt next to episode.ru.srt.

The interface language is switched next to the application name and remembered between runs. The UI ships in English and Russian; messages that come from the server, such as the readiness panel, arrive in the chosen language too. Pipeline log messages stay in Russian.

Execution profiles

	offline	hybrid (default)
Network after the models are downloaded	not needed	only for translation
API keys	none	a Kilo Gateway key for S3
Translation	a local LLM through Ollama	anthropic/claude-sonnet-4.5
Everything else	local, or your GPU server on the LAN	local, or your GPU server on the LAN

Switching takes one line in config.yaml:

profile: offline

If the key is missing or the gateway is unreachable in the hybrid profile, translation degrades to the local model instead of failing the run.

Local translation through Ollama

The offline profile translates with a local model: nothing leaves the machine, no keys are needed, and every episode is free. What it takes:

1. Install Ollama (winget install Ollama.Ollama, or from ollama.com) and pull a model: ollama pull qwen2.5:7b-instruct. The offline profile uses that one unless you choose another.
2. In the interface: Settings → Translation, profile "offline" — the model field then lists what is pulled locally. The same in config.yaml:

profile: offline
translate:
  model: qwen3:8b            # the name as `ollama list` shows it
  ollama_endpoint: http://127.0.0.1:11434

The Environment screen does not check that Ollama is installed; it asks the daemon whether the chosen model is actually there. If it is not, but translate.fallback_model is, translation runs on the fallback model and the run warns about it: a model name left over from the cloud or from another computer is no reason to stop.

An integrated GPU has to be enabled explicitly. Ollama sees it and silently drops it — in the log that reads dropping integrated GPU; to enable, set OLLAMA_IGPU_ENABLE=1. Set that variable and restart Ollama. Measured on a Radeon 780M with a real translation request:

	through Vulkan	CPU only
Prompt processing	243 tok/s	59.6 tok/s
Generation	8.9–16.2 tok/s	6.0 tok/s

ROCm does not work on that card (no rocblas support for gfx target gfx1103); Vulkan does, through AMD's own driver. The GPU is not limited to the BIOS carve-out either: with unified memory Ollama reported 27.9 GB of 40 available, so a 14B model fits comfortably.

How they translate. Measured with the project's own dub compare on twenty lines of a real episode: how many translations fit the character budget their slot allows (the spec asks for ≥90%).

Model	Fit	Requests for 20 lines	Time
qwen3:4b	6/20	21	133 s
qwen3:8b	7/20	13	227 s
qwen3:14b	5/20	13	402 s
mistral-nemo:12b	6/20	13	284 s
qwen2.5:7b-instruct	6/20	13	300 s

Size does not decide it here: the 14B was the worst of the five and three times slower than the 4B. All of them translate understandably, but none of them respects the length budget — and that budget is what decides whether a line lands in its slot. S6 then has to make up the difference by shortening lines with the same local model, so the slow part multiplies. The sample is small, twenty lines, and other material may rank them differently, but the order of magnitude is this.

What to expect in time. Generation is bound by memory bandwidth, so speed falls with model size: 7–8B at Q4 gives roughly 9–16 tokens per second, 12–14B about half that. An episode of 136 lines is on the order of 20,000 prompt tokens and 4,000 generated, which means minutes, not seconds; the same work in the cloud takes about a minute and costs cents. Local translation is worth it for offline use and privacy, not for savings.

The final translation review

Translation runs in batches of ten lines and sees three neighbours — enough for a phrase, not for a film. A whole class of errors is invisible in that frame: a character's name spelled three ways, masculine verbs for a female character, informal address in one scene and formal in the next, a fragment left untranslated. The review is one more pass that sees the whole translation at once, along with each speaker's gender (measured from the voice in S2), the character names and the glossary.

translate:
  review:
    enabled: true   # the hybrid profile turns it on by itself; set false to refuse it

The profile decides. With hybrid the review runs by default, because a cloud reviewer earns the pass: on a deliberately damaged episode claude-sonnet-4.5 proposed 13 edits and all 13 survived the guards, in 18 seconds. With offline it stays off — the same run gave mistral-nemo:12b 133 seconds to find one edit, and that one was an invention. Whatever is written in config.yaml wins over both.

Edits are not taken on faith. Each one goes through the same fit ruler the translation and the length pass use: an edit that fits its slot worse than the old text is rejected, because trading accuracy for drift is a bad deal. And a review that rewrites more than half the lines is discarded whole - that is a retranslation, not an edit.

An edit that misses its slot is not thrown away in silence: the reviewer is shown its own proposal, its own stated reason and the exact shortfall in characters, and asked to fit while keeping the correction (fit_retries).

Measured on a real episode - 136 lines, the translation deliberately damaged by a weak model, reviewed by anthropic/claude-sonnet-4.5, 18 seconds:

Edits proposed	13
Accepted	13
Of those, recovered by the refit round	4

Among them: «ты действительно начал вести себя странным — что случилось с Эвей» → «начала вести себя странно — что случилось с Евой» (gender, grammar and the name's case in one line), «Я бы хотел их узнать» → «хотела», and «ты, наверно, прав» → «права» - the gender of the person addressed, inferred from neighbouring lines - plus six lines left untranslated.

The ruler went from ceiling to target and back, and the journal says why. The first version told the model "no longer than N characters", and it dutifully came back at half that: 27 of 46 proposals were discarded. Replacing the ceiling with a target and its tolerance bounds fixed that and created something worse: of 18 accepted edits on episode 3, every one was about length, and several padded the line with what had just been said — «Ты убил троих наших людей, помнишь?» became «Ты убил троих наших. Троих наших людей.» Forbidding it in the prompt changed nothing; the model obeys the data. So the reviewer is no longer told that a line is short at all: it gets max_chars and over, which is zero unless the line does not fit. The edits are language again — a calque undone, a filler cut, verb government fixed.

An undershoot is also priced lower than an overshoot when an edit is judged, at a third: a line that runs long forces S6 to speed it up or cut it and is heard on every viewing, while a line that ends early leaves a pause nobody notices. Pricing it at zero was tried and reverted — the reviewer then stripped content it judged redundant, and the fit fell from 72.4% to 60.5%.

The review reaches the subtitles too. The Russian subtitles are built from the same segments.json as the voicing, so corrected text lands in them by itself.

Choose a strong reviewer. The guards protect the timing, not the meaning: an invention that happens to fit the slot passes straight through, and there is no cheap mechanical check for it - telling that an edit is about something else requires understanding the text. Measured on the same material:

	claude-sonnet-4.5	mistral-nemo:12b	qwen3:14b
Edits proposed	13	1	5
Accepted	13	1	0
No-ops (text unchanged)	0	0	4
Time	18 s	133 s	196 s

The single edit mistral-nemo proposed was an invention: it replaced «Hey, I care about Ethan just as much as you do get a grip» with «Ну да, я был не в себе, но я не могу просто так взять и уйти» - and that passed the length check. qwen3:14b returned four no-ops and changed nothing at all.

So it is worth having the review done by a cloud model even when the translation runs locally: review.model is set separately from translate.model, and it picks its own engine by the name it is given — ollama:qwen3:14b is local, anything else goes to the gateway, the same rule dub compare uses. A reviewer that does not answer costs you the review, not the translation: the finished text is kept and the run says why it was not reviewed.

translate:
  engine: ollama              # the draft is written locally
  model: qwen3:8b
  review:
    enabled: true
    model: anthropic/claude-sonnet-4.5   # and read by a cloud model

It is one pass per film, cheaper than the translation itself.

The knobs live in config.yaml: which checks run (checks), the discard threshold (max_changes_share), and the pass size for very long films (batch_lines).

The gateway key

# Linux/macOS
export KILO_API_KEY="your-token"
# Windows PowerShell
$env:KILO_API_KEY = "your-token"

The variable name is set by kilo_gateway.api_key_env; the key itself never goes into config.yaml. In the app you paste it into a field and it is stored separately from the settings. The gateway routes /chat/completions only — it has no speech synthesis, and its transcription returns no timings — so the key is used for translation only. Recognition and synthesis never go through the gateway: they run on this computer or on your own GPU server.

Commands

Command	Purpose
dub process INPUT [--out FILE] [--out-dir DIR] [--subtitles] [--model ID] [--from-stage ID] [--to-stage ID]	process an input
dub fetch URL [--quality Q] [--audio-only] [--list-formats] [--all-playlist] [--playlist-items SPEC] [--cookies-from-browser B] [--out-dir DIR]	download a link, without dubbing
dub tools update [NAME]	update an external component (yt-dlp by default)
dub config init [--force]	create config.yaml from the example
dub cache clear [INPUT]	clear the stage cache
dub doctor [--fetch]	check and install dependencies
dub models [--search TEXT] [--free]	list translation models with prices
dub compare INPUT --models A,B [--apply M]	compare translation quality across models
dub ui [--port N] [--no-open]	graphical interface
dub voices list [--demo]	voices of the active TTS engine
dub export-srt INPUT [--lang en|ru]	export subtitles
dub evaluate INPUT [--golden FILE]	compare timings against a reference

Exit codes: 0 success, 1 stage error, 2 configuration error, 3 missing external dependency, 130 cancelled by the user.

Downloading from YouTube

A link does not have to be a preface to dubbing: dub fetch downloads and stops.

dub fetch "https://youtu.be/…"                  # best up to 1080p, mp4
dub fetch "https://youtu.be/…" --audio-only     # audio only — enough for subtitles
dub fetch "https://youtu.be/…" --list-formats   # what tracks exist at all
dub fetch "https://youtu.be/…" --all-playlist   # the whole playlist

What happens, step by step:

• The link is resolved before downloading. Title, channel, length and approximate size are printed, instead of the earlier hour of silence. A live stream is refused at once: it cannot be downloaded as a whole.
• Progress is visible while it downloads — percent, megabytes, time left. Ctrl+C interrupts it, the unfinished pieces are removed and the folder stays clean.
• The file lands in the working folder (or download.dir, or --out-dir), not in the cache: it is still wanted after the run — to watch, to re-run stages, to hand to another tool. The name is Title [id].ext: the id keeps same-named videos apart, and a repeat run recognises what is already there and does not download it twice.
• Tags and a cover are written into the file (download.embed_metadata, embed_thumbnail): title, channel, date and the source link, plus the picture, so a player shows a named file rather than a nameless one. Chapters for long lectures are a separate embed_chapters. The dub is signed as well: its title gets a “(RU)” mark and its comment carries the source link.
• Playlists under download.playlist: ask are treated as a single video, but the number of others is reported. all fetches the whole list, --playlist-items 1-3,7 fetches a selection. Only the first file is dubbed: the pipeline takes one input.
• Several links at once — one per line. The queue lives on the server, so the tab can be closed and the downloads continue. One runs at a time, so the connection is not split.
• Defaults live in the Downloads group of Settings: quality, container, folder, playlist mode, cookies, thumbnail and subtitles. The Downloads tab overrides them for a single run.
• Age-restricted videos and “confirm you are not a bot” need browser cookies: --cookies-from-browser chrome, or download.cookies_from_browser in config.yaml. That is account access, so it is off by default.
• Errors are explained. Instead of a tail of someone else’s log — “members-only video”, “not available in your region”, “the file is held by another process” — and what to do.
• yt-dlp is updated on demand: dub tools update yt-dlp; dub doctor warns when the bundled copy is more than 45 days old, since YouTube changes within that time.

Settings live in the download: section of config.yaml (quality, container, filename template, subtitles and thumbnail, cookies, fragment count).

Choosing a translation model

The model is set by translate.model, or once by --model. To choose with data rather than by gut feeling, run the same replicas through several models:

npx tsx src/cli.ts models --search claude   # what is available and at what price
npx tsx src/cli.ts compare video.mp4 --models anthropic/claude-sonnet-4.5,openai/gpt-5.2

The output is a summary plus a line-by-line comparison; --apply MODEL writes the variant you liked into segments.json without translating again. Local models join the comparison with an ollama: prefix. The app has the same thing on the Model comparison tab, plus a Test model button in the settings that translates three sample lines and reports whether the model answers at all, answers in Russian, and at what cost.

Character voices

Who is speaking is determined by pyannote (pyannote/speaker-diarization-community-1) through the python/diarize.py sidecar — on this computer (CPU, DirectML or CUDA, see Using the GPU) or on the GPU server. Speakers are named speaker_0, speaker_1… in order of first appearance; a replica where the speaker changes mid-phrase is split on a word boundary. One-time setup:

1. Python 3.10+ and pip install pyannote.audio (the app does this from its Download missing button).
2. A free Hugging Face account: accept the conditions on the model page, create a read token and paste it into the settings (Compute → Hugging Face token) or the HF_TOKEN variable. The token is only needed to download the weights (~30 MB); after that the stage works offline.

Without any of that, the stage is skipped with a stated reason and every line gets speaker_0.

The gender of each speaker's voice is estimated from the pitch of the original recording, so a speaker with no explicit assignment gets a voice of their own gender.

When the pitch says nothing, the text does. Pitch measurement starves on real audio: only 9% of frames yield a stable pitch, one speaker on episode 3 produced 0.62 seconds of it, and 171 Hz sits in the middle of the band where the detector refuses to answer. Russian states gender outright — «я пришёл» is a man speaking, «ты сказала» a woman being addressed. The clue stays independent: the translator is told the gender only of speakers whose voice is already decided — by the pitch or by hand. For the rest, the ones this clue is needed for, the gender in the text comes from the scene rather than from our own measurement; telling the translator would invent the very clue the gender is later judged by. It is collected after S3, when there is a text to read.

The two clues are not equally strong. First person speaks about whoever says the line. Second person speaks about whoever is addressed, and the pipeline does not always know who that is: it counts only when the line names a character the viewer has already labelled, or when exactly one other voice is present in the scene. A verdict needs two agreeing clues and none against it — one slip should not change a character's voice for the whole film.

The pitch stays a measurement; the text never overwrites it. Pitch is confident — pitch decides; pitch is silent — the text answers; they disagree — nobody decides, and the dispute shows in "Voices and characters" along with what it rests on. The review is given both, and told to leave gender alone where they disagree: it used to correct gender from a wrong measurement and damage text that was right.

The text is then made to agree with the voice. The model slips even when told the gender, and a male voice saying «я могла» is heard at once. So after translation the endings are changed to match the gender of the voice that will speak: first person to the speaker's voice («я могла» → «я мог»), «ты» to the voice of the person addressed when it is known — named, or the only other voice in the scene. It is done again before synthesis, because the voice may have been changed in the panel after translation, and the first person once more after shortening, because the model rewrote the line. The run reports how many lines were changed, with examples.

Where the engine's own voices have a measured pitch — as all the silero speakers do — the choice goes further than gender: an actress gets a voice at her pitch rather than "a female voice". On a real episode the two actresses sit at 195 and 176 Hz and are given speakers at 195 and 177; before that they shared piper's single female voice and blurred into one. A voice already handed out is not handed out twice. Piper's voices carry no measured pitch, so there the rotation by gender still applies. Explicit tts.voice_map entries and per-video edits always win over the automatic choice.

Two synthesis engines

	piper	silero
Russian voices	4, one of them female	34, nineteen of them female
Size	21 MB + ~60 MB per voice	92 MB for 29 voices, 138 MB for the 5 natives
Speed	0.45 s per line	0.03 s per line
Requires	nothing	Python with torch and soundfile
License	MIT	CC BY-NC-SA 4.0 (non-commercial)

Measured on the same 136-line episode: piper voiced it in a minute, silero in twelve seconds.

They also differ in how clearly they finish a line, and that was found by ear. Piper swallows a short final word of a short line: «Это всё твоё» comes out as «Это всё тво…», and so do «Это моё», «Это моя», «Это мой». The letter ё has nothing to do with it — «Это оно» and «Это она» are the same length and come out clean, and so does «Я сказал, что это твоё». What gets swallowed is a short possessive pronoun at the end. Silero says the same lines clearly. Checked against the audio: the clip is whole and neither alignment nor mixing cuts it — that is how the voice says it.

Pick one with tts.engine. Voice names do not overlap between the engines (ru_RU-irina-medium for piper, ru_zhadyra for silero), so change tts.default_voice along with the engine — otherwise the configuration is rejected and says so. The silero speakers read Russian as speakers of other CIS languages, and some of them have an accent. Piper stays the default because its voices are MIT-licensed, while the silero model is non-commercial.

To measure the pitch of your own samples with the same code the program uses on actors:

npx tsx scripts/voice-pitch.mts <directory with wav files> --match 195,176

Review and fix the result

The studio shows the current replica with controls for translation, speaker, voice and volume levels. The Replicas tab offers search and table editing, alongside the same video. Save changes writes edits without starting processing. Save and update dubbing saves and re-runs only the affected synthesis/alignment/mixing stages; volume-only changes need just a re-mix. Saved edits that have not reached a new output remain marked as pending even after restarting the app, and a failed launch can be retried.

Edits remain while moving between tabs and sections. Changing the video asks before discarding a draft. Invalid timecodes prevent saving; a running job prevents a duplicate launch. Per-video voice and mix overrides live in overrides.json, replica edits in segments.json, and the pending update marker in pending-review.json in the video's working folder. These files do not alter global settings. Subtitle drafts have a separate Save SRT action; rebuilding from replicas replaces their manual edits deliberately.

Configuration

config.yaml is created by dub config init from config.yaml.example, which documents every field. The main groups: profile, asr (model, language, VAD, diarization), translate (engine, model, batch size, length tolerance), tts (voice, voice_map, actors' voices), separation, alignment, mix, subtitles, download, server, cache.

Between stages you can edit segments.json by hand in the working folder and continue from any stage:

npx tsx src/cli.ts process video.mp4 --from-stage s3

The same folder holds run.log, a log of every run of that file. If a result looks strange, that is the first place to look.

Using the GPU

The two most expensive steps can run on the GPU, by two different mechanisms.

Recognition. whisper.cpp publishes no AMD builds, so the CPU build is chosen automatically. A Vulkan build can be selected by hand — asr.backend: vulkan — and the program says out loud that the archive comes from a third party. On a Radeon 780M an episode was recognised twice as fast: 2m18s against 4m57s. The text differs by about 9%: backends round differently, and on hard passages — shouting, singing — the decoding paths diverge.

Working out who speaks. Here 96% of the time goes into a single network, the speaker embedding: 105 seconds out of 108 on a five-minute clip. It is exported to ONNX and runs through DirectML on any DirectX GPU: 27 seconds against 122 on the CPU, four times faster, with results identical down to fifty milliseconds and the speaker label. The segmentation network stays on the CPU — it is tiny, and on the GPU it is ten times slower, because the transfers cost more than the arithmetic saves.

Separating voice from music. The same shape again: a heavy network next to light signal processing. The network moved to the GPU, the Fourier transform stayed in numpy. A minute of audio is processed in 3 seconds instead of 17 — five times faster — and the result matches the CPU one to within −94 dB, below the threshold of hearing.

Nothing needs configuring: with onnxruntime-directml installed the embeddings run on the GPU, otherwise the old path is used. The ONNX export happens once and is kept next to the weights.

The device is chosen per stage — asr.diarization.device and separation.device, sharing one vocabulary: auto, cpu, gpu, igpu, dgpu, and cuda, which means the same as gpu. There is deliberately no single "everything on the GPU" switch: the segmentation network is ten times slower there than on the CPU.

The speaker breakdown is also not recomputed for nothing. It depends on the audio and on its own settings, never on which model transcribed the words, so the result is stored with a fingerprint of the recording and reused. Switching the ASR model used to cost all seventeen minutes again; now it costs nothing.

dub doctor prints what runs where: the adapters it found, each stage and the device it will use, and what is missing to make something faster.

ROCm on Windows is neither needed nor advised for this: MIOpen compiles its kernels at run time, the ROCm packages ship no C++ standard headers, and the compilation fails on every kernel. That is an AMD defect, reproducible on supported cards too (ROCm#6150).

A GPU server

If there is a machine with an NVIDIA card in the house, the heavy networks of the pipeline can run there. The pipeline itself stays on the workstation — files, interface, ffmpeg, mixing — and the server takes:

Stage	How the server runs it
Recognition	faster-whisper (CTranslate2) in fp16, model from asr.model
Speaker diarization	the whole pyannote pipeline on CUDA
Voice separation	MDX-Net on onnxruntime with CUDA
Synthesis (silero)	the voice model stays on the GPU
Actors' voices	X-Voice, see below
Local translation	the server's Ollama, model from translate.model

The model name for local translation is the same field as without a server — the model list in the settings then comes from the server's Ollama. Only the untouched default, qwen2.5:7b-instruct, which is meant for an Ollama on this computer, is replaced with gemma4:26b-a4b-it-qat while translation is given to the server.

The service is python/dubpipe_worker.py. It is thin on purpose: it runs the same scripts as the local path, so the result does not depend on where it was computed — only the speed does.

Installing with Docker is the easiest: it needs the NVIDIA driver and the NVIDIA Container Toolkit (on Windows, Docker Desktop with WSL 2 passes the GPU through by itself). From the repository root:

docker compose -f docker/worker/compose.yaml up -d --build

The image is built once, in about half an hour, and takes about 22 GB: it holds two Python environments with their own CUDA libraries — the service's and X-Voice's, which are incompatible. Package versions are pinned (docker/worker/*.lock.txt), so the image is built from exactly what was tested on the server. Model weights are not in the image: X-Voice, its vocoder and the voice-check model (about 3 GB) download on first start, the rest on the first job; everything lives on the dubpipe-data volume and survives a rebuild. The port, actors' voices and the addresses of ComfyUI and Ollama on the same machine go into docker/worker/.env (see .env.example). No Ollama of your own — --profile ollama starts one next to it. The details — settings, updating, moving from the systemd unit, troubleshooting — are in docker/worker/README.md.

A GPU in the same computer as DubPipe uses the same container, with the address 127.0.0.1 in the settings.

Installing without Docker — one command on Linux with an NVIDIA card (sudo is needed for the systemd unit; --with-clone adds actors' voices):

scp -r python scripts/server/install-worker.sh user@server:dubpipe-worker/
ssh user@server 'bash dubpipe-worker/install-worker.sh'

Then point DubPipe at it — Settings → Compute → GPU server, or config.yaml:

server:
  host: 192.168.0.70

Everything else has a default: all six parts — recognition, diarization, separation, synthesis, actors' voices and local translation (server.stages) — go to the server, translation uses gemma4 26B-A4B, and when the server does not answer a stage runs here and says so in the report. dub doctor shows what runs where and whether the server answers.

A few decisions that are not visible from the settings:

• Recognition uses faster-whisper, not Speaches, even if Speaches is on the server: it does not let you turn off conditioning on the previous text, and with it one hallucination pulls in the next (see the hallucinations in How it works).
• GPU memory is freed when a job lacks it: the card is shared with ComfyUI, which keeps its last models there, and with Ollama, which holds gemma4 (15 GB) for minutes after translating. Without that a job does not fail — it quietly runs on the CPU, ten minutes instead of one for recognising an episode. The service unloads an idle ComfyUI, then the Ollama models; a running generation is never touched (server.free_gpu_memory). If the server still ends up on the CPU, the run report says so.
• Audio travels as FLAC, lossless and half the size of WAV: on a 100 Mbit/s link the network, not the GPU, is what separation waits for.
• GPU jobs run one at a time; the second waits in a queue, and the log shows it. Stopping a run also stops its job on the server.
• The service only answers private and loopback addresses — it has no password, like Ollama and Speaches next to it. A VPN on ordinary private addresses works; Tailscale's IPv4 addresses (100.64.0.0/10) do not count as private and are refused. Do not forward the port to the internet.

Actors' voices

A line can sound in the voice of whoever speaks it in the original: tts.clone.enabled: true or the "Actors' voices" switch under Voice. X-Voice does it on the GPU server (installed with install-worker.sh --with-clone).

• Fitting decides the line length, not the model. The base engine (silero or piper) stays: fitting uses its clips to decide how much time each line gets and shortens what does not fit. X-Voice voices the line in exactly that time, at its own pace, instead of being stretched afterwards. A line that fitting would have cut at its slot is spoken whole.
• The voice sample is one line of the character, taken from the separated voice track: about five seconds, dense speech, nobody else talking. A line with a long pause inside gives the model text that does not match the sound, and stray sounds creep into the start of phrases. The edges are cut at real pauses in the voice track, not at the recognizer's marks: those are off by tenths of a second, and a sample cut mid-word made the model finish that word at the start of every line. Stage S4 is recommended: without it the sample is taken from the original track, music and all, and the run warns about it. In the Voices and characters panel the sample can be played, replaced by another line, or cloning turned off for the character.
• Every line is checked (tts.clone.check, on by default). X-Voice slips on single lines, differently each time: it finishes a scrap of the sample at the start, or drops the last word. The server recognizes the finished clip, the program compares what was heard with the text and voices a failed line again — up to three tries, the best one stays. Slips of the recognition itself (Latin for Cyrillic, names, endings, interjections) do not count. Lines still wrong after three tries are listed in a warning with timecodes. On a FRANCE 24 clip (45 lines) one faulty line remained instead of four, and voicing took 26% longer; on the eighth episode 10 lines of 222 were voiced again, actors' voicing took 5 min 53 s.
• A refusal leaves no gaps. A line the server did not voice, and a character with no fit sample, speak with the base voice; the report says how many.

Measured on the eighth episode (27.6 min, 222 lines, CMP 50HX): 1.46 s per line, 6 minutes per episode; intelligibility by back-recognition 95.5% against 99% for silero. X-Voice slips on names and single words. X-Voice weights are CC BY-NC: non-commercial use only.

A GPU in the workstation itself needs no server: leave server.host empty. Diarization on CUDA moves the whole pyannote pipeline to the card, not only the embedding network, as on DirectML; silero picks CUDA by itself, and recognition uses asr.engine: faster-whisper (pip install faster-whisper) or the CUDA build of whisper.cpp, which is chosen automatically.

Performance

Measured on a Ryzen 7 8745HS with a Radeon 780M integrated GPU, on a 29-minute episode with Korean speech, large-v3 for recognition and Sonnet 4.5 for translation:

Stage	Time	Where
Recognition, large-v3	13 min 15 s	GPU (Vulkan)
Recognition, small	2 min 18 s	GPU (Vulkan)
Working out who speaks	2 min 14 s	GPU (DirectML)
the same on the CPU	17 min 32 s	CPU
Translation, 257 lines	6 min 47 s	the gateway, $0.70
the same on a cheap model	4 min 5 s	the gateway, $0.017
Separating voice from music	2 min 21 s	GPU (DirectML)
Synthesis	1 min 9 s	CPU (piper)
Fitting	39 s	CPU (ffmpeg)
Mixing and muxing	1 min 38 s	CPU (ffmpeg)
Whole run	23 min 9 s

The speaker breakdown is cached against the recording, so a second run of the same file skips it entirely. A whole run served from cache takes milliseconds. Subtitles for an already processed file are written just as fast.

With the GPU server (NVIDIA CMP 50HX, see A GPU server), a 27.6-minute English episode, large-v3 recognition, gemma4 26B-A4B translation on the same server, models warm:

Stage	Time	Where
Recognition, large-v3	1 min 0 s (46 s on the server)	server, faster-whisper
Speaker diarization	39 s	server, pyannote on CUDA
Translation, 223 lines	4 min 24 s	server, Ollama
Voice separation	1 min 58 s (65 s of compute)	server, onnxruntime CUDA
Synthesis	12 s	server, silero
Fitting	13 s	CPU (ffmpeg)
Mixing and muxing	1 min 41 s	CPU (ffmpeg)
Whole run	about 10 min

The rest of separation is the network: the workstation sits on 100 Mbit/s, and the film's tracks take longer to travel than to compute. gemma4 fits 31–43% of lines into their slot from run to run — on par with the earlier local models and well below the cloud Sonnet (72%) — so the hybrid profile keeps translation in the cloud and uses the server as its fallback.

Downloaded components and their licenses

DubPipe itself is MIT. It does not bundle or redistribute third-party binaries: they are downloaded from their upstream sources on first use and stay under their own licenses.

Component	Size	License
ffmpeg (gyan.dev build)	106 MB	GPL/LGPL
yt-dlp	17 MB	Unlicense
whisper.cpp (BLAS build)	20 MB	MIT
whisper small ggml model	465 MB	MIT
silero-vad (ONNX)	2 MB	MIT
piper	21 MB	MIT
piper ru_RU-* voices	~60 MB each	MIT / CC BY
silero v5_5_ru (5 accent-free voices)	138 MB	CC BY-NC-SA 4.0
silero v5_cis_base (29 voices)	92 MB	CC BY-NC-SA 4.0
pyannote diarization (optional)	38 MB	library MIT; speaker-diarization-community-1 weights CC BY 4.0, gated by a free Hugging Face account
MDX-Net separation model (optional)	67 MB	MIT

The GPU server downloads its own models, on that machine: the faster-whisper weights in CTranslate2 format (Systran/faster-whisper-*, deepdml/faster-whisper-large-v3-turbo-ct2) are MIT; X-Voice is MIT for the code and CC BY-NC 4.0 for the weights — non-commercial; its vocoder charactr/vocos-mel-24khz is MIT; the fastText language-identification model (the julien-c/fasttext-language-id mirror) is CC BY-SA 4.0.

See THIRD-PARTY-NOTICES.md for the details, including what this means if you package the application yourself.

On networked synthesis: there is no cloud synthesis. piper and silero run on this computer; silero and the X-Voice actors' voices can also run on your own GPU server. The unofficial edge-tts client was once listed in the settings but never implemented, and has been removed: using it violates Microsoft's terms and its limits move without notice.

Privacy and security

• The local API server listens on 127.0.0.1 only, on a random port, and every start issues a one-time token. Paths coming from requests are never read outside the working directory.
• The API key is never sent to the web page — only the fact that it is set, plus a masked form. Keys live in secrets.json inside the cache directory, separately from config.yaml, which stays safe to share.
• Without a GPU server, in the offline profile nothing leaves the machine once the models are downloaded. In hybrid, only the text of the replicas goes to the translation gateway — never audio or video.
• With server.host set, the stages given to the server send their data to your dubpipe-worker over plain, unencrypted HTTP on the local network: the audio track (as FLAC), the text of the lines and the voice samples for actors' voices. Local translation goes to that machine's Ollama. The worker deletes a job's files once they are fetched, but keeps the voice samples so they are not sent again. The Hugging Face token travels in the X-HF-Token header of diarization requests, only so the server can download the weights; the worker does not store it.
• Report vulnerabilities privately: see SECURITY.md.

How much disk this takes

The pipeline passes uncompressed audio between stages, so the working directory grows fast. The scratch tracks are deleted after mixing - 585 MB on a 15-minute episode - but each stage's inputs stay, so a second run does not redo the work. As an order of magnitude: a 15-minute episode holds about 660 MB, plus the shared models (whisper, pyannote, MDX, voices) at roughly 5 GB for all files together.

The current total is shown under Settings → System beside the clear button, and each file's own share on its page. Clearing never touches the downloaded programs and models.

To keep the scratch tracks around while working on the sound: cache.keep_intermediate: true.

Development

npm test              # the whole suite; no ffmpeg, no network and no keys required
npm run typecheck
npm run lint
npm run build
npm run test:e2e       # browser + Electron integration tests; see prerequisites below
npx electron scripts/screenshot-ui.cjs     # screenshots of every screen
npx tsx scripts/check-subtitles.mts FILE   # validate generated SRT files
node scripts/make-fixture.mjs              # rebuild the test clip and its reference

The test clip is synthesized locally with piper, so the fixture is reproducible and carries nobody's rights: every boundary in tests/fixtures/golden.json is constructed, not measured.

npm test runs 711 offline tests. The separate Playwright suite runs 37 integration scenarios against isolated media and configuration under output/playwright/runtime. It needs Chrome, FFmpeg and FFprobe (on PATH, or the project's downloaded Windows bundle), plus a current npm run build for Electron. It synthesizes short color/tone clips without piper or model downloads, checks real S1 extraction and S7 mixing, and mocks external downloads/model responses. Screenshots cover every section and all settings categories at 1400, 900 and 760 px under light and dark OS preferences; the application stays graphite in both. Reports, screenshots and failure traces are under output/playwright/ and are not committed. The existing CI runs offline tests, types, lint and build; Playwright is a separate local check.

Contributions are welcome — see CONTRIBUTING.md. Engineering decisions, measurements and the reasoning behind them are recorded in SPEC.md (in Russian).

Known limitations

1. Diarization does not separate overlapping voices: a replica goes to whoever speaks longer.
2. On-screen text and captions are not translated.
3. With separation.apply: everywhere, songs lose their vocals along with the speech. The default — removing the voice only under our own speech — leaves an undubbed song intact.
4. The GPU is used where it pays and only there: recognition through a Vulkan build of whisper.cpp (twice as fast, chosen by hand because the project publishes no AMD builds), speaker embeddings and voice separation through DirectML; with an NVIDIA card, faster-whisper and the whole of pyannote on CUDA, and silero takes CUDA by itself when it is there — here or on the GPU server. Fitting and mixing stay on the CPU. ROCm under Windows is not used — MIOpen fails to build its kernels, an open AMD defect.
5. Estimating a line's duration is a fitted model — a fixed cost per replica plus characters over a rate — measured per voice, not a constant. It is still an estimate: the exact fit is done by stage S6 through tempo, borrowed silence and, last of all, shortening.
6. Source languages other than English do run (whisper is multilingual), but several defaults assume Latin script: the translation length cap, word joining for languages written without spaces, and the subtitle line width.

License

MIT © 2026 Mikhail Sergeev.

The license covers this source code. Downloaded third-party components keep their own licenses, and the material you process keeps its copyright — see the personal-use notice at the top of this file.