Conversation
The voice conditioning cache added with the bf16 work was unbounded and kept GPU tensors alive for every distinct voice ever used. Each new voice pinned roughly 200-300 MiB of VRAM, so a server that cycled through a few dozen predefined voices and clones exhausted a 12 GiB GPU and every request failed with "TTS engine failed to synthesize audio for chunk 1". - engine.py: make the cache an LRU bounded by tts_engine.voice_cache_size (default 8, 0 disables), store entries on CPU and copy to the device on a hit, evict the oldest beyond the limit, and release CUDA allocator blocks after each newly encoded voice. Add clear_voice_cache(). - config.py: add tts_engine.voice_cache_size default. - docker-compose.yml: set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. - docs: document the new key. Verified on an RTX-class 12 GiB card: VRAM climbs while the cache fills, then holds flat at ~6.4 GiB across 31 distinct voices and 3 clones. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGsQBKMnHzENiqR5oRQrSy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The voice conditioning cache introduced in 9d0b139 (
_conds_cacheinengine.py) is unbounded and holds GPU-resident tensors for every distinct voice ever used. On a long-running server each new predefined voice or clone permanently pins roughly 200-300 MiB of VRAM. After a few dozen voices the GPU is exhausted and every request fails with:Measured on a 12 GiB card with the current
main, short text, one request per voice:This is the same user-facing symptom reported in #27, though that issue predates the cache so it may have a different cause.
Fix
engine.py: the cache is now an LRU bounded by a newtts_engine.voice_cache_sizeconfig key (default 8,0disables caching). Entries are stored on the CPU and copied to the device on a hit, so the GPU only ever holds the active voice. Because bothConditionals.to()andT3Cond.to()mutate in place, the copy shallow-clones the containers first. After a cache miss the freed allocator blocks are handed back withtorch.cuda.empty_cache(). Aclear_voice_cache()helper is added for anyone who wants to wire it to the unload endpoint.config.py: default forvoice_cache_size.docker-compose.yml: setsPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, as recommended by the OOM message, to reduce fragmentation on long-running servers.README.md/documentation.md: document the new key.Verification
Rebuilt the container and swept 31 distinct predefined voices plus 3 clones through
/v1/audio/speechand/tts, watchingnvidia-smiafter each request. VRAM climbs while the cache fills to 8 entries, then stays flat:Cache hits still skip
prepare_conditionals, so repeat-voice latency is unchanged (~1.0-1.4 s for a short sentence on this card).🤖 Generated with Claude Code
https://claude.ai/code/session_01AGsQBKMnHzENiqR5oRQrSy