diff --git a/tools/paradee/README.md b/tools/paradee/README.md index 67f3b6ac6..791558ea9 100644 --- a/tools/paradee/README.md +++ b/tools/paradee/README.md @@ -83,7 +83,55 @@ front end. - **GGUF:** the expected sha256 of the F32 converter output. The converter is byte-reproducible. `--check-output` enforces this sha256. - **No F16 pin:** `--outtype f16` halves the file (16 MB) but fails the parity gate (end-to-end SNR - 16–23 dB), so it is not pinned or shipped. + 16–23 dB), so it is not pinned or shipped. See [F16 and v1.1](#f16-and-v11) for the full evaluation. + +## Memory + +`paradee_load_model` memory-maps an all-F32 GGUF (the converter's default) read-only and the layers read the +weights in place. The 32 MB of weights are then clean, reclaimable page cache rather than private memory. +`PARADEE_USE_MMAP=0` reads the file into private memory instead. Any other file (for example `--outtype f16`) +is read and widened to F32 at load, so it keeps a private F32 copy of every weight as well as the file data. + +On a Lenovo TB336FU (4 GB), synthesizing 19.8 s of speech with `paradee-tts` gives byte-identical audio either +way. Peak private memory is 170 MB mapped against 202 MB read, and peak file-backed memory is 34 MB against +3 MB. Activations make up most of the peak. + +## F16 and v1.1 + +Measured October 19, 2026 against the pinned v1.0 GGUF, with this directory's parity test, its reference dump, +and 40 more sentences (20 assistant-style prompts and 20 Harvard sentences). Those sentences were phonemized +with `tools/misaki-g2p` and transcribed with Phonon-2 (`tools/parakeet`) and with whisper.cpp `base.en`. + +| File | Size (deflate -9) | Parity test | Durations (40 sentences) | WER Phonon-2 / whisper | Log-mel vs F32 | RTF, TB336FU / M4 Max | +|---|---|---|---|---|---|---| +| F32 (shipped) | 32.1 MB (29.7) | pass, end to end 58–76 dB | – | 3.90% / 3.90% | – | 0.30 / 0.024 | +| `--outtype f16` | 16.2 MB (14.9) | fail: end to end 9.8–23 dB, stage errors up to 1.1e-2 × RMS | 1 of 40 shorter by 25 ms | 3.90% / 3.90% | 0.039 mean, 0.56 max | 0.32 / 0.024 | +| F16 decoder, F32 BERT and predictor (experiment, not in the converter) | 22.1 MB (20.4) | end to end 56–61 dB; fails the 1e-3 × RMS stage gate on `asr` and `dec` only | identical | 3.90% / 3.90% | 0.0026 | 0.30 / 0.024 | + +- The end-to-end error of `--outtype f16` comes from the BERT and the duration/F0 predictor. The decoder in F16 + costs little. F0 errors near 1e-3 accumulate in the harmonic source's phase, which the waveform SNR punishes + while the spectrum barely changes. +- F16 does not speed anything up here, because the runtime computes in F32. It does not save RAM either: an F16 + file cannot be mapped and is widened at load, so private memory goes up by the size of the file (about 16–22 MB). +- So F32 stays the shipped and pinned file. F16 would only make sense with layers that read F16 weights from + the mapping. The ggml conv1d path already converts its weights to a cached F16 copy (`cached_f16_weight`, + about 10 MB for the 3-D weights). +- **v1.1** ([sahilmahendrakar/Paradee-8M-v1.1](https://huggingface.co/sahilmahendrakar/Paradee-8M-v1.1) at + `330617ad`, Apache-2.0) is not a weight update this runtime can load. It keeps the text side (same + `text_side.pt`) and the decoder's AdaIN trunk, but it replaces the iSTFTNet generator and the phase-locking + filter with a frame-rate source-filter head (`training/scripts/vhead.py` at github.com/sahilmahendrakar/paradee + `606e16e1`). That head is ConvNeXt blocks at 80 fps that shape the STFT of a 64-harmonic sine bank and of + random noise, plus a 3 ms noise path. Its output is not deterministic, because it draws random noise. +- Upstream's fp32 ONNX graphs, one onnxruntime thread on the M4 Max, the same 40 sentences: + + | Graph | RTF | WER Phonon-2 / whisper | + |---|---|---| + | v1.0 | 0.051 | 4.16% / 3.90% | + | v1.1 | 0.010 | 3.90% / 3.12% | + + int8 graphs: v1.0 0.054, v1.1 0.011, with WER within one word of fp32. +- Running v1.1 natively needs a new decoder head (sine bank, ConvNeXt, two STFT/iSTFT sizes, a seeded noise + source) and a new parity gate that does not compare the noise. ## Parity gate diff --git a/tools/paradee/include/paradee.h b/tools/paradee/include/paradee.h index 49b38d157..42c43fa51 100644 --- a/tools/paradee/include/paradee.h +++ b/tools/paradee/include/paradee.h @@ -63,7 +63,9 @@ const char * paradee_status_str(paradee_status st) noexcept; // Load a Paradee GGUF written by convert_paradee_pt_to_gguf.py. The file must // declare general.architecture == "paradee"; every hparam the graph reads is -// required. Returns nullptr and sets `err` on failure. +// required. An all-F32 file is memory-mapped read-only and used in place +// (PARADEE_USE_MMAP=0 reads it into memory instead); other files are read and +// widened to F32. Returns nullptr and sets `err` on failure. paradee_model_ptr paradee_load_model(const std::string & gguf_path, std::string & err) noexcept; // 24000 for Paradee-8M v1.0 (0 for a null model). diff --git a/tools/paradee/src/paradee.cpp b/tools/paradee/src/paradee.cpp index a00d72323..aad224d29 100644 --- a/tools/paradee/src/paradee.cpp +++ b/tools/paradee/src/paradee.cpp @@ -37,6 +37,13 @@ #include #include +#if !defined(_WIN32) +#include +#include +#include +#include +#endif + namespace eliza_paradee { using eliza_kokoro::DecoderConfig; @@ -74,6 +81,8 @@ struct paradee_model { gguf_context * gguf = nullptr; ggml_context * file_ctx = nullptr; // tensors as stored ggml_context * ctx = nullptr; // all-F32 view the layers read + void * map_addr = nullptr; // read-only mapping of an all-F32 file (else null) + size_t map_size = 0; std::vector style_text, style_dec; std::mutex mu; }; @@ -83,11 +92,53 @@ void paradee_model_deleter::operator()(paradee_model * m) const noexcept { if (m->ctx && m->ctx != m->file_ctx) ggml_free(m->ctx); if (m->file_ctx) ggml_free(m->file_ctx); if (m->gguf) gguf_free(m->gguf); +#if !defined(_WIN32) + if (m->map_addr) munmap(m->map_addr, m->map_size); +#endif delete m; } namespace { +// PARADEE_USE_MMAP=0 reads the weights into private memory instead of mapping them. +bool paradee_use_mmap() { + const char * e = std::getenv("PARADEE_USE_MMAP"); + return !(e && (std::strcmp(e, "0") == 0 || std::strcmp(e, "false") == 0 || std::strcmp(e, "off") == 0)); +} + +// Map an all-F32 GGUF read-only and point every tensor of `ctx` (a no_alloc +// context from gguf_init_from_file) at its bytes in the mapping. The layers +// only read the weights, so the pages stay clean and the kernel can drop and +// re-read them under memory pressure instead of swapping a private copy. +bool map_f32_tensors(paradee_model * m, const std::string & path, std::string & err) { +#if defined(_WIN32) + (void) m; (void) path; (void) err; + return false; +#else + const int fd = ::open(path.c_str(), O_RDONLY); + if (fd < 0) { err = "cannot open '" + path + "'"; return false; } + struct stat st {}; + if (fstat(fd, &st) != 0 || st.st_size <= 0) { ::close(fd); err = "cannot stat '" + path + "'"; return false; } + void * addr = mmap(nullptr, (size_t) st.st_size, PROT_READ, MAP_PRIVATE, fd, 0); + ::close(fd); + if (addr == MAP_FAILED) { err = "mmap failed for '" + path + "'"; return false; } + m->map_addr = addr; + m->map_size = (size_t) st.st_size; + const size_t base = gguf_get_data_offset(m->gguf); + for (int64_t i = 0; i < gguf_get_n_tensors(m->gguf); ++i) { + ggml_tensor * t = ggml_get_tensor(m->file_ctx, gguf_get_tensor_name(m->gguf, i)); + if (!t) continue; + const size_t off = base + gguf_get_tensor_offset(m->gguf, i); + if (off + ggml_nbytes(t) > m->map_size) { + err = std::string("tensor '") + ggml_get_name(t) + "' runs past the end of the file"; + return false; + } + t->data = (char *) addr + off; + } + return true; +#endif +} + bool read_i32(gguf_context * g, const char * key, int & out, std::string & err) { const int64_t id = gguf_find_key(g, key); if (id < 0) { err = std::string("missing GGUF key ") + key; return false; } @@ -180,12 +231,37 @@ paradee_model_ptr paradee_load_model(const std::string & gguf_path, std::string err.clear(); try { paradee_model_ptr m(new paradee_model()); - gguf_init_params params = { /* no_alloc = */ false, /* ctx = */ &m->file_ctx }; - m->gguf = gguf_init_from_file(gguf_path.c_str(), params); + // Metadata first. An all-F32 file (the converter's default) is then + // mapped and used in place; anything else is read into memory and + // widened below. + gguf_init_params meta = { /* no_alloc = */ true, /* ctx = */ &m->file_ctx }; + m->gguf = gguf_init_from_file(gguf_path.c_str(), meta); if (!m->gguf || !m->file_ctx) { err = "gguf_init_from_file failed for '" + gguf_path + "'"; return nullptr; } + bool mapped = false; + if (paradee_use_mmap()) { + bool f32_only = true; + for (int64_t i = 0; i < gguf_get_n_tensors(m->gguf); ++i) { + f32_only = f32_only && gguf_get_tensor_type(m->gguf, i) == GGML_TYPE_F32; + } + if (f32_only) { + mapped = map_f32_tensors(m.get(), gguf_path, err); + if (!mapped && !err.empty()) return nullptr; + } + } + if (!mapped) { + ggml_free(m->file_ctx); + m->file_ctx = nullptr; + gguf_free(m->gguf); + gguf_init_params params = { /* no_alloc = */ false, /* ctx = */ &m->file_ctx }; + m->gguf = gguf_init_from_file(gguf_path.c_str(), params); + if (!m->gguf || !m->file_ctx) { + err = "gguf_init_from_file failed for '" + gguf_path + "'"; + return nullptr; + } + } gguf_context * g = m->gguf; { const int64_t id = gguf_find_key(g, "general.architecture");