Skip to content

audio-mel: FFT instead of the direct DFT in MelSpectrogram - #20

Merged
michalharakal merged 2 commits into
developfrom
feature/mel-fft
Oct 21, 2026
Merged

michalharakal merged 2 commits into
developfrom
feature/mel-fft

Conversation

@michalharakal

Copy link
Copy Markdown
Contributor

MelSpectrogram computed each frame's power spectrum with a direct DFT over precomputed cos/sin tables: O(n²/2) per frame and 640 KB of tables for Whisper's nFft = 400. This replaces it with an FFT.

What changes

  • PowerSpectrum (internal): a mixed-radix FFT for any frame size.
    • Radix 4, 2 and 5 butterflies are written out in closed form; radix 3 and larger factors use the plain sum. Whisper's 400-point frame is 4·4·5·5, so no power-of-two size is needed.
    • Twiddles are computed in double precision. The buffers are allocated per logMel call, so MelSpectrogram is still safe to share across threads.
  • Each mel filter is summed only over its non-zero band. This is exact, because the skipped terms are 0.

Numbers

Before After
JVM, one 4 s Whisper window 38 ms 5.8 ms
ART on an arm32 Android device (log-mel inside a Whisper tiny transcription, same transcript) ≈ 0.88 s ≈ 0.2 s
whisper.log_mel_spectrogram golden (MelSpectrogramTest) cosine 0.99999998, max abs diff 1.3e-5 cosine 0.99999999, max abs diff 1.5e-5

Tests
PowerSpectrumTest in commonTest, green on jvm and linuxX64:

  • the power spectrum against a double-precision direct DFT for 400, 512, every radix path, primes and 7·7
  • a pure tone lands in its bin
  • the factorization

The existing mel tests pass unchanged.

PowerSpectrum: a mixed-radix FFT for any frame size — closed-form radix 4, 2 and 5
butterflies, radix 3 and larger factors by the plain sum — so Whisper's 400-point frame
(4*4*5*5) needs no power-of-two size. Twiddles in double precision, buffers per call
(MelSpectrogram stays safe to share across threads), no more 640 KB cos/sin tables.
Each mel filter is summed over its non-zero band only (exact: the skipped terms are 0).

- JVM: 5.8 ms per 4 s Whisper window instead of 38 ms
- arm32 Android device (ART), Whisper tiny transcription: log-mel ≈ 0.2 s instead of
  ≈ 0.88 s, same transcript
- vs the whisper.log_mel_spectrogram golden: cosine 0.99999999 (was 0.99999998)
- PowerSpectrumTest (commonTest, jvm + linuxX64): power against a double-precision
  direct DFT for 400, 512, every radix path, primes and 7*7; a pure tone; factorization
@michalharakal
michalharakal merged commit 862d69f into develop Oct 21, 2026
1 check passed
@michalharakal michalharakal mentioned this pull request Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant