Skip to content

perf(audio): optimize cross-platform processing and complete native audio settings - #1258

Open
aeoform wants to merge 8 commits into
Open-Less:betafrom
aeoform:fix/android-audio-performance
Open

aeoform wants to merge 8 commits into
Open-Less:betafrom
aeoform:fix/android-audio-performance

Conversation

@aeoform

@aeoform aeoform commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Problem and behavior

Microphone enhancement was repeatedly allocating temporary buffers and calculating sinc kernels per output sample. This increases processing cost, especially alongside local ASR on mobile devices.

Reuse shared processing buffers and cache bounded rational resampling phases. Preserve the original filter, including at native 48 kHz, to keep the denoiser input unchanged. Add baseline x86-64 SSE and ARM64 NEON projection kernels with separate multiply/add and unchanged accumulation order. Software DSP stays at 48 kHz; ASR still receives 16 kHz mono PCM16. Android requests 48 kHz capture and retains a native-format CPAL fallback. Both egui and Tauri expose all six EQ presets and the expanded parameter ranges.

Complete MicYou audio-settings parity with six EQ presets, expanded parameter ranges, optional Android session-bound system AEC/NS/AGC, and opt-in raw/processed FFT monitoring. Monitoring is throttled and expires when the UI stops renewing it. Capture queues are not enlarged. AEC7 remains unexposed without a synchronized speaker reference; playback-buffer tuning is not applied to ASR. Preserve the pinned upstream importer and refresh its reviewed adapter patch.

Validation

  • Shared audio regressions: exact duration, irregular callback boundaries, short recordings and flush, dry/wet alignment, input sanitization, alias rejection, conservative AGC, and preserving the 48 kHz input filter for every denoiser.
  • Differential resampling oracle, scalar/SIMD projection equivalence including tails, and 17 DSP/model/monitoring tests.
  • Core/DSP/inference Clippy with warnings denied; frontend build and contract/component tests; native recorder worker regressions.
  • Android arm64 full Rust library check; inference-module cross-checks for Windows x64, Android arm64, and Harmony arm64.
  • Release probe feeds 30 seconds of 48 kHz synthetic audio and checks exactly 960000 output bytes for all four modes. Local comparisons show lower steady processing cost in every mode; host-load variation prevents claiming a fixed speedup. The reproducible audio_performance example is included.

Additional platform UI and numerical comparison

Add native iOS audio settings for the enhancement toggle, all software denoisers, gain/AGC, gate, comb filter, ten-band EQ/preamp, six EQ presets and processing order. Connect these persisted settings to the native Rust recorder with session-frozen processing and a stop/drain flush; default enhancement remains off. Writes are blocked while recording. Tauri (Windows/macOS/Android), Harmony's shared settings, and Linux egui retain their applicable settings and presets. Android system effects remain platform-specific.

Reuse input conversion scratch in the external PCM recorder and bounded, exact-capacity output PCM scratch in the native iOS Swift actor. Add a frozen Swift converter oracle and byte-for-byte tests over up to 32 format/ownership/packetization combinations and flush. CI exposed pre-existing frame loss in the original variable-packet converter (15040 instead of 16000 samples for 44.1 kHz); honor requested input frame counts with a cursor and drain available output to avoid dropping packet suffixes. Both fixed and variable new packetizations must match the complete, original fixed-100ms waveform, with the duration assertion retained. These Apple SDK tests require CI execution, not claimed as locally passed.

Independently built the same audio_equivalence probe against baseline f0744e10 and the optimized Rust sources. On Linux x86-64 all 48 synthetic cases produced byte-identical PCM: 0 differing cases, 0 LSB maximum error, identical lengths. Cases cover 16/44.1/48 kHz, mono/stereo, all four denoiser modes, combined effects, silence/noise/tones/impulses, irregular chunks and a partial final frame. Per-case hashes and diagnostic timings are in scripts/fixtures/audio-equivalence-2026-10-24.json. The comparator's negative controls reject a one-LSB mutation and truncation. External recorder regressions and strict Core Clippy passed locally.

No real-device test, listening study, speech-recognition quality evaluation, or battery measurement was performed. Android system-effect quality remains OEM-dependent. Native Apple changes require Apple CI validation. Existing CI workflows are unchanged.

@H-Chris233 H-Chris233 added area:asr ASR area area:recording Recording area area:ui UI area enhancement New feature or request large-change Large or risky change needs-review Needs review needs-tests Needs tests P2 Medium priority platform:android Android-specific platform:linux Linux-specific platform:macos macOS-specific platform:windows Windows-specific labels Oct 23, 2026
@aeoform aeoform changed the title perf(audio): optimize shared DSP and complete egui/Tauri audio settings perf(audio): optimize cross-platform processing and complete native audio settings Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:asr ASR area area:recording Recording area area:ui UI area enhancement New feature or request large-change Large or risky change needs-review Needs review needs-tests Needs tests P2 Medium priority platform:android Android-specific platform:linux Linux-specific platform:macos macOS-specific platform:windows Windows-specific

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants