Details
mtmd: fix mel preprocessor in LFM2 audio (#29403)
which resulted in different greedy transcripts for 4.5% of English and 6.5% of Japanese
test utterances. In Japanese, some differences changed entire words.
This change:
- uses
log(x + 2^-24)instead of clamping to the log floor - uses a symmetric Hann window, equivalent to
torch.hann_window(periodic=False) - adds the normalization epsilon to the standard deviation instead of inside the square root
Only the lfm2a preprocessor opts into these behaviors. Other audio preprocessors are unchanged.
Tested on top of 84e76d8 using llama-server with CUDA and temperature=0, compared against
http://github.com/Liquid4All/liquid-audio fp32.
Test set:
- 200 LibriSpeech
test-cleanutterances (EN) - 200 Common Voice
jatest utterances (JP) - identical 16 kHz audio passed to both implementations
Greedy transcript identical to liquid-audio
| Without fix | With fix |
|---|---|---|
| EN F16 | 191/200 | 200/200 |
| JP F32 | 187/200 | 200/200 |
| JP F16 | 187/200 | 199/200 |
The remaining JP F16 difference is a comma and matches the reference implementation's own bf16
output.
Mel relative L2 error versus liquid-audio:
- EN: 3.2% -> ~2e-6 median
- JP: 3.9% -> ~2e-6 median
Website:
Attestations:
macOS/iOS:
- macOS Apple Silicon (arm64)
- macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED
- macOS Intel (x64)
- iOS XCFramework
Linux:
- Ubuntu x64 (CPU)
- Ubuntu arm64 (CPU)
- Ubuntu s390x (CPU)
- Ubuntu x64 (Vulkan)
- Ubuntu arm64 (Vulkan)
- Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries
- Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries
- Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries
- Ubuntu x64 (ROCm 10.0)
- Ubuntu x64 (OpenVINO)
- Ubuntu x64 (SYCL FP32)
- Ubuntu x64 (SYCL FP16)
- Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide
Android:
Windows:
- Windows x64 (CPU)
- Windows arm64 (CPU)
- Windows arm64 (OpenCL Adreno)
- Windows x64 (CUDA 12) - CUDA 12.4 DLLs
- Windows x64 (CUDA 13) - CUDA 13.4 DLLs
- Windows arm64 (CUDA 13) - CUDA 13.4 DLLs
- Windows x64 (Vulkan)
- Windows x64 (OpenVINO)
- Windows x64 (SYCL)
- Windows x64 (ROCm 10.0)
openEuler:
- DISABLED
- openEuler x86 (310p)
- openEuler x86 (910b, ACL Graph)
- openEuler aarch64 (310p)
- openEuler aarch64 (910b, ACL Graph)
UI: