⚠️ A bug affecting the gpt-oss:20b model was fixed in v0.9.30. Please use v0.9.30 or later.
📦 What’s New
⚡ Massive Prefill Speed Upgrade
We introduced a new attention engine that dramatically accelerates prefill, with larger gains at longer context lengths (especially 16K+).
- Up to 2.6× faster prefill
- More speedup at longer prompts
- No model re-download required (seamless upgrade)
📈 Prefill Speed with 32K prompt (tok/s)
| Model | Before → After | Speedup |
|---|---|---|
| lfm2:1.2b | 1059 → 1916 | 1.8× |
| lfm2:2.6b | 654 → 1053 | 1.6× |
| lfm2-transcript:2.6b | 654 → 1053 | 1.6× |
| lfm2.5-it:1.2b | 1059 → 1916 | 1.8× |
| lfm2.5-tk:1.2b | 1059 → 1916 | 1.8× |
| llama3.2:1b | 577 → 1157 | 2.0× |
| llama3.2:3b | 214 → 500 | 2.3× |
| llama3.1:8b | 167 → 281 | 1.7× |
| deepseek-r1:8b | 167 → 281 | 1.7× |
| Phi4-mini-it:4b | 173 → 447 | 2.6× |
🔜 Prefill speed upgrades for other models are on the way too — stay tuned!
📊 Detailed Benchmarks
🩹 “No Pain” Update 😎
Just update and run — long-context prefill is now much faster. 🚀
🛠️ Tool Call Bug Fix (Non‑Stream Case)
We’ve squashed a pesky bug affecting tool calls when not using streaming.
Now it behaves exactly like you thought it should. 😌
Thanks for being awesome!
Happy modeling 🤖💙
— The FastFlowLM Team