ROCm/FastFlowLM v0.9.21

🦃 FastFlowLM v0.9.21 — Thanksgiving LLaMA Turbocharge Release

on GitHub

Happy Thanksgiving!

Today we’re dropping one of our biggest speed upgrades ever for LLaMA and DeepSeek models (our first batch of models) — just in time for the holiday break. Fire up your Ryzen™ AI NPU and enjoy some seriously boosted performance. 🔥

🔄 1. Quantization Upgrade

• All models migrated from AWQ to Q4_1
• Better LLM accuracy and quality.

⚡ 2. Massive Decoding Speedup

• llama3.2:1b: ~50% faster decoding, reaching 66 tps
• llama3.2:3b: ~40% faster decoding, reaching 28 tps
• llama3.1:8b: ~40% faster decoding, reaching 13 tps
• deepseek-r1:8b: ~40% faster decoding, reaching 13 tps

🚀 3. Prefill Phase Optimized

• Slight improvements to prefill speed of all above, especially impactful for large context initializations.

🎙️ 4. Standalone Whisper ASR Server

You can now serve Whisper (OpenAI’s ASR model) as a standalone model for speech transcription — or pair it with GPU LLMs in a hybrid pipeline.

Use either:

flm serve -a 1

or

flm serve --asr 1

This release wraps up a bundle of performance gifts for LLaMA models on FastFlowLM.

Thank you for being part of the FastFlowLM journey — and happy Thanksgiving! 🦃🔥