ROCm/FastFlowLM v1.0.12

🚀 FastFlowLM v1.0.12 — Speculative Decoding, MiniCPM-V MoE & Less RAM

on GitHub

This release does four things:

• New models. MiniCPM-V 4.7 35B-A3B, two more Decision 3.0 sizes (d3:edge, d3:flash), and qwen3.5-flash:2b.
• Faster decoding. New DFlash speculative-decoding tags for Nemotron 3.5 Bolt (up to 1.9× faster) and Qwen3.8-27B. qwen3.8-mtp:27b itself decodes 1.6× faster. The Whisper encoder is about 3× faster.
• Less host memory. Gemma 4, Nemotron 3.5 Bolt and Qwen3.5 (0.8B/2B/4B) read their embedding tables from disk instead of copying them into RAM. That saves up to 3.7 GiB per model.
• Many server fixes. These include a server that stopped accepting connections after clients hung up, Qwen3.5 answering ////// after hitting the context limit, wrong vectors from embed-gemma, and Gemma 4 tool-call parsing.

🆕 New Models

🖼️ MiniCPM-V 4.7 35B-A3B: minicpm-v-4.7-moe:35b-a3b

openbmb/MiniCPM-V-4.7-35B-A3B is a Mixture-of-Experts vision model: 35B parameters, about 3B active per token.

• Takes text and images.
• Thinking can be turned on. It is off by default, as the model card recommends.
• Supports tool calling and the prefix cache, pictures included.
• Context: up to 256k tokens (32k by default).
• Needs about 22 GB of memory.

flm run minicpm-v-4.7-moe:35b-a3b

⚡ DFlash Speculative Decoding: nemotron3.5-bolt-dflash:e4b, qwen3.8-dflash2:27b

A DFlash draft model proposes up to 7 tokens at a time, and the NPU checks all of them in one pass. The output follows the main model's own distribution.

• Two-package tags. A DFlash tag downloads only its small draft (0.27 GB for Bolt, 1.4 GB for Qwen3.8) and reuses the main model's package. flm pull downloads whichever is missing. flm rm removes only the draft, and warns when another tag still needs a base model.
• Everything else works as on the main model: thinking, tool calls, vision, audio (Bolt) and the prefix cache.
• With a long context, qwen3.8-dflash2:27b is about level with qwen3.8-mtp:27b.
• Set FLM_DFLASH=0 to turn speculation off and compare against the main model.

flm run nemotron3.5-bolt-dflash:e4b
flm run qwen3.8-dflash2:27b

📖 Nemotron Bolt DFlash · Qwen3.8 benchmarks

⚡ New Flash Models: qwen3.5-flash:2b, nemotron3.5-bolt-dflash-flash:e4b

Flash models are built for single-turn requests: a fixed 1k context, a clean state for every request, and a system prompt that is saved once and reused. That makes the first token come back faster.

• qwen3.5-flash:2b runs the same weights as qwen3.5:2b, so there's nothing new to download. Time to first token is 0.49 s instead of 0.81 s at 1k tokens. Decode is 33 tok/s instead of 29. Large pictures need -r 1.
• nemotron3.5-bolt-dflash-flash:e4b combines Flash serving with DFlash speculation. It uses the two packages above, so there's no new download. Time to first token is 0.32 s at 64 tokens, and decode is 33–40 tok/s on code.

Flash models don't support tool calling or continue_final_message.

📖 Flash Models · Flash benchmarks

🧭 Decision 3.0: d3:edge and d3:flash

Two more sizes of the d3 decision model, served with flm serve --decision <tag> on /v1/systemone. They take text and images, like d3:lite, d3:nano and d3:mini.

• d3:edge is the smallest and fastest d3 model.
• d3:flash sits between d3:mini and d3:turbo.

📖 d3 models · d3 benchmarks

✨ Improvements

🚀 Faster Models

• qwen3.8-mtp:27b decodes 1.6× faster, from 1.77 to 2.84 tok/s, thanks to new layer kernels.
• The Whisper encoder is about 3× faster: 0.6 s per 30-second window instead of 1.7–2.0 s. A 5-minute recording now takes 25 s instead of 45 s.
  • ⚠️ A Whisper-only server now uses about 2.3 GB of memory instead of 1.0 GB. Set WHISPER_CACHE_DEQUANT=0 to get back to 1.0 GB.

📖 Whisper benchmarks

💾 Less Host Memory

Token and per-layer embedding tables are now read from model.q4nx through a read-only memory map, one row per token, instead of being copied into RAM. Output, decode speed and time to first token are unchanged.

The catalog footprints of these models are lower too, so tools that filter models by memory (such as Lemonade) list them on more machines. qwen3.5:9b and other models are unchanged.

🛠️ Server and CLI

• ignore_eos and min_tokens are now applied on the OpenAI and Ollama endpoints. Before, they were accepted but ignored.
• One timing line per request. flm serve logs a Request done: line for every finished request, streamed or not. It shows prompt and generated tokens, prefill and decode speed, cached tokens and the finish reason:

  [FLM]  Request done: prompt 11 tokens (prefill 0.442 s, 24.88 tok/s), generated 10 tokens (decode 0.424 s, 23.61 tok/s), kv 21/32768, finish stop
  

• -a and -e take a model tag, so you can run a standalone ASR or embedding server. 1 / 0 still work as before.

  flm serve -a whisper-v3:turbo          # ASR only
  flm serve -e embed-gemma:300m          # embeddings only
  flm serve qwen3:0.6b -a whisper-v3:turbo

• The flm launcher on Linux works through a symlink, such as ~/.local/bin/flm.

🐛 Bug Fixes

Server

• The server no longer stops accepting connections after clients hang up. Each dropped request leaked a connection slot. After 10 of them, flm serve rejected everything, even GET /v1/models, until it was restarted.
• An unknown model name returns 404. Before, /v1/chat/completions returned a 200 with an error body and /v1/completions returned a 500.
• /v1/embeddings returns proper error codes. Bad input gets a 400 and an engine failure gets a 500. Before, both came back as a 200.
• Malformed UTF-8 no longer fails a request. A model that writes a broken multi-byte character (seen on gemma3:1b with CJK text) used to get a 500, or an error in the middle of a stream. The broken bytes are now dropped.
• min_p accepts fractions. A value such as 0.05 was rounded down to 0.

Models

• embed-gemma:300m returns correct embeddings again. Since v0.9.45, its vectors were unrelated to the model's, changed between server starts, and some requests hung for 60 seconds or more.
• Qwen3.5 no longer answers ////// after hitting the context limit. When a reply ran into the context limit, the engine was corrupted and every later request returned ////////// until a restart. Similar overruns are fixed for Qwen3.5-Omni, gemma4-it:12b, lfm2.5-tk, qwen3-tk, Qwen3.8-MTP, the DFlash tags and the shared decode loop. A reply that reaches the limit now ends with finish_reason: length.
• Gemma 4 tool calls:
  • A string argument that contains a quote, such as delimiter=",", no longer splits the call in two.
  • gemma4-it:12b with thinking off no longer leaks tool calls into content or runs to max_tokens.
• Flash models no longer reuse a stale system prompt. qwen3vl-flash, gemma4e-flash and nemotron3.5-bolt-dflash-flash could give a different answer to a repeated request, or see the system prompt twice.

🙌 Community Contributions

Thank you to everyone who reported the issues fixed in this release:

• @Jcollier0120 proposed memory-mapping the embedding tables (#807).
• @zeddius1983 asked for a per-request timing log (#841).
• @chetan-metrum-ai reported ignore_eos and min_tokens being ignored (#881).
• @LalaDango reported the gemma4-it:12b tool-call leak and the min_p rounding (#886, #887).
• @Matzebhv reported and narrowed down the Qwen3.5 ////// bug (#893).
• @markl-a reported Gemma 4 tool calls splitting on quotes (#900).
• @rizno asked for symlink support in the Linux launcher (#912).
• @mariolpantunes reported the /v1/embeddings failures (#939).

🌟 Summary

Thanks for your support — see you in the next one! 🚀