ROCm/FastFlowLM v1.0.13

🚀 FastFlowLM v1.0.13 — AMD GPU Backend (Preview), Hy-Memory, MiniCPM-OCR & Faster Prefill

on GitHub

This release does five things:

• New AMD GPU backend (preview). On Windows, the new oga-hipep backend runs 13 models on the Radeon GPU, from Gemma 3 4B up to GPT-OSS 120B and Qwen3.8-Flash-Next 125B.
• New NPU models. hy-memory:30b-a3b, a MoE model built to drive memory frameworks, and minicpm-ocr:1.0, a 1B OCR model.
• Faster models. MiniCPM-V 4.7 1B prefills about 2× faster. Gemma 4 E4B prefills up to 2× faster on long prompts. MoE models prefill up to 2× faster on HRX. Hy-MT2, Nemotron 3.5 Bolt, Qwen3.6 MoE and Gemma 3 1B are faster too.
• A safer, stricter server. Browser requests from other machines are refused by default. Bad requests get a clear 400 instead of a crash, a 500, or a silent wrong answer.
• Many fixes. Request settings no longer leak between requests, vision models no longer answer about the wrong image, and 15 models get a decoding-layer rounding fix.

⚠️ Some of these changes can affect existing setups. See Before You Upgrade.

⚠️ Before You Upgrade

• Web UIs on another machine need --cors-origins. The server now only answers browser requests from local origins. A web UI served from another machine (for example Open WebUI at http://nas:3000) gets HTTP 403 until you start the server with its origin:

  flm serve qwen3:8b --cors-origins "http://nas:3000"
  flm serve qwen3:8b --cors-origins "*"      # old behaviour: allow every origin

  • Clients that send no Origin header are not affected. This includes curl, the OpenAI SDKs and a web UI's backend.
  • Pages opened from a local file need --cors-origins "null".
  • Tunnels and reverse proxies that keep their own Host header need their origin listed too.
  • Clients pointed at http://0.0.0.0:52625 keep working.
• qwen3.5-flash:2b is removed. Use qwen3.5:2b instead. It uses the same weights, so there is nothing new to download.
• Request settings no longer stick. Every request now starts from the model's defaults. If you relied on an earlier request's think, temperature or similar, send it with each request.
• Two models must be downloaded again. Older copies show as outdated and are pulled again.
  • minicpm-v-4.7:1b: its vision weights (about 1.1 GB) were re-converted.
  • hy-mt2-flash:1.8b: it moved to new Q4_K weights (about 1.8 GB). The new engine can't read the old ones.
• /v1/embeddings now uses the model you send. Before, the field was ignored. An unknown tag, or a tag that isn't an embedding model, gets 400.
• qwen3.8-mtp:27b only speculates on greedy requests. Its default requests now decode at normal speed. Send temperature: 0 (or top_k: 1) to keep speculative speed.
• Stricter HTTP errors. Some requests that used to return 200 now return an error:
  • Unusable images (GIF, http(s) URLs, images in system messages, corrupt files) get 400.
  • Failed prefill on /v1 endpoints now returns an error instead of a 200: a prompt longer than the context gets 400, and other prefill failures get 500.
  • Connections over the --socket limit get 503 instead of a connection reset.
• top_k: 0 or -1 now turns top-k off. Before, it was ignored and the model's default top-k stayed on.
• Seeded sampling now gives the same result on Windows and Linux. On Linux, a request with a fixed seed may give different text than in v1.0.12.
• Hy-MT2 replies without max_tokens are capped. A hy-mt2-flash:1.8b reply now stops at about twice the prompt length plus 64 tokens (finish_reason: "length"). Send max_tokens for longer replies.

🆕 New Models

🎮 AMD GPU Backend: oga-hipep (Preview, Windows)

FastFlowLM can now run models on the AMD GPU, not just the NPU. The new oga-hipep backend uses ONNX Runtime GenAI with AMD's hip-ep runtime. It is part of the standard Windows ZIP and MSI.

🧪 Preview. The oga-hipep backend is an early release. Model tags, options and behaviour may change in later releases, and it has the known limitations listed below. Please report problems on GitHub.

• Pull, run and serve these models like any other tag. Tags ending in -oga-hipep pick the GPU backend automatically.
• The first flm run or flm serve of a GPU model downloads the hip-ep runtime (about 137 MiB) into your user profile.
• The REST/SSE server, logprobs and streaming work as on the NPU. Tool calling works on the Qwen3.5/3.6/3.8 and Gemma 4 GPU models, and reasoning on the Qwen3.x, Gemma 4 and GPT-OSS ones.
• In multi-turn chats, the GPU models skip the part of the conversation they have already processed instead of starting over (Qwen3.x and GPT-OSS always; Gemma and Qwen2.5 when the previous reply comes back unchanged).
• Ctrl+C and /bye end flm and free GPU memory.
• qwen3.5-oga-hipep:9b on a Ryzen AI MAX+ 395 decodes at about 36 tok/s and prefills at about 1,150 tok/s.

All have a 32k default context.

flm run qwen3.5-oga-hipep:9b
flm serve qwen3.5-oga-hipep:9b

• Qwen3.8-Flash-Next has 125B parameters with 6B active. It decodes at about 34.5 tok/s, and needs about 78 GiB of GPU memory.
• Images share a budget of about one 1440×1440 picture per prompt. Several images are scaled down to fit.
• flm uninstall hipep removes the runtime that flm installed. It keeps your models and any runtime you installed yourself. Stop running flm processes first.
• HIPEP_PKG_ROOT points flm at a hip-ep package you installed yourself. When set, it is the only place flm looks.

⚠️ Limitations:

• Windows only. Not available on Linux or HRX builds.
• The machine must still have an NPU. GPU-only machines are not supported yet.
• Gemma and GPT-OSS GPU models are text only. Gemma requests with images or audio get HTTP 400.
• A new or changed image, or a Gemma 4 follow-up with thinking on, re-processes the whole conversation.
• On gemma4-it-oga-hipep:12b, a tool-call round trip (a reply after a tool result) can fail.

📖 CLI: backends

🧠 Hy-Memory 30B-A3B: hy-memory:30b-a3b

tencent/Hy-Memory-30B-A3B is a MoE model with about 3B active parameters. It is built to be the language model behind memory frameworks such as Mem0, MemOS and Hy-Memory.

• Takes one extract or reconcile prompt and answers in JSON.
• Text only. No thinking, no tool calling.
• Greedy decoding by default.
• The prefix cache reuses the shared part of repeated requests.
• Download: about 17.9 GB. 32 GB of system RAM or more is recommended.
• Speed on Strix Halo: decode 19.3 tok/s at 1k and 14.5 tok/s at 8k.

⚠️ Prefill runs at about 200 tok/s, so a 16k prompt takes over a minute to the first token. Keep prompts short, or reuse a shared system prompt through the prefix cache.

flm run hy-memory:30b-a3b

📖 Hy-Memory · Hy-Memory benchmarks

📄 MiniCPM-OCR 1.0: minicpm-ocr:1.0

A 1B OCR model that reads text from images. It pairs the MiniCPM-V 4.7 text model with a SigLIP vision encoder.

• Takes images and text.
• Supports tool calling and the prefix cache. No thinking.
• Context: 32k by default.
• Download: about 2.0 GB.
• Speed on Strix Halo: decode 65 tok/s at 1k and 47 tok/s at 24k.

flm run minicpm-ocr:1.0

✨ Improvements

🚀 Faster Models

MiniCPM-V 4.7 1B (minicpm-v-4.7:1b)

• Prefill is about 2× faster on XRT and 1.6–1.8× faster on HRX.
  • XRT at 1k: 1,545 → 3,337 tok/s.
• Decode is 18–44% faster.
  • XRT at 1k: 50.7 → 72.4 tok/s.
• The vision encoder is about 2× faster on requests with many images or video frames.
  • 120 frames: 14.3 s → 7.6 s.
• On Windows, prefill is now close to Linux speed.
  • Text, 3,359 tokens: 1,124 → 666 ms.
  • 32 images: 860 → 483 ms.
• ⚠️ Needs a re-download (see above).

Gemma 4 and Nemotron 3.5 Bolt

• gemma4-it:e4b prefill is much faster on long prompts. Decode is about 9% faster.
  • Prefill: 527 tok/s at 1k (was 419), 1,243 tok/s at 16k (was 573).
  • Time to first token on a 4k prompt: 6.2 s → 2.6 s.
  • Image: 1.9 s → 0.9 s to the first token. A 30-second audio clip: 2.7 s → 1.1 s.
• nemotron3.5-bolt:e4b and nemotron3.5-bolt-dflash:e4b prefill faster too (about 30–40% for Bolt). Decode is unchanged.
  • Bolt time to first token: 1.45 → 0.83 s at 1k, 3.57 → 2.51 s at 4k.
  • Images start about 20% sooner, audio 15–40% sooner.
• nemotron3.5-bolt-dflash-flash:e4b starts images and audio sooner. Its prefill is unchanged.
• gemma4-it:e2b is faster from 2k tokens up.
• No re-download needed.

MoE models on HRX

• gpt-oss:20b and qwen3.6-moe:35b-a3b prefill much faster on the HRX runtime. They now match or beat XRT.
  • gpt-oss:20b: 80.8 → 126.2 tok/s at 1k.
  • qwen3.6-moe:35b-a3b: 25.9 → 52.5 tok/s at 1k.
• qwen3.6-moe:35b-a3b also prefills faster on both runtimes: up to 22% on XRT and up to 18% on HRX.

Hy-MT2 (hy-mt2-flash:1.8b)

• New engine and new Q4_K weights. At 1k on XRT, prefill is about 1.5× faster and decode about 6% slower than in v1.0.12.
  • Prefill: 1,117 → 1,686 tok/s.
  • Decode: 52.9 → 49.8 tok/s. The new weights have a larger output layer.
• Inputs longer than 1k tokens are no longer cut off, and translations no longer stop early on inputs of a few thousand tokens.
  • Past about 4k–6k prompt tokens the model itself starts to drop text, so translate long documents in pieces of 3k–4k tokens.
• Default context is now 32k (was 1k).
• On Windows, prompts of 512 tokens or fewer start about 20× sooner.
• ⚠️ Needs a re-download (about 1.8 GB).

More

• gemma3:1b decodes 4–8% faster.
• qwen3.8-mtp:27b decodes about 2× faster without speculation. With response_format or tools and temperature: 0, it now keeps speculation on: a JSON reply takes 11.8 s instead of 28.9 s.
• qwen3.8-mtp:27b and qwen3.8-dflash2:27b take contexts up to 16,377 tokens (was 8k). Any --ctx-len from 8193 to 16384 used to fail to load. 16k uses about 0.5 GB more memory than 8k.
• Sampling is faster when top-k is off (top_k: 0 or -1): about 2.5 ms per token instead of 14 ms on a 152k vocabulary, and less with min_p. The same seed gives the same tokens as before.

🛠️ Server and CLI

• Raw RGB8 frames for minicpm-v-4.7:1b. Video pipelines can send decoded frames directly as an image_raw part on /v1/chat/completions. There's no image encoding on the client.
  • New flm serve options: --max-raw-frames, --max-raw-mb, --allow-raw-path DIR.
  • --allow-raw-path reads only regular local files inside DIR. Network and device paths are refused.
  • About 73 tokens per frame.
• flm bench downloads missing models first, like flm run and flm serve.
• Model extensions. The new --model-extension <extension.json> option loads externally built model extensions. They can add new model families, including embedding models, without rebuilding FastFlowLM. This is aimed at developers.
• Embeddings:
  • /v1/embeddings loads and downloads the requested embedding model on demand, even without -e.
  • embed-gemma:300m takes the optional task and dimensions request fields. task picks the prompt prefix (query, document, clustering, classification, …).
  • flm run and flm bench refuse embedding tags and tell you to use the server.
• WebP images sent as data:image/webp, and upper-case types such as data:image/PNG, now work.
• Shorter server logs. Long strings in logged requests (images, frames, audio) and JSON parse errors are cut short.
• Building on Windows. A new WinSetup.ps1 script sets up the tools and builds flm.exe in one step.

📖 Raw RGB8 frames · Request parameters · Build on Windows

🐛 Bug Fixes

Server

• Bad requests no longer take the server down.
  • A request body with invalid UTF-8 used to kill a server thread. After 10 of them the server stopped answering everything. It now gets a 400.
  • Malformed JSON gets 400 instead of 500.
  • A missing required field on /api/show, /api/generate, /api/chat or /api/embeddings could crash the server on Windows. It now gets a 400.
• Unusable images are refused, not silently dropped. Before, the request returned 200 and the model answered about an image it never saw. Now the 400 names the bad part.
• Connections over the --socket limit get 503 with Retry-After: 1 instead of a connection reset.
• Prefill failures on /v1 return a real error. Before, streaming clients saw an empty reply. A prompt longer than the context now gets a clear message.
• --cors 0 now really sends no CORS headers. Before, it still sent Access-Control-Allow-Origin: *.
• /v1/models/ and /v1/models?x=1 now work. 404 replies are JSON.

Request settings

• Settings no longer leak between requests. Before, think, temperature, top_p, top_k, min_p, the penalties, reasoning_effort (GPT-OSS, Hy-Memory) and Gemma 4's conversational_mode stayed applied for later requests from any client. For example, after one request with "think": true, later requests kept thinking.
• top_k: 0 or -1 turns top-k off. Before, it was ignored.
• A tiny temperature (below 1e-6) is treated as greedy. A value such as 1e-38 used to produce a wrong token.
• ignore_eos works with tools and response_format. The reply now runs to max_tokens.

Models

• Vision models no longer answer about the wrong image.
  • Asking about a second image of the same size could give an empty reply, a refusal, or a wrong answer.
  • Resending a conversation with a different picture in place of an earlier one gave an answer about the old picture.
  • Cached images and audio are now matched by their content. This affects the CLI and the server.
• Decoding-layer rounding fix for 15 models. Since v1.0.10, generated text could drift slightly from the reference result, usually a few hundred tokens into a reply. No re-download needed. Affected tags:
  • llama3.1:8b, llama3.2:1b, llama3.2:3b
  • qwen3:0.6b, qwen3:1.7b, qwen3:4b, qwen3:8b, qwen3-it:4b, qwen3-tk:4b, qwen3vl-it:4b, qwen3vl-flash:4b
  • phi4-mini-it:4b, nanbeige4.1:3b, hy-mt2-flash:1.8b, deepseek-r1:8b, deepseek-r1-0528:8b
• lfm3:1.7b no longer leaks tool-call tags such as </tool_call> into content. Tool calls with no arguments, and string arguments that contain XML, now work too.
• Gemma 4 tools with nullable parameters work. A parameter typed ["string", "null"], common in MCP tools, made every request fail with a 400.
• Gemma 4 puts audio after the text, as the model card recommends.
• Hy-MT2 no longer loops on junk input. Repeated characters or base64 used to run until the context was full. The reply now stops on repetition, or at about twice the prompt length plus 64 tokens when the request sets no max_tokens. Both stops report finish_reason: "length".
• MiniCPM-V requests with too many pictures (over about 17,300 picture tokens) get a 400 instead of a 500.

CLI

• Linux flm run accepts non-ASCII input. Cyrillic and other non-ASCII characters were dropped.

🙌 Community Contributions

Thank you to our contributors:

• @NVolcz made request settings reset on every request (#978), fixed Gemma 4 nullable tool parameters (#977), and moved Gemma 4 audio after the text (#979).

And to everyone who reported the issues fixed in this release:

• @tranrn reported the Linux CLI dropping Cyrillic input (#964).
• @mitar reported qwen3.8-mtp turning speculation off for response_format and tools, and failing to load with contexts above 8k (#1016, #1001).

🌟 Summary

Thanks for your support — see you in the next one! 🚀