ROCm/FastFlowLM v0.9.40

🚀 FastFlowLM v0.9.40 - Gemma4 E4B + Reliability Updates

on GitHub

📦 New Model Support

🌎 Gemma4-IT-E4B

FastFlowLM now supports gemma4-it:e4b for language, vision, audio workloads, including concurrent multimodal input for omni-model use cases.

• Tag: gemma4-it:e4b

Run in CLI mode:

flm run gemma4-it:e4b

Run in server mode:

flm serve gemma4-it:e4b

For more details, see the model card and benchmark results.

✨ Improvements

🔥 Performance Boosts for gemma4-it:e2b

This release brings meaningful speed improvements to the gemma4-it:e2b model:

• Prefill: up to 11.4% faster
• Decoding: up to 10.2% faster

⚡ Chunk Prefill

This release adds chunk prefill support, significantly reducing memory usage for long prompts and larger workloads.

You can configure the prefill chunk length with --prefill-chunk-len in both CLI and server modes. The default value is 4096.

Run in CLI mode:

flm run gemma4-it:e4b --prefill-chunk-len 8192

Run in server mode:

flm serve gemma4-it:e4b --prefill-chunk-len 8192

In server mode, you can now cancel a request even while it is still in the prefill stage. No more waiting around for a huge prompt to finish prefill: just hit the stop button in higher-level apps such as Open WebUI and move on.

🔐 Hash Checking

A new hash checking command is now available to help verify downloaded model files.

If you have trouble running a model and suspect a corrupted download, run:

flm check gemma4-it:e4b

If corrupted files are detected, you will see output like this:

[FLM]  Checking model: llama3.2:1b...

[FLM]  Checking file: config.json...
[FLM]  Fail!
[FLM]  Removing corrupted file: config.json...
[FLM]  Successfully removed config.json!
[FLM]  Checking file: model.q4nx...
[FLM]  Success!
[FLM]  Checking file: tokenizer.json...
[FLM]  Success!
[FLM]  Checking file: tokenizer_config.json...
[FLM]  Success!
[FLM]  Model check completed with errors. Please use `flm pull llama3.2:1b` to re-download corrupted files.

🐛 Bug Fixes

🛠️ Tool Calling

Fixed an issue where tool calls could return an incorrect finish reason.

Thanks to @antrv for reporting this issue!

⛓️‍💥 Empty Multimodal Input Handling

Fixed an issue where empty image or audio input in server mode could cause the server to break.

Thanks to @antrv for reporting this issue!

🧠 Memory Limits

Fixed a memlock limit issue that could affect loading ASR or embedding models standalone.

Thanks to @sofiageo for reporting this issue!

🌟 Summary

FastFlowLM v0.9.40 expands the Gemma4 lineup with gemma4-it:e4b. This release delivers meaningful speed improvements to gemma4-it:e2b, introduces chunk prefill for more efficient handling of long prompts, adds check command for verifying model files, and improves reliability across tool calling, multimodal input, and memory handling.