ROCm/FastFlowLM v0.9.10

🚀 FastFlowLM v0.9.10 — Enhanced Serving, Stability Fixes & Preemption

on GitHub

✨ What’s New

1. Request Queue in Server Mode

FLM now queues incoming requests in server mode to handle high traffic without overload. This ensures stable, orderly processing when multiple requests arrive at once.

• Default queue length: 10
• Customize: use --q-len (or -q)

flm serve llama3.2:1b --q-len 20

2. Customizable Socket Connections

You can now set the maximum number of concurrent socket connections to better control network resource usage.

• Default sockets: 10
• Customize: use --socket (or -s)

flm serve llama3.2:1b --socket 20

3. Preemption

Preemption allows high-priority tasks to interrupt ongoing NPU jobs, improving responsiveness for critical workloads.

flm serve llama3.2:1b --preemption 1

⚠️ Note: Preemption is for engineering testing/optimization only. It requires a special driver + toolkit and is not for public use.

4. CLI Improvements

New aliases make configuration faster:

• --ctx-len / -c → Context length
• --q-len / -q → Request queue size
• --socket / -s → Socket connection limit

5. Bug Fixes 🐞

• Fixed crash when checking version with flm --version / -v.
• Fixed error when using non-power-of-2 context lengths (--ctx-len).

🌟 Summary

This update makes FLM more robust under load, adds queueing and connection controls, introduces preemption (testing only), and fixes stability issues.