oMLX 0.8.0rc1

This is a release candidate. Please report any bugs through GitHub Issues. If you need a stable release, please use oMLX 0.7.3.

The release candidate for oMLX 0.8.0 adds the Prefill Accelerator, which lets a PC with NVIDIA GPUs or a second Mac speed up prefill for a big model on your Mac, and makes DeepSeek V4.1 faster with many requests and on clusters.

Download: macOS 26 / 27 | macOS 15 Sequoia

Prefill Accelerator with NVIDIA GPUs or a Mac

A PC with NVIDIA GPUs, a DGX Spark or a second Mac with a faster GPU runs the first layers of each prompt chunk, while your Mac runs the rest, decodes and keeps the KV cache. Auto picks how many layers the helper takes from the measured speed of both machines and the link, and keeps the prompt on the Mac when a split would be slower (#5065, #5163, #5195).

The CUDA helper builds on the great CUDA kernels of TensorFold by @ashhart and mlx-serve by @ddalcu, and would not have been possible without them. Thank you both for sharing them in the open.

Prefill tok/s on a 64K prompt, oQ4e models, M3 Ultra 512GB with a CUDA helper over 2.5 GbE. Decode speed stays about the same.

Turn on Global Settings > Advanced > Enable Distributed Inference, pair the helper in the Cluster tab, then pick it under Prefill Accelerator in the model's settings (guide). An NVIDIA PC needs only the command below and the code from docker logs omlx-helper, with no SSH or oMLX install. It needs driver 580 or newer and an Ampere or newer GPU or a GB10, and has not been tested on Windows yet.

docker run -d --name omlx-helper --restart unless-stopped --gpus all --network host -v omlx-helper:/data jundot/omlx-cuda-helper

Faster DeepSeek V4.1 with Many Requests and Clusters

With the DSpark drafter on, DeepSeek V4.1 now verifies the drafts of several requests together, so the drafter keeps its speedup as requests add up (#5154). Tensor-parallel clusters now run the same fast prefill and decode kernels as a single Mac, and pipeline clusters run the drafter for several requests (#5086, #5154).

New Features and Improvements

• Concurrent requests: the new Concurrent Request Priority setting decides what runs first when a prompt arrives while others stream. Its Balanced default keeps running streams at 10 tok/s or more and cuts the new prompt's time to first token by 26% (#5164).
  • A short request that arrives during a long prefill starts after the running chunk, so its first token comes in 4.4 s instead of 13.4 s (@arnavprabhu in #5160, #5175).
  • Each row on the dashboard has a stop button that ends one request without unloading the model (#5156).
• Speed:
  • Affine4 and Affine8 KV compression decode up to 58% faster at long context, and models with attention sinks like gpt-oss prefill about 4x faster (@avifenesh in #5095, #5088, #5101).
  • MiniMax M3.1 prefill on M5 runs on the tensor units, up to 43% faster per MoE layer at 64K (#5084, #5157, #5083).
  • Two long Qwen3.5 to 3.8 requests at once decode up to 30% faster (@StefVanG in #5090).
  • DeepSeek V4.1 with MoE expert offload runs the experts already in memory first, for 17% lower TTFT (@cropduster in #4959).
• macOS app:
  • Settings get a sidebar search, plus Web Search and MarkItDown in Integrations (@ngutech21 in #5111, #5172, #5107).
  • The model library list is simpler and shows aliases, which the copy button now copies (@ngutech21 in #5091).
  • The app speaks Italian (@fabriziosalmi in #4506, #5198), and window counts can show exact digits (@LXD-8 in #4641).
• Web UI: a new Theme Settings editor sets the dashboard's light and dark colors, checks contrast, drafts dark colors from light ones, and saves themes as JSON or images (#5037, #5038, @LXD-8 in #5097, #5098, #5102, #5113, #5114, #5115, #5117).
• GPU keep-warm: Keep GPU Awake After Requests in Advanced settings sets how long the GPU stays warm after the last request, 5 minutes by default (@parasew in #5085).
• Server: a long text prompt no longer stalls other streams while it is prepared, and plain streamed text skips the tool-marker scan (@Woody-Hu in #5140, @LXD-8 in #4945).
• Benchmarks: local accuracy runs take a Max Tokens Override so thinking models are not cut off (#5189), labels spell PP and TG (@LXD-8 in #4619), and pp result logs include gen_tps (@yanzhaohui1999 in #5126).
• Dependencies and builds: transformers up to 5.22 (#5131), dev tools as a PEP 735 dependency group (@ngutech21 in #5188), and CI builds and tests the macOS app (@ngutech21 in #5120, #5119, #5174).

Bug Fixes

• INT8 Activation Prefill stays off unless a model turns it on, with a "May reduce quality" note in Model Settings and its own SSD cache blocks (#5108, f61b29e).
• Unloading a model no longer affects the models still loaded, as the Metal wired limit stays in place (#5165).
• reasoning_effort: "none" and "off" turn thinking off, instead of mapping to low (@syntax922 in #5151).
• Tool calls are parsed more reliably, with tool names like read counted only in call position and schema keys in their usual order (@Flo5k5 in #4433, #5161).
• Tool call errors are clearer, with a coded error for an empty tool_calls finish and escaped model text in harmony logs (@joshuaswarren in #5139, @yanzhaohui1999 in #5125).
• Memory no longer grows with many distinct JSON schemas, as the compiled-grammar cache is capped at 1 GiB (#4321 with @yanzhaohui1999).
• Prefill Accelerator helpers recover instead of failing, with OOM step-down, layers handed back for long prompts, and retries when offline (#5180, d83d1d4, @yanzhaohui1999 in #5169).
• Clusters pass DeepSeek V4.1 tool arguments correctly, and the link gate names the slow link when it blocks tensor parallel (ca70d2b with @twiedenbein, @hellodk in #3024 with @HamedShams).
• Speculative decoding stays on its fast paths, with wide TurboQuant verify windows on the fused kernel and DFlash drafter context kept on exact prefix hits (#4866, 54f6797, 22f2c43 with @avifenesh).
• Qwen3.8 Flash-Next runs cleanly on M1 and M2, taking the MLX path instead of failing the fused kernel probes (@begcdn in #5089).
• VLM and audio requests no longer fail silently, with Nemotron Omni audio read from sound_clips, Florence-2 skipped at discovery, and realtime STT fixes (@shaanchander in #5162, @Woody-Hu in #5142, @Aminark in #5092, #5043).
• oQ handles more models, with oQe on Spark-X and loadable LFM3-Omni output (@Woody-Hu in #5184, @arnavprabhu in #5153).
• Model Settings saves work again after a draft is deleted, with profiles checking their drafts and rejected saves naming the field (@LXD-8 in #4401, #5105, #4513).
• Splash follows oMLX more closely, with its SSD cache off when oMLX's is off, MTP shown as unused, a clean stop and fixed bench runs (#5082, @yanzhaohui1999 in #5168, #5078, #4938 with @OmVidana).
• Integrations connect more reliably, with configured headers on MCP SSE servers and omlx launch codex finding newer ChatGPT app bundles without a slow DNS lookup (#5080, #5079, 9715639).
• Supervised logs stay small, as the console log under brew services and the macOS app keeps only warnings and errors (@zviratko in #5179).
• Web UI polish, with SSD totals that include unloaded models, paired stat tiles on phones, rounded panel corners and a heatmap legend that reads right in dark mode (#5063, @LXD-8 in #5123, @CamJN in #5159, @yanzhaohui1999 in #5144).

Highlights from 0.7.4.dev1

• Faster DeepSeek V4.1. Prefill is 43-48% faster on M5 Ultra (@junmo-kim in #5031), oQ4e builds on M5 can run INT8 Activation Prefill (#5058), and the first decode on a cold shader cache no longer waits on kernel compiles (@junmo-kim in #5025).
• Faster Qwen3.8 Flash-Next. FP16 checkpoints run the fused kernels, up to +62% decode (#5028 with @qoli), and back-to-back long prompts on 128 GB Macs stop slowing down (@yohann-bearzi in #4902).
• Codex and Copilot. omlx launch codex, codex_app and copilot hand the agent every chat model on the server, so /model switches between them (#5026 with @scaryrawr, @scaryrawr in #5066).

Benchmarks, the full feature and fix lists, and every contributor credit are in the 0.7.4.dev1 release notes.

Upgrade Notes

• Source installs: run make install after updating, which also rebuilds the native kernels MiniMax M3 uses on M5 (#5157). transformers up to 5.22 is allowed (#5131, #5004), jinja2 3.1.6 or newer is required, and ddgs moves to 9.16.1 (#5002). make dev installs the dev tools from a dependency group and needs pip 25.1 or newer (#5188).
• INT8 Activation Prefill: it is off until a model's settings turn it on (#5155). Models without saved settings that ran it in 0.7.4.dev1, such as DeepSeek V4.1 oQ4e builds on M5, prefill at 16-bit again.
• Concurrent requests: existing installs move to Balanced. decode_fairness: false maps to Prefill First and true to Balanced. Prefill Priority is now called Long Prompt Handling, with the same key and values (#5164).
• Prefill Accelerator: both Macs need the same oMLX version (#5065). A CUDA helper is checked per model family instead, so pull jundot/omlx-cuda-helper again to get helper updates (#5163).
• Logging: under brew services and the macOS app, the captured console log keeps only warnings and errors. Every record is still in ~/.omlx/logs/server.log (#5179).
• API: reasoning_effort: "none" and "off" now turn thinking off (#5151). A request stopped from the dashboard gets Request stopped by the server admin, with 409 for non-streaming (#5156). A trailing assistant message on /v1/messages is continued instead of starting a new turn (#4995), and an attachment with a supported extension is accepted whatever MIME type the client reports, except spreadsheets (#4887).
• Affine KV: decode with Affine4 or Affine8 KV cache compression is not token-identical to 0.7.3, so greedy output can differ (#5095).
• Integrations: omlx launch codex and omlx launch codex_app need Codex 0.148 or newer (#5026). omlx launch copilot writes the oMLX provider into Copilot's providers.json and needs Copilot CLI 1.0.83 or newer (#5066).
• Cluster: existing deployments pick up the new prompt cache slot count when they are activated again (#5062), and a DeepSeek V4.1 tensor-parallel model reloads when its A8 settings change (#5086).

New Contributors

@fabriziosalmi in #4506, @alexandergreil in #4673, @christianschmidt in #5014, @begcdn in #5018, @ianderse in #5050, @ska2704 in #5057, @StefVanG in #5090, @Aminark in #5092, @avifenesh in #5101, @syntax922 in #5151, @CamJN in #5159.

Full changelog