github jundot/omlx v0.8.1
0.8.1

one hour ago

oMLX 0.8.1

oMLX 0.8.1 adds Hy4, Hy4-flash and Inkling 2 Mini, lets you apply a LoRA adapter from a model's settings, and makes MTP verify on M5, GLM-5.3 with several requests and new questions on a cached document faster.

Download: macOS 26 / 27 | macOS 15 Sequoia

New Models: Hy4, Hy4-flash and Inkling 2 Mini

Tencent's Hy4 and Hy4-flash run with image input, Lightning MTP and oQ/oQh, and their FP8 releases load as is (#5463). Lightning MTP takes Hy4-flash FP8 from 23.1 to 37.7 tok/s at 32K (+63%). Ready-made oQ4h builds are Jundot/Hy4-flash-oQ4h-mtp (132 GB) and Jundot/Hy4-oQ4h-mtp (452 GB).

M3 Ultra 512GB, 32K prompt, Lightning MTP on. MTP decode moves with draft acceptance, so single runs can differ.

Model Build Prefill tok/s Decode tok/s
Hy4-flash oQ4h-mtp 749 42.7
Hy4-flash FP8 720 37.7

Inkling 2 Mini loads with text and image input, DSpark works with it, and oQ can quantize it (#5468, 13fd0c9). Ready-made oQ3e and oQ4e builds are mlx-community/Inkling-2-Mini-oQ3e-mtp (13.7 GB) and mlx-community/Inkling-2-Mini-oQ4e-mtp (16.9 GB). The Prefill Accelerator supports only Inkling 2 Small for now.

LoRA Adapters

Your oMLX can now run LoRA adapters such as Qwen3.8-27B-opencode-adapted and Qwen3.8-27B-pi-adapted.

Pick one in Model Settings > Advanced, in the web dashboard or the macOS app (#5449). PEFT and mlx-lm adapters both work. The adapter is applied on top of the model at load, so it also works on oQ, oQh and Lightning MTP models. Turn on "Expose as model" in a profile to serve the adapter under its own id, <model>:<profile>.

Adapter folders in your model directories show up in the picker, not in the model list, and the picker says when an adapter does not fit a model. DFlash, VLM MTP, ANE prefill, MoE expert offload and the Prefill Accelerator cannot be used with an adapter yet.

New Features and Improvements

  • Speed:
    • MTP verify on M5 is faster with many requests: Qwen3.8 27B with 8 requests goes 308.3 -> 353.7 tok/s (+15%) as oQ4e and 210.9 -> 265.6 tok/s (+26%) as oQ8e on M5 Ultra (@Li-Lee in #5401).
    • GLM-5.3 Flash decodes 43.0 -> 58.5 tok/s in total with 2 requests (+36%) and 63.2 -> 81.8 tok/s with 4 (+29%) (#5414).
    • A new question on a cached long document starts sooner on hybrid models, with TTFT 16.4 -> 2.9 s at 245K tokens on Qwen3.8 27B (-83%) (@avifenesh in #5251).
    • A thinking budget that does not fire no longer slows MTP decode, 46.3 -> 51.5 tok/s (+11%) (@dkta0 in #4926).
    • DeepSeek V4.1 sampled decode with Engram on SSD goes 48.0 -> 54.5 tok/s (+14%) on two Macs in tensor parallel (@Jiffywu in #5274).
    • DeepSeek V4.1 with expert offload frees expert slots when a prefill runs short of memory, so a 64K prompt's TTFT goes 120.3 -> 85.6 s (-29%) (@williamxie1989 in #5406).
    • MoE verify on Qwen3.6 35B-A3B with DFlash2 goes 193.3 -> 204.5 tok/s (+6%) on M1 Ultra (@fneto1977 in #4828).
    • Also faster: Affine KV attention for short blocks on a long cache (@avifenesh in #5294), Qwen verify on M5 Pro (@masterfasa in #5318, #5320), Gemma 4 and Flint MoE routing (@masterfasa in #5315, @avifenesh in #5254) and the DeepSeek V4 decode loop (@bleugreen in #5395).
  • Cluster: DeepSeek V4.1 tensor parallel sizes each Mac's share to its memory, so an M5 Ultra 256GB and an M5 Max 128GB run DeepSeek V4.1 Flash oQ4e at about 45 tok/s (@Jiffywu in #5272).
  • oQ: DeepSeek V4.1 builds below oQ4 get the oQe code refinement, with 5.4% lower KL at oQ3e (d675c0d).
  • Speech to text: /v1/audio/transcriptions returns segments with speaker labels and times for response_format=diarized_json (@lybertybox443 in #5307), and duration is the audio length (dcfaa6b).
  • API: /tokenize and /detokenize work on embedding models (@junmo-kim in #5268).
  • Prefill Accelerator: the accelerator checks warn when the CUDA helper is older than the server expects (#5422).
  • macOS app: the Throughput benchmark gets the Force mlx-lm and external endpoint options of the web UI (@ngutech21 in #5257).
  • Web UI: Italian translation for the web UI and README (@fabriziosalmi in #4505).

Bug Fixes

  • More checkpoints load, with Qwen3.8 Flash-Next 4-bit conversions that fuse experts (@Li-Lee in #5388), Inkling 2 releases with a ModelOpt quantization_config (9d0f474), 3-, 5- and 6-bit Inkling 2 experts (@Hygaard in #5433) and single-file Flash-Next checkpoints with PLE on SSD (#5439).
  • oQ builds finish and stay correct, as the streamed oQe imatrix no longer runs out of memory on Flash-Next (@ltkn in #5269), streamed Qwen3.5 oQe and oQh builds match resident ones (@keefar in #5459), and two oQ size estimates at once no longer break the server (#5447).
  • Long sessions survive memory pressure, as a cache store no longer blocks the pool clear that keeps the model loaded (#5454 with @chigkim) and two long quantized-KV sessions decode together without an abort (@avifenesh in #5321, #5253).
  • MoE expert offload restores slots after a release that failed partway (@williamxie1989 in #5419), and CUDA helper 0.1.7 frees about 40 MiB per Qwen3.5 linear-attention layer between chunks (@davesecops in #5396).
  • oQh models are no longer refused by the CUDA Prefill Accelerator (1f7761c).
  • DeepSeek V4.1 tool calls that are malformed or cut off by the token limit come back as invalid_tool_call or keep the truncation status, instead of leaking DSML text (@clong in #5416).
  • SpecPrefill no longer hangs under Metal fast sync when it restores a cached system prefix (@pierre427 in #5408).
  • Integrations work in more setups, with Claude Code's auto mode on local models without a Claude server message (@jmunderwood in #5409) and Ask AI in the macOS app reaching plain http servers by hostname (693a607 with @jasonpaulso).

Upgrade Notes

  • Source installs: run make install after updating to rebuild the native kernels for Hy4 (#5463).
  • LoRA adapters: adapter folders in a model directory no longer show up as models in /v1/models (#5449).
  • Prefill Accelerator: pull jundot/omlx-cuda-helper again to update an NVIDIA helper to 0.1.7. The accelerator checks warn until it is updated (#5396, #5422).

New Contributors

@jmunderwood in #5409, @clong in #5416, @davesecops in #5396, @dkta0 in #4926, @bleugreen in #5395, @ltkn in #5269, @Hygaard in #5433, @Jiffywu in #5272.

Full changelog

Don't miss a new omlx release

NewReleases is sending notifications on new releases.