jundot/omlx v0.6.0.dev1

0.6.0.dev1

on GitHub

This development release introduces opt-in distributed model serving across Macs and mixed Metal/CUDA nodes, adds built-in web search and streaming speech-to-text to chat, keeps active generation responsive during concurrent prefills, and brings SSD-backed prompt reuse to distributed ranks.

• Added experimental distributed serving across Macs. One model can now be split across multiple Apple Silicon machines using tensor or pipeline parallelism, with capability-aware planning, selective shard staging, per-rank memory guards, liveness supervision, and one-click activation from the Cluster dashboard. Qwen3.6-27B tensor parallel reached 28.6 tok/s across two Macs versus 16.1 tok/s on one Mac, with byte-identical output; a 225 GB MiniMax-M3 checkpoint that could not fit either machine alone also loaded across 128 GB and 256 GB Macs and passed its distributed canary. The feature is disabled by default under Settings > Advanced while it remains an experimental source-build preview. By @ashhart in #2423, with follow-up fixes for SSH failure handling, clock-safe liveness, MLA/MTP memory accounting, and load admission.

• Added heterogeneous Metal and CUDA model pools. Apple Silicon and NVIDIA CUDA workers can contribute to one logical model-memory pool, with automatic contiguous-layer placement based on usable memory, short-lived GUI-generated worker enrollment, ConnectX discovery and NCCL verification, and cluster diagnostics in the dashboard. The current compatibility path keeps every physical worker in the outer MLX Ring; verified CUDA pairs are placed together, but the future hierarchical Ring-to-NCCL gateway is not implemented yet. By @ashhart in #2591.

• Added decode fairness during concurrent prefill. Prefill now yields GPU time to active decodes, including streams running in another engine, and sizes contended chunks by a target stall time instead of a fixed token count. The feature is enabled by default and can be changed live from the dashboard. On an M3 Ultra, overlap decode improved by about 1.6x to 43x across the tested model pairings, while contended prefill took 1.15-2.0x longer and solo prefill remained unchanged. By #2633.

• Added process-lifetime SSD prompt caching for distributed ranks. Each rank now stores a chain of cache-boundary snapshots and restores the longest prefix held by every rank when the in-memory prompt cache misses. Incremental KV segments keep storage growth linear instead of copying the full prefix at every boundary; a 12k-token GLM-5.2 chain used 1.17 GB instead of about 6.5 GB. Snapshots are scoped to the current rank process and are cleaned up on restart or teardown. By #2620.

• Added built-in web search to the admin chat. The chat composer can expose web_search and fetch_url tools to the selected model, with keyless DDGS metasearch by default and optional DuckDuckGo, Brave Search API, or SearXNG providers. Results stay visible as source cards across tool rounds, while fetched URLs are checked against private and loopback destinations on every redirect and capped before MarkItDown conversion. External API clients are unchanged because only the chat UI advertises the built-in tools. By #2596.

• Added file and realtime speech-to-text to chat. Selecting an audio STT model switches the composer into ASR mode for streamed audio or video transcription, and compatible models also expose live microphone input over WebSocket. Whisper file streaming now uses bounded segments with the full anti-hallucination path and per-segment language detection, while Qwen3-ASR uses cumulative token decoding so multi-token CJK characters no longer turn into replacement characters. Multipart request logging also skips binary bodies instead of buffering and printing them. By #2603.

• Improved Meta Muse Glimmer and DFlash. Long Muse Glimmer oQ4 prefill now routes quantized projections through the native QMM tile, measuring about 4.5% higher prompt throughput on a cold 51k-token prompt with bit-exact output. DFlash responses now preserve parser-emitted tool calls instead of dropping them during speculative generation.

• Fixed MTP performance state after cross-engine prefill contention. The adaptive loop-tax probe could sample a handoff while another engine was prefilling, latch the resulting slowdown onto the model, and suppress MTP performance until restart. Contaminated measurements are now skipped or discarded, and stale high measurements decay back toward the default margin.

• Fixed DeepSeek-V4 affine AWQ checkpoint loading. Per-expert bias and zero-point tensors are now stacked with weights and scales during sanitize, so externally quantized affine checkpoints no longer fail strict weight loading with thousands of leftover *.biases keys. By @True2456 in #2598.

• Improved oQ discovery for Hugging Face cache models. Original, unconverted checkpoints inside the Hugging Face snapshot cache now appear as quantization sources with their canonical repository identity instead of a snapshot hash. By @monroewilliams in #2605, with a follow-up identity-preservation fix.

• Exposed model-management state through the API. /v1/models/status now includes is_favorite and is_hidden, allowing API clients to reproduce dashboard filtering without an additional model-list request. By @monroewilliams in #2602.

New Contributors

• @ashhart made their first contribution in #2423.