EricLBuehler/mistral.rs v0.5.0 on GitHub

Highlights

Blog post: https://huggingface.co/blog/EricB/mistralrs-v0-5-0

Thank you to all contributors for this release! This release includes the following highlights but also countless improvements, fixes, and optimizations.

Support for many more models:
- Gemma 3
- Qwen 2.5 VL
- Mistral Small 3.1
- Phi 4 Multimodal (image only)
Native tool calling support for:
- Llama 3.1/3.2/3.3
- Mistral Small 3
- Mistral Nemo
- Hermes 2 Pro
- Hermes 3
Tensor Parallelism support (NCCL)!
FlashAttention V3 support and integration in PagedAttention
30x reduction in ISQ times on Metal!
Revamped prefix cacher system

What's Changed

Allow using library in CurrentThread runtime by @sgrebnov in #1082
Improve accuracy of uqff auto device map by @EricLBuehler in #1084
DeepSeekV3 sigmoid support by @EricLBuehler in #1092
GPU-accelerated sampling (+5% decode perf) by @EricLBuehler in #1094
Fix missing perceiver_config in qwen2vl by @EricLBuehler in #1096
More topk methods for deepseek 2/3 by @EricLBuehler in #1097
More accurate layer size computation for deepseek 2/3 by @EricLBuehler in #1098
Improve streaming UX by @EricLBuehler in #1102
Faster fp8 blockwise dequant by @EricLBuehler in #1100
DS2/3 paged attn by @EricLBuehler in #1103
Faster bincount by @EricLBuehler in #1104
PagedAttention prompt chunking support by @EricLBuehler in #1105
Refactor server SSE by @EricLBuehler in #1107
PagedAttention + FlashAttention (and FlashAttention V3) by @EricLBuehler in #1109
Take KEEP_ALIVE_INTERVAL into account by @EricLBuehler in #1111
Refactor enable of flash attn by @EricLBuehler in #1110
Fix imatrix isq quantize_onto by @EricLBuehler in #1112
Tensor parallelism and pipeline parallelism by @EricLBuehler in #1113
Bump openssl from 0.10.69 to 0.10.70 by @dependabot in #1121
Allow chat streaming to use tools by @Jeadie in #1088
New file format for imatrix: .cimatrix by @EricLBuehler in #1004
Fix isq with bias for column parallel by @EricLBuehler in #1128
Multi-node support for tensor parallelism by @EricLBuehler in #1125
Add an NCCL feature flag by @EricLBuehler in #1129
Fix mistral 2501 gguf by @EricLBuehler in #1131
Add jinja strftime_now function by @EricLBuehler in #1132
Multiple models multi node by @EricLBuehler in #1136
Remove unexpected cp behavior by @jncraton in #1141
Revamp speculative decoding! by @EricLBuehler in #1027
Fuse MLP mul-and-act by @EricLBuehler in #1142
Short-circuit dry sampling: +6% T/s by @EricLBuehler in #1143
Integrate fused MLP mul-act for more models! by @EricLBuehler in #1144
Use cudarc 0.13.5 by @EricLBuehler in #1145
Handle HF_HUB_CACHE env var by @EricLBuehler in #1146
FlashAttention V2/V3 metadata with support for device location by @EricLBuehler in #1148
FP8 blockwise dequant cuda kernel by @EricLBuehler in #1149
Blockwise FP8 CUDA for cc < 800 by @EricLBuehler in #1150
Fix chat sampling response by @EricLBuehler in #1154
Multiple processes for TP by @EricLBuehler in #1152
Ensure we do not bind the port for daemon processes by @EricLBuehler in #1158
Handle CUDA_NVCC_FLAGS in flash attn v3 by @EricLBuehler in #1160
build fix for arm. by @jamesvren in #1164
Working PrefixCacherV2! by @EricLBuehler in #1168
Implement Phi-4 Multimodal! by @EricLBuehler in #1163
No extra split/cat pair in rope by @EricLBuehler in #1169
Remove gpu<>cpu sync for faster long-context by @EricLBuehler in #1170
Refactor NCCL device mappers by @EricLBuehler in #1172
Bump ring from 0.17.11 to 0.17.13 by @dependabot in #1179
DSV3/R1 fixes by @EricLBuehler in #1173
Fix diffusion device mapping by @EricLBuehler in #1187
Internal abstraction for distributed op by @EricLBuehler in #1188
Make Sequence::set_toks more safe by @EricLBuehler in #1190
Fix CI tests out of storage? by @EricLBuehler in #1191
Internal abstraction for distributed op by @EricLBuehler in #1189
Fix build_cuda_all.yaml CI by @EricLBuehler in #1193
Support tensor parallelism for vision models! by @EricLBuehler in #1194
Always pass _USE_MATH_DEFINES for CUDA by @EricLBuehler in #1195
Remove matmul via f16 framework by @EricLBuehler in #1196
Remove API for matmul_via_f16 by @EricLBuehler in #1197
Add UQFF text/vision model API by @EricLBuehler in #1198
Complete qwen2_5_vl, and some fixes by @brrr in #1184
Implement Gemma 3 by @EricLBuehler in #1201
Add Gemma 3 vision support! by @EricLBuehler in #1202
Manually fixup sentencepiece detok by @EricLBuehler in #1204
More vision models with TP by @EricLBuehler in #1200
Fix topology link in the docs by @etiennebalit in #1205
Gemma3 1b support and optimized rotating cache by @EricLBuehler in #1206
Improve rotating kv cache, prefix cacher system by @EricLBuehler in #1207
Better handling for kvcache set_len by @EricLBuehler in #1208
Update deps and use rand 0.9 by @EricLBuehler in #1210
Update hf hub dep, add initial blockwise fp8 GEMM tests by @EricLBuehler in #1212
Growable RotatingKvCache and fixes for Phi-4 mini by @EricLBuehler in #1215
Gemma 3 cuda fixes by @EricLBuehler in #1217
Add pydantic schema examples! by @EricLBuehler in #1219
Sliding window attention fixes by @EricLBuehler in #1220
adapt to rig crate as client by @benliao in #1214
Implement Mistral 3! by @EricLBuehler in #1221
Metal SDPA with masking by @EricLBuehler in #1225
Send [DONE] SSE chunk per openai spec by @EricLBuehler in #1226
Fix handling of device when compiled for but disabled nccl by @EricLBuehler in #1227
Fix nccl blocking case by @EricLBuehler in #1228
Native Llama, Mistral Small 3.1, Mistral Nemo, Hermes 2 Pro, Hermes 3 tool calling! by @EricLBuehler in #1229
OpenAI API compatability fixes by @EricLBuehler in #1230
[Breaking] Automatic server logging by @EricLBuehler in #1231
Use default stream for flash attn by @EricLBuehler in #1232
Bump version to 0.5.0 by @EricLBuehler in #1233

New Contributors

@sgrebnov made their first contribution in #1082
@jncraton made their first contribution in #1141
@jamesvren made their first contribution in #1164
@brrr made their first contribution in #1184
@etiennebalit made their first contribution in #1205
@benliao made their first contribution in #1214

Full Changelog: v0.4.0...v0.5.0