Prerelease for Infinity 3.0.1, including the inference improvements merged since 3.0.0.
Changes
- Faster Qwen3.5 dynamic FP8 inference through reused matmul plans, fused normalization/FP8 packing, and improved cached-state staging. Original projection shapes and recurrence policy are preserved. These optimizations activate with the existing
--enable-fp8-dynamicoption. (#827) - Qwen BF16 attention performance improvements. (#828)
- Gemma FP16 activation, normalization, and rotary kernel improvements. (#826)
- Updated H100 measurements and the documented BERT NER tokenizer revision. (#825, #831)
The separate INFINITY_TORCH_GDN_SUFFIX_PREFILL=1 setting opts cached suffixes into tensor-core prefill. It remains off by default because its alternative recurrence arithmetic can change probabilities and selected answers. CUDA graph replay includes block-prefix metadata; existing graph admission and memory budgets apply.
Images
docker pull ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.1-beta.1
docker pull ghcr.io/michaelfeil/infinity_v3:cpu-3.0.1-beta.1Both images target linux/amd64. The NVIDIA image uses native LibTorch; the CPU image uses Candle with separate model coverage. This prerelease does not move stable latest image or documentation aliases.
Source builds must rebuild the native backend and GDN provider together for provider ABI 2; packaged NVIDIA images include the matching provider.
Model catalog · Configuration · Licensing · Changes since 3.0.0
Validation
Both versioned images passed version reporting, readiness, and MiniLM CPU inference checks before publication. The published NVIDIA image also passed H100 CUDA smoke validation with dynamic FP8 decision inference.