michaelfeil/infinity v3.0.2-beta.3

Infinity v3.0.2-beta.3

on GitHub

Infinity V3 3.0.2-beta.3 updates native CUDA inference and restores compatibility with legacy rerank requests.

• Rerank requests accept documents as an alias for texts (#906).
• Dynamic quantization is selected per model with --dynamic-quantization none|fp8|mxfp8|nvfp4 and --dynamic-quantization-scope auto|mlp|all (#910). Quantization remains disabled by default.
• Compatible Blackwell BF16 decoder MLPs can use native MXFP8 or opt-in NVFP4. FP8 conversion, projection scaling, activation, and RoPE paths have further native optimizations (#905 and related commits).
• Shared-prefix attention uses paged FA4 for supported layouts (#914). GDN convolution and provider packing are fused, and eligible cached GDN suffix prefill is selected automatically (#911, #912).

Migration from earlier betas:

• Replace --enable-fp8-dynamic and the retired FP8/MXFP8/NVFP4 environment selectors with the new per-model options. Old selectors are rejected, including when set to 0 or false. /info now reports dynamic_quantization and dynamic_quantization_scope. See the migration table.
• The INFINITY_TORCH_GDN_SUFFIX_PREFILL opt-out is removed. Cached suffix prefill is automatic when its prefix state and provider support are available. BF16 arithmetic can change probabilities and selected answers; validate outputs for your workload.

Images for linux/amd64:

• ghcr.io/michaelfeil/infinity_v3:cpu-3.0.2-beta.3
• ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.2-beta.3

Both images pass version/startup and MiniLM embedding inference smoke checks before publication. The NVIDIA image's CI inference check runs on CPU; GPU hardware execution is not part of that check. This prerelease leaves stable image and documentation aliases unchanged.

Changes since beta.2 · V3 licensing and retained notices