michaelfeil/infinity v3.0.2-beta.1

Infinity v3.0.2-beta.1

on GitHub

Infinity V3 3.0.2-beta.1 improves shared-prefix decision inference and consolidates the native backend on LibTorch.

• OneJev block-attention requests execute as whole groups and can share a backend batch within configured limits. Oversized groups return HTTP 422 before inference.
• Shared prefix K/V pages, bounded temporary K/V copies and compact last-token readout reduce fanout memory use. Compatible Qwen tokenizers reuse the shared prefix encoding within each request.
• Eligible cached GDN suffixes use tensor-core prefill by default. This matches the previous INFINITY_TORCH_GDN_SUFFIX_PREFILL=1 setting; use =0 for recurrent suffix execution. BF16 normalization and chunk arithmetic can change probabilities and selected answers relative to the previous default.
• Optional service-tier queue token limits shed excess queued work. Pixtral image preprocessing is faster.
• The Candle backend has been removed. Both CPU and NVIDIA images now use native LibTorch.

Large fanouts still require suitable compute, expanded-token and question-count limits. See the batch configuration documentation.

CPU/H100 native regressions, GDN reference checks and tokenizer parity checks passed during integration. The default GDN setting matched explicit opt-in exactly in the Elephant fanout check.

Container builds are triggered for linux/amd64:

• ghcr.io/michaelfeil/infinity_v3:cpu-3.0.2-beta.1
• ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.2-beta.1

Images become available after their build and inference smoke checks finish. This prerelease leaves stable latest aliases unchanged.

Changes since v3.0.1 · V3 licensing and retained notices