michaelfeil/infinity v3.0.0

Infinity v3.0.0

on GitHub

Infinity 3.0.0 is the first stable release of the native v3 server: a Rust HTTP server with LibTorch inference for embeddings, reranking, classification and typed decisions.

Highlights

• Native serving: packed variable-length batches, token-based scheduling and dedicated tokenizer workers.
• Embeddings: dense text and multimodal embeddings, ColBERT-style token-vector retrieval, and SPLADE sparse embeddings.
• Classification: cross-encoder reranking, sequence classification, reward scoring and token classification/NER.
• Typed decisions: native OneJev (including Elephant and Mammoth), Perplexity Decider, D3/D3-mini, Clef, Laya and Rune protocols with their checkpoint-specific heads and input formats.
• Model coverage: a task-grouped catalog of 129 named checkpoints, with launch settings, input requirements, native implementation details and model-license notes. Includes BERT, ModernBERT, UltraBERT, Nomic, GTE, Qwen, Nemotron and other implemented families.
• CUDA execution: native inference kernels and FlashAttention-4 for supported attention shapes, with existing attention kernels for other shapes.
• Deployment: automatic checkpoint pooling, full model replicas across visible GPUs, persistent model caching and startup diagnostics. The default native weight reader requires no additional container capabilities.

Run on NVIDIA

Requires Linux x86-64, an SM80/Ampere or newer GPU, a compatible NVIDIA driver and NVIDIA Container Toolkit.

docker run --rm --gpus all -p 8000:8000 -v infinity-cache:/data \
  ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.0 \
  --model-id sentence-transformers/all-MiniLM-L6-v2 \
  --torch-device cuda --dtype bfloat16 \
  --hostname 0.0.0.0 --port 8000

curl http://localhost:8000/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"sentence-transformers/all-MiniLM-L6-v2","input":["A document about semantic search."]}'

All visible GPUs are used by default. Each holds a complete replica, including every expert for MoE checkpoints. /health reports readiness after loading and warmup; /docs exposes the running server's API reference.

Versioned images:

docker pull ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.0
docker pull ghcr.io/michaelfeil/infinity_v3:cpu-3.0.0

Both image variants are published for linux/amd64. The NVIDIA image uses native LibTorch. The CPU image uses Candle and has separate model coverage. Stable releases update nvidia-latest and cpu-latest; development builds use nvidia-main and cpu-main.

The image pipeline checks version reporting, readiness and MiniLM embedding inference on CPU for both variants before publishing. CUDA-workload validation is a separate step.

Moving from Python / v2

The v3 image starts infinity_v3 directly, without a v2 subcommand. The infinity-emb Python package and infinity_emb v2 command remain the Python server. Its in-process Python engine APIs and backend options are separate from the native v3 server.

Use the model catalog and the running API schema for each task's request format. Structured decision protocols have different input contracts; architecture support requires the appropriate trained head and checkpoint metadata.

Support and licensing

Dynamic FP8 and CUDA graphs remain opt-in. DeBERTa-v2/v3 support is experimental. Model compatibility and validation coverage vary by checkpoint, precision and GPU; the catalog describes the native implementation behind each entry.

Covered original v3 contributions use Infinity Restricted MIT License 1.0, a custom source-available license with provider restrictions and retained third-party and earlier grants. Python packages retain MIT licensing. Model weights and dependencies retain their own terms, including non-commercial licenses where specified.

Model catalog · Configuration and API inputs · Measured performance · Python / v2 · License

Verified image build and smoke tests · Changes since alpha.19