MakazhanAlpamys/Soup v0.8.0

v0.8.0 — vLLM Backend for soup serve

on GitHub

What's New

vLLM Backend for soup serve — 2-4x better inference throughput for production serving.

Features

• --backend vllm flag for soup serve — uses vLLM's AsyncLLMEngine for high-throughput inference
• --tensor-parallel (--tp) — multi-GPU tensor parallelism (vLLM only)
• --gpu-memory — control GPU memory utilization (default: 0.9)
• Auto-detection: when vLLM is installed, Soup shows a hint suggesting --backend vllm
• Native streaming: true token-by-token SSE streaming via vLLM (not simulated)
• LoRA support: vLLM backend supports LoRA adapters via LoRARequest

Install / Upgrade

# Upgrade
pip install --upgrade soup-cli

# Install with vLLM support
pip install 'soup-cli[serve-fast]'

Usage

# Start with vLLM backend
soup serve --model ./output --backend vllm

# Multi-GPU
soup serve --model ./output --backend vllm --tensor-parallel 2

# Control memory
soup serve --model ./output --backend vllm --gpu-memory 0.8

Stats

• 560 tests, 39 test files
• Ruff lint clean
• CI on Python 3.9/3.11/3.12

Full Changelog: v0.7.3...v0.8.0