MakazhanAlpamys/Soup v0.30.0

v0.30.0 — Inference Excellence

on GitHub

What's New

Seven features that make soup serve the best OSS inference CLI.

• Prefix caching — soup serve --prefix-cache enables vLLM's automatic prefix cache. Big latency win for RAG / agent workloads with shared system prompts.
• Speculative decoding auto-pairing — soup serve --auto-spec picks a draft model based on the target family (Llama 3.1/3.3/4, Qwen 2.5/3, Mistral Large, Mixtral, DeepSeek V3/R1, Gemma 2/3). Targets under ~8B fall through with a yellow "no draft" note.
• Dynamic LoRA hot-swap — POST /v1/adapters/activate/{name} + POST /v1/adapters/deactivate switch the active adapter at runtime, no restart. Active state is thread-safe; name regex is enforced by FastAPI before the handler runs.
• Structured output — soup serve --structured-output json --json-schema s.json or --structured-output regex --regex-pattern '…'. Schema is cwd-confined, capped at 64KB, and must declare a top-level type; regex is capped at 2048 chars, null-byte rejected, must compile.
• Continuous-batching dashboard — soup serve --dashboard + GET /metrics (requests, tokens, active, p50/p95 latency). ServerMetrics is thread-safe; record_latency always runs so failure paths aren't a tail-latency blind spot.
• OpenTelemetry request tracing — soup serve --trace --trace-endpoint http://localhost:4317. OTLP endpoint SSRF-hardened: scheme allowlist, 0.0.0.0 rejected, RFC1918 / link-local / cloud-metadata IPs rejected via ipaddress.ip_address, plain HTTP only for loopback. Idempotent provider install preserves operator-supplied instrumentation.
• Auto-quant picker API — Candidate dataclass + pick_best() ship as a pure utility. soup serve --auto-quant registers the flag and prints a yellow deferral warning; the live eval loop ships in v0.30.1 (same pattern as v0.28.0 kernel_picker).

Install / Upgrade

pip install -U soup-cli

Docker: ghcr.io/makazhanalpamys/soup:v0.30.0.

Security

• OTLP endpoint SSRF hardening (matches v0.29.0 HF_ENDPOINT): scheme allowlist, 0.0.0.0 rejected, RFC1918 / link-local / cloud-metadata via ipaddress.ip_address, plain HTTP loopback-only.
• pick_draft_model rejects URL-scheme targets (http://, https://, file://), null bytes, names >200 chars.
• validate_regex_pattern length-capped at 2048 + null-byte rejection + re.compile check.
• validate_json_schema dict shape + 64KB serialised cap + required type field.
• --json-schema file path confined to cwd via is_under_cwd.
• --structured-output json fail-fast when --json-schema missing (no silent no-op).
• /v1/adapters/activate/{name} regex enforced by FastAPI Path(pattern=…); activate/deactivate state protected by threading.Lock; /v1/adapters never leaks filesystem paths.
• CORS on transformers backend restricted to loopback origins since hot-swap endpoints mutate state without auth.
• OTel span context uses contextlib.ExitStack so __exit__ receives real exception info; spans correctly marked errored on HTTPException.
• build_tracer idempotent — only installs provider when current is ProxyTracerProvider/NoOpTracerProvider (preserves operator-supplied instrumentation, prevents silent drop of unflushed spans).

Tests

2801 → 2886 (+85). New tests/test_inference_advanced.py covers all seven parts plus review-driven negatives.

Known Limitations

• Constrained generation deferred — --structured-output validates the constraint + stores a descriptor on app.state.output_constraint, but the transformers backend does not yet enforce the constraint at token-sampling time. outlines / lm-format-enforcer wiring tracked for v0.30.1.
• --auto-quant eval loop deferred — the Candidate + pick_best() picker API ships; the live eval loop that invokes it is deferred to v0.30.1. The flag prints a yellow deferral warning today (never a silent no-op).
• Streaming requests not in /metrics — the streaming path returns a StreamingResponse before track_request opens, so request.stream=true calls are not counted. Fix deferred to v0.30.1 (requires wrapping the async generator).
• --prefix-cache, LoRA hot-swap, OpenTelemetry tracing — all ship live.

Full changelog: v0.29.0...v0.30.0