🚀 Foundry Local v2.4.0

Foundry Local v2.4.0 adds ranking and typed-decision models alongside generation, transcription, and embeddings. You can now score candidates and make structured decisions on-device with the same model lifecycle, local catalog, and SDK session patterns you already use. These features are available in C++, C#, JavaScript, Python, and Rust.

This release also adds AMD GPU execution-provider support, updates ONNX Runtime GenAI to 0.19.0, verifies model downloads, and adds request cancellation in every language SDK.

🆕 What's New

🏆 Ranking and typed-decision models

• Two new local-catalog tasks: text-ranking and typed-decision. Register a multi-component package (for example, CLM for ranking or KEV for decisions) in the local catalog, load it, and run it locally on CPU or CUDA.
• New HTTP endpoints: POST /catalogs/local/v1/rank and POST /catalogs/local/v1/systemone. Pass the registered model ID or alias in model.
• New RankingSession and DecisionSession APIs in every SDK: C++, C#, JavaScript, Python, and Rust. They return typed results and use the same Session/Request/Response ABI as the other sessions. You can reuse one session across many requests.
• Ranking results include each candidate's id and zero-based index, plus a probability relative to the submitted candidate set. Ties keep input order.
• Decision results support choice criteria, which keep your IDs and order, and ordered score criteria, which return a probability-weighted score that can be fractional.
• Built-in safeguards:
  • Packages are validated, including manifests, tasks, providers, and paths. Paths cannot escape the package root, including through symlinks.
  • Concurrent cold loads of the same model are deduplicated.
  • A memory budget is reserved before loading. Configure it with NonGenerativeMemoryBudgetBytes.
  • NonGenerativeReadinessConcurrency limits concurrent initialization per provider.
  • Unload and unregister wait for active requests to finish.

with RankingSession(ranking_model) as session:
    ranking = session.rank(RankingRequest(
        context={"weather": "rain"},
        question="Which activity is better?",
        answers=["picnic", "museum"],
    ))
    print(ranking.ranked)

See the package guide and each SDK's README for complete examples.

🟥 AMD GPU execution provider

• Foundry Local now recognizes AMDGPUExecutionProvider and can select it for models that support it on AMD GPUs.

🛑 Request cancellation in every SDK

• C#: CancellationToken works on typed, compatibility, and streaming calls.
• JavaScript: Pass an AbortSignal.
• Python: Pass a cancel_event.
• Rust: Dropping a stream or future cancels the request.

Canceled requests roll back their history, and you can reuse them.

⚡ Improvements

• ONNX Runtime GenAI 0.19.0, with refreshed CUDA EP bundles for Windows and Linux on x64 and ARM64.
• Verified model downloads: each 2 MiB range is checked with Azure Storage MD5 and pinned to the blob ETag. Corrupt chunks are downloaded again on resume. Incomplete downloads are never treated as cached.
• Better automatic model selection: variants that need an unavailable execution provider are skipped when choosing defaults. They remain listed, and you can still select them explicitly.
• Embedding token usage is now reported in native responses, OpenAI JSON, and the HTTP endpoint.
• Better tool calling: the tool protocol is inferred from the model's rendered chat template.
• Model context length and capabilities (for example, reasoning and tool-calling) are restored from catalog tags when structured metadata is missing.
• Image inputs are documented for the JSON APIs and native SDKs.
• WinML updated to 2.5.96.

🐛 Fixed

• Qwen2.5 required and named tool choices no longer return HTTP 500.
• image_url vision inputs now work in Chat Completions.
• The Microsoft.AI.Foundry.Local.Runtime NuGet package no longer overrides the consuming project's RuntimeIdentifier.
• Fixed a startup race in the shared host.

⚠️ Behavior Changes

• The telemetry opt-out environment variable is now ORT_DISABLE_TELEMETRY=1, matching ONNX Runtime. It replaces ORT_TELEMETRY_DISABLED. Update scripts and deployments that set the old variable.

📥 Source Code

GitHub automatically includes source archives with the v2.4.0 release:

• Source code (zip)
• Source code (tar.gz)

📚 Resources

💙 Feedback

File issues at https://github.com/microsoft/Foundry-Local/issues and include the SDK version, platform, and relevant diagnostics.

Full Changelog: v2.3.1...v2.4.0