BerriAI/litellm v1.109.0-dev.1 on GitHub

Verify this release

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/e82a3c646908ba0a6ee0beddc4310d7a6cac4065/cosign.pub \
  ghcr.io/berriai/litellm:v1.109.0-dev.1

Same command for litellm-database, litellm-non_root and the docker.io/litellm/* images

What's new in 1.109.0-dev.1

704 PRs. 113 added, 312 fixed, 46 changed, 11 improved, 8 removed. 15 need action

Action required

Deployment: Update image verification to use the new cosign.pub key (#48103)
Chat completions: Guard chunk.choices[0] reads with if chunk.choices: when include_usage is set (#47109)
Vertex AI and Gemini: Set drop_params: true or stop sending temperature, top_p and top_k to Gemini 3.6 Flash, 3.5 Flash-Lite and newer (#47645)
Cost tracking and pricing: Read max_input_tokens and max_output_tokens instead of max_tokens from the cost map (#48471)
Anthropic: Set max_tokens on requests to unmapped Anthropic models, or set DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS to 4096 (#48502)
AWS Bedrock: Set streaming_buffer_until_moderated: true on the guardrail if converse-stream replies still need masking (#46307)
Logging and observability: Set the GCS bucket and service account in proxy config instead of per request (#48599)
Token counting: Set a provider key on deployments used with count endpoints, since a local estimate is no longer returned (#46815)
Keys, teams and auth: Make budget_duration and budget_id changes on keys as an admin, since owners and team members now get 403 (#47681)
Keys, teams and auth: If personal_key_model_access_from_teams is on, grant needed MCP servers and agents to owners' teams first (#46293)
Routing and fallbacks: Add the tag_routing_prefix to caller routing tags, or move them to the key or team (#47428)
Chat completions: Handle 408 litellm.Timeout and request_timeout where you matched 500 APIConnectionError or server_error (#47198)
Deployment: Set image_tag or all four *_image URIs when sourcing the Terraform module from a git ref (#46960)
Vertex AI and Gemini: Raise a low tpm on Gemini TTS deployments, since limits now count TTS tokens (#44551)
AWS Bedrock: Increase budgets for keys or teams that run a traced guardrailConfig on every Bedrock call (#48278)

Highlights

Admin UI: Added a Chat UI with a model hub, image generation and usage metrics (#48108)
Routing and fallbacks: Added mid-stream fallback continuation so a streaming chat cut off mid-answer continues on the configured fallback (#41127)
SDK: Fixed tool calls on completion() and responses() failing with No module named 'mcp' on base SDK installs, broken since v1.106.0 (#45961)
Other providers: Added OpenInfer as an OpenAI-compatible provider with openinfer/<model> calls and spend tracking (#41034)
Other providers: Added xAI X Search to the Search API and WebSearch interception, with citation URLs and cost tracking (#48677)
Deployment: Added LITELLM_OFFLINE_MODE to turn off every fetch LiteLLM makes on its own with one switch (#48096)
Cost tracking and pricing: Added model_info.pricing_provider to price a deployment from another provider's catalog rates instead of $0 (#47620)
Vertex AI and Gemini: Added Gemini 3.1 Flash TTS support, including multi-speaker voices and mp3, opus, alaw and mulaw output (#31915)
Agents: Added agent_access_default_deny to stop keys with no agent grants from reaching every agent (#43979)

More changes by area

OpenAI (10)

Added per-deployment workload identity on non-chat routes, realtime and the moderation guardrail (#39966)
Added OpenAI gpt-6.1-terra with pricing (#48429)
Fixed prompt_cache_retention such as 24h being silently dropped instead of sent upstream (#47480)
Fixed 12-hour prompt cache writes on openai/gpt-5.6 and newer billing at the 30-minute rate (#48088)
Plus 6 smaller fixes and updates (#43935, #44021, #46967, #47954, #48066, #48246)

Azure (18)

Fixed /v1/responses 404 on dated api_version deployments and azure_ai Grok reasoning token billing (#46377)
Fixed Azure realtime over WebRTC failing with a 404 on client_secrets and an invalid token on the SDP exchange (#47316)
Improved Entra auth by reusing credentials across requests and refreshing tokens off the event loop (#47348)
Plus 15 smaller fixes and updates (#43955, #46421, #46539, #46969, #47292, #47567, #47779, #48032, #48033, #48118, #48224, #48252, #48434, #48475, #48529)

Anthropic (10)

Fixed a fake [System: ...] message being added to tool-call-only assistant turns sent to Anthropic (#34822)
Fixed Claude json_schema answers losing keys listed only in required and escaping non-ASCII characters, broken since v1.107.0-rc.1 (#47478)
Fixed replayed redacted_thinking blocks being dropped, and Bedrock Converse 400s when sent twice (#47489)
Fixed streamed responses leaving native web search fees out of the reconstructed usage cost (#48528)
Plus 6 smaller fixes and updates (#41397, #46102, #46741, #47394, #47987, #48633)

AWS Bedrock (33)

Fixed Bedrock geo models priced at $0 and dictionary changed size during iteration errors during price reloads (#41523)
Fixed 400 invalid beta flag on Claude Code auto mode requests that carry no safeguards (#44471)
Fixed Nova Sonic /v1/realtime tool calls arriving with empty arguments and sessions breaking on flat tools (#46214)
Fixed post_call guardrails not scanning Claude output on invoke-with-response-stream passthrough (#46524)
Fixed Claude json_schema streams dropping reasoning and thinking blocks and ending on tool_calls with no tool calls (#46863)
Fixed JSON schema requests failing on Fable 5 and Sonnet 4.6 and losing adaptive thinking on Haiku 5.5 (#46870)
Fixed Bedrock Opus 4.8 and Sonnet 5 dropping reasoning on json_schema requests by trying native output first (#47100)
Fixed AWS region lookups through Vault blocking the event loop and stalling other requests on the worker (#47331)
Fixed Converse profile calls recording zero or wrong spend and partial custom prices dropping token charges (#47419)
Changed passthrough errors to the AWS shape so AWS SDKs read the error code and message and retry throttling (#46522)
Changed /bedrock pass-through routes to apply a model's guardrailConfig, returning 400 if Bedrock cannot use it (#47429)
Plus 22 smaller fixes and updates (#43875, #45453, #45992, #46670, #46936, #46987, #47000, #47040, #47144, #47196, #47202, #47211, #47226, #47260, #47283, #47441, #47462, #48150, #48247, #48441, #48479, #48682)

Vertex AI and Gemini (30)

Added an opt-in free-tier allowance so Google Search grounding spend matches the GCP invoice (#47494)
Fixed streamed Vertex AI pass-through calls missing the regional 10% uplift and custom deployment prices (#44795)
Fixed cache-marked requests listing all cachedContents on every call and creating duplicate caches (#46857)
Fixed audio prompt tokens billing $0 and video or image tokens ignoring the above-200k tier price (#47373)
Fixed text-embedding-005 and text-multilingual-embedding-002 billing per token instead of per character (#47460)
Fixed Gemini 2.5 rejecting tool results with images and the history replay that follows, broken since v1.89.0 (#48428)
Fixed slow Google credential requests blocking the proxy worker on Claude /v1/messages (#48523)
Plus 23 smaller fixes and updates (#36460, #40868, #41683, #44908, #46793, #46826, #47134, #47185, #47294, #47404, #47447, #47466, #47542, #47646, #47666, #47847, #48085, #48086, #48248, #48430, #48442, #48587, #48690)

Other providers (69)

Added ChatGPT subscription forwarding so Codex or Pi users keep their login behind a separate LiteLLM key (#41522)
Added Volcengine Doubao-Seedance video generation on /v1/videos, with prices for the four models (#46156)
Added Nace as a provider with Drex decision models, available in Add Model (#46523)
Added DeepSeek as a /v1/search provider with search_provider: deepseek (#47401)
Added /v1/extraction endpoint with Cloudflare Clairic support (#47473)
Added OpenCode Go routing, pricing and model discovery (#48140)
Added the Refuseless provider with model pricing and capability metadata (#48497)
Added Abliteration model pricing and capabilities, and fixed reasoning_effort being dropped (#48498)
Fixed SageMaker streams that lost part of the answer or usage, and failed streams now return an error (#47495)
Fixed Groq HTTP 400 on the next turn after reusing a streamed reasoning reply as history (#48268)
Plus 59 smaller fixes and updates (#33121, #38736, #39227, #40152, #40605, #41021, #45092, #45567, #46091, #46171, #46792, #46794, #46795, #46945, #46959, #46964, #46991, #47085, #47132, #47252, #47253, #47254, #47291, #47318, #47326, #47346, #47370, #47371, #47395, #47423, #47424, #47487, #47492, #47518, #47570, #47591, #47610, #47667, #47690, #47784, #47785, #47963, #48121, #48158, #48244, #48251, #48254, #48265, #48270, #48448, #48450, #48453, #48455, #48522, #48527, #48621, #48651, #48668, #48676)

Chat completions (6)

Fixed chat streams failing with a proxy error when progress chunks have no choice index (#46947)
Plus 5 smaller fixes and updates (#39762, #44349, #46928, #47790, #48052)

Responses API (17)

Fixed non-stream MCP auto-execute on /v1/responses logging and billing only one of its two model calls (#43667)
Fixed later turns on an open Responses WebSocket skipping deployment checks and not counting toward budgets (#45727)
Fixed /v1/responses on Claude dropping returned and replayed compaction items (#46754)
Fixed native WebSocket connections writing one summed spend-log row at close instead of one per response.create turn (#46768)
Fixed Responses WebSocket running pre_call guardrails on turn one only and rebilling that cost every turn (#47013)
Fixed OpenAI pass-through GET /responses/{id} being billed again on every call (#47171)
Fixed background spend not counting toward the key's project or per-model budgets (#47504)
Fixed 500 when replaying web_search_call input items that have no action (#48417)
Fixed audio, PDF and image tool results and history media being dropped in Responses and Gemini conversions (#48484)
Plus 8 smaller fixes and updates (#31074, #46662, #46724, #47445, #47506, #47688, #48119, #48686)

Messages API (10)

Fixed retried /v1/messages attempts mixing up usage and spend logs, and exhausted retries billing only the last (#45194)
Fixed 400 Tool reference not found for tool names over 64 chars when a guardrail rewrites /v1/messages (#47434)
Fixed stop_sequences being ignored for OpenAI models on /v1/messages (#48534)
Fixed /v1/messages streams reporting zero input_tokens in message_start when bridged to Chat Completions (#48618)
Plus 6 smaller fixes and updates (#45136, #46096, #46749, #47123, #47347, #48220)

Realtime (1)

Fixed sessions going silent when a transcript guardrail sent turn_detection with no type to OpenAI (#44939)

Embeddings, images and audio (1)

Added a per-request input batch limit on /v1/embeddings to cap how many inputs one call can send (#44574)

Batches and files (8)

Added routing of POST /v1/batches to batch deployments and supported_endpoints on /model_group/info (#47376)
Fixed 400 on GET /files/{id}/content with raw gs:// file ids from the unmanaged Vertex batch flow, broken since v1.90.0 (#40621)
Fixed fine-tuning jobs failing with 400 for files uploaded with target_model_names (#46552)
Fixed 403 when team keys retrieve, cancel or download their own batches (#47067)
Fixed batch cost logging crashing before the batch is marked processed when every request fails (#47155)
Fixed batch spend going unrecorded when a provider returns malformed response envelopes or metadata writes fail (#47184)
Fixed batch cost tracking being starved by older retryable batches and losing spend on completed ones (#47190)
Fixed completed batches staying permanently unbilled after a temporary accounting failure (#47625)

Vector stores (3)

Fixed /rag/ingest and /rag/query ignoring team provider credentials and denying viewers and team JWTs (#45497)
Fixed team key vector store search failing when the team's first model is from a different provider (#46894)
Fixed proxy-created OpenAI vector stores filling all 16 metadata slots and then failing updates with HTTP 400 (#47021)

Token counting (11)

Added token_count_framing and token_count_model settings and a count source in token count responses (#46777)
Fixed /v1/messages/count_tokens on Gemini dropping PDFs, images and tool blocks, and 500s on AI Studio (#46759, #46767, #46772)
Fixed DeepSeek V4 token counts missing Azure's billed tokens by using DeepSeek's own tokenizer (#47173, #47353)
Changed Responses input token counts to match inference, fixing Claude counts with tools and text.format (#47363)
Plus 7 smaller fixes and updates (#46583, #46839, #46843, #47163, #47586, #48526, #48624)

MCP (27)

Added support for redeeming an ID-JAG as an RFC 7523 jwt-bearer grant on the gateway token endpoint (#46748)
Fixed tools/list returning an empty list when servers answer 429, now retried with an error if all fail (#46649)
Fixed the caller's LiteLLM key being forwarded to upstream MCP servers when Authorization is in extra_headers (#46932)
Fixed VS Code sign-in to DCR bridge MCP servers when the upstream rejects loopback redirect URIs, broken since v1.103.0 (#46940)
Fixed large MCP SSE tool results failing, and invalid size limits breaking MCP discovery and tool calls, broken since v1.103.0 (#47034)
Fixed MCP OAuth sign-in callback returning 403 behind AWS WAF by sealing the litellm_cp_return_to cookie (#47595)
Fixed per-user auth MCP servers missing from /mcp-rest/tools/list and /mcp tools/list, broken since v1.104.2 (#47677)
Fixed a readonly MCP toolset URL returning every tool instead of only the toolset's tools, broken since v1.106.0 (#48095)
Fixed revoked or expired browser sessions still being accepted during gateway authorization (#48494)
Plus 18 smaller fixes and updates (#41197, #44125, #44957, #46264, #46513, #46745, #46747, #46968, #46994, #47004, #47094, #47569, #47897, #48424, #48474, #48622, #48625, #48669)

Agents (2)

Added the pi harness to litellm.agent() and fixed harness turn cost ignoring prompt cache prices (#45467)
Fixed OpenCode via litellm.agent() using the script's folder instead of the sandbox, and hanging on resume (#47530)

Guardrails (10)

Added a native Reco guardrail that blocks requests on Reco errors unless fail_on_error: false (#48243)
Fixed guardrail masks on messages and tools not reaching Bedrock Converse requests (#45874)
Fixed corrupted numbered PII tokens in Presidio when one value matches several entities with output_parse_pii (#47024)
Fixed hide-secrets missing secrets in tool results, tool inputs, tool_calls arguments, and system prompts (#47122)
Fixed pass-through guardrails scanning litellm_metadata fields added by LiteLLM, which caused false matches (#48276)
Changed guardrail blocks mid-stream to end chat completions streams with an error event, not a cut connection (#46953)
Plus 4 smaller fixes and updates (#46338, #46606, #46673, #46965)

Caching (1)

Fixed streamed cache hits dropping prompt cache token counts from usage, including on /v1/messages (#47073)

Routing and fallbacks (29)

Added Kubernetes pod discovery to spread load across pods behind a headless service api_base (#44342, #44393, #44666)
Fixed region, RPM and param pre-call checks being skipped when the router cannot token-count a request (#44544)
Fixed stream_timeout cutting off slow answers when Bedrock serves a stream=True call without streaming (#47162)
Fixed key and team router_settings reaching the provider and dropping retries, fallbacks and timeout (#47257)
Fixed 500s from router_settings.default_priority or an integer priority, and now reject a malformed priority (#47498)
Fixed IndexError and wrong deployment lookups when deployments are deleted or added concurrently (#47680)
Fixed the router sending image requests to supports_vision: false deployments in mixed model groups (#48149, #48152)
Fixed 500s on every request to a model group after one negative or NaN latency sample under latency routing, broken since v1.102.0 (#48216)
Fixed image requests reaching declared text-only deployments unless enable_pre_call_checks was on (#48241)
Fixed one caller's exhausted forwarded Claude subscription putting the shared model into cooldown for others (#48485)
Fixed proxy-wide wildcard deployments shadowing a team's own wildcard deployment for that team's keys (#48490)
Changed per-deployment reasoning_effort_levels to apply on every provider and route, including azure_ai (#46601)
Changed router allowed_fails to keep a lone deployment with no fallback in rotation instead of returning 429 (#48644)
Plus 16 smaller fixes and updates (#45795, #46197, #46263, #46390, #46411, #47120, #47406, #47565, #47617, #47727, #48175, #48206, #48470, #48627, #48632, #48645)

Cost tracking and pricing (33)

Added one spend row per timed-out retry attempt with num_retries, so spend matches the provider invoice (#47416)
Fixed cached and reasoning token counts dropped from streamed usage, pricing streams at the full input rate (#36678)
Fixed Responses API image generation and usage-reported web search missing from cost_breakdown.tool_usage_cost (#41877)
Fixed streams with no backend usage being recorded at 0 tokens and $0 instead of the token estimate, broken since v1.104.0-dev.1 (#44230)
Fixed background interactions going unbilled after a crash, a late restart or on models with their own api_key (#45743)
Fixed calls billing $0 when the response model name only matches a price-free capability rule (#47037)
Fixed Tavily advanced and Linkup deep searches being billed at the basic and standard prices (#47411)
Fixed chat completions MCP auto-execute spend rows pricing only the first of its two paid model calls (#47422)
Fixed requests that time out at the provider logging $0 spend instead of their estimated input cost (#47435, #47436, #47437)
Fixed deployments with model_info.base_model such as azure/eu/o3 billing $0 on every request (#47497)
Fixed non-streamed requests adding double their recorded cost to key spend (#47510)
Fixed spend log batches being dropped when Postgres restarts mid-flush, so requests lost their spend row (#47571)
Fixed missing spend rows when clients disconnect from streamed /v1/responses and OpenAI or Azure /v1/messages (#47611)
Fixed spend priced from a caller-sent base_model and rows lost to lone surrogates in multipart metadata (#47639)
Fixed key, team, user and daily spend rows being dropped when the proxy stops right after a request (#48131)
Fixed Price Data Reload wiping custom pricing of DB models, which logged $0 spend on /v1/messages (#48486)
Plus 17 smaller fixes and updates (#46972, #46975, #47114, #47307, #47561, #47568, #47623, #47647, #47648, #47652, #47655, #47658, #48259, #48261, #48275, #48433, #48620)

Logging and observability (33)

Added maximum_spend_logs_size to delete the oldest spend logs first once they exceed a size limit (#47450, #47451, #47500)
Fixed Responses instructions still reaching request and response logs when message logging is turned off (#43315)
Fixed intermittent 500 dictionary changed size during iteration when caching or a sync callback is on (#46044)
Fixed provider-side 401s missing from litellm_deployment_failure_responses and litellm_deployment_state, broken since v1.80.13 (#46668)
Fixed the proxy printing plaintext at startup with json_logs on, so the banner and model list are now JSON (#47035)
Changed langfuse_otel to set trace-level name, input, output and tags on traces LiteLLM starts (#42705)
Changed rate-limited model groups to return rate_limit_error and Sentry to start at boot without stack locals (#46344)
Plus 26 smaller fixes and updates (#41150, #42740, #43594, #43604, #44185, #44187, #44199, #45555, #45597, #46956, #47019, #47045, #47124, #47129, #47156, #47175, #47182, #47183, #47192, #47384, #47449, #47456, #47468, #47848, #48239, #48613)

Keys, teams and auth (19)

Fixed /team/block and /team/unblock not applying to cached teams until the cache expired (#35643)
Fixed end-user budget resets never committing once LiteLLM_EndUserTable has millions of rows (#46502)
Fixed SAML SSO login 401 when the IdP callback lands on a different replica or worker (#46807)
Fixed lowered user, end-user or tag limits sometimes being ignored, and max_budget undercounting across workers (#46997)
Fixed SCIM group delete leaving memberships and members' team keys active, and removing members one at a time (#47544)
Fixed concurrency slots staying occupied after finished streams, so keys got rejected with SSE keepalive on (#48452)
Fixed team members' spend never resetting after a temporary budget increase on a member using the team default (#48495)
Changed free installs to refuse new SSO accounts past five users without locking out existing SSO users (#48097)
Plus 11 smaller fixes and updates (#41236, #44763, #46190, #46738, #46855, #46979, #47341, #47358, #48260, #48444, #48623)

Admin UI (54)

Changed the Chat page to be on by default, and enable_chat_ui: false still hides it (#46918)
Changed the Playground chat to a single composer with a Settings popover and a reply inspector panel (#46934)
Plus 52 smaller fixes and updates (#26667, #41350, #43425, #45674, #45893, #46064, #46778, #46830, #46834, #46941, #46978, #47025, #47065, #47440, #47552, #47612, #47630, #47653, #47662, #47671, #47974, #47998, #48074, #48081, #48082, #48094, #48105, #48106, #48107, #48109, #48123, #48124, #48137, #48199, #48207, #48223, #48225, #48425, #48431, #48451, #48454, #48456, #48464, #48478, #48487, #48488, #48500, #48504, #48532, #48629, #48630, #48634)

Deployment (12)

Added a pre-upgrade check for old spend-log rows that would fail the (api_key, startTime) index build (#48493)
Fixed terraform apply failing with Status: 201 Created on litellm_mcp_server and creating duplicates (#46550)
Fixed Terraform additional_drop_params deleting the keys it names instead of sending them to the proxy (#46782)
Removed the eMudhra root CA from certificates trusted by Python, Rust and the Docker image (#47670, #47969, #48139)
Plus 8 smaller fixes and updates (#43313, #46709, #46781, #47056, #47165, #47378, #48065, #48674)

SDK (4)

Plus 4 smaller fixes and updates (#48266, #48359, #48615, #48650)

Performance (11)

Fixed memory growth on proxies without Redis, which queued a budget spend increment per request (#46831)
Improved streaming so long buffered streams no longer stall other requests on the same worker (#46925)
Improved proxy latency by freezing the startup heap so gen-2 GC passes no longer pause requests for 0.6 to 0.8 s (#47137)
Plus 8 smaller fixes and updates (#46712, #46803, #47030, #47352, #47393, #47400, #47614, #48089)

Other (21)

Fixed 400 Invalid JSON on request bodies sent with Content-Encoding: zstd or gzip (#47418)
Plus 20 smaller fixes and updates (#43624, #44614, #46942, #47385, #47501, #47508, #47535, #47564, #47700, #47979, #47985, #48221, #48250, #48438, #48505, #48524, #48619, #48628, #48640, #48662)

Full changelog