BerriAI/litellm v1.76.3.rc.1

on GitHub

What's Changed

• [Feat] Add Initial support for Bedrock Batches API by @ishaan-jaff in #14190
• [Feat] Allow using Veo Video Generation through LiteLLM Pass through routes by @ishaan-jaff in #14228
• [Security] fix memory_usage_in_mem_cache cache endpoint vulnerability by @ishaan-jaff in #14229
• fix: Key Budget not resets at expectable times by @22mSqRi in #14241
• Fixes #14204 TPM Rate Limit Bug by @yeahyung in #14237
• [docs]: added more info to load balancing & pass through endpoints by @mubashir1osmani in #14232
• Fix custom callbacks doc by @mubashir1osmani in #14216
• [docs]: fixed prometheus metrics by @mubashir1osmani in #14199
• fix: image_generation supports extra_body parameter by @zhxlp in #14211
• Litellm passthrough cost tracking chat completion by @kankute-sameer in #14256
• [BUG] Fix response api for reasoning item in input for litellm proxy by @kankute-sameer in #14200
• fix: Support AWS_BEDROCK_RUNTIME_ENDPOINT on bedrock passthrough, make work for URLs with a base path by @byrongrogan in #14156
• Add client side pagination on All Models table by @NANDINI-star in #14136
• Model Filters UI improvement by @NANDINI-star in #14131
• Bedrock fix structure output by @moshemorad in #14130
• [Fix] DD LLM Observability - Ensure apm_id is set on traces by @ishaan-jaff in #14272
• Validate store model in db setting by @ishaan-jaff in #14269
• [Fix] Ensure team_id is a required field for generating service account keys by @ishaan-jaff in #14270
• Add support for anthropic citation api in Databricks by @TomeHirata in #14077
• feat: Add guardrail to the Anthropic API endpoint by @uc4w6c in #14107
• Add Volcengine embedding module with handler and transformation logic by @onlylhf in #14028
• Team name badge added on the User Details by @NANDINI-star in #14003
• [Bug Fix] x-litellm-tags not routing with Responses API by @ishaan-jaff in #14289
• feat: add redis ssl and username support by @btpemercier in #11319
• [Feat] Add pass through image gen and image editing on OpenAI by @kankute-sameer in #14292
• [Feat] Litellm x CloudZero Integration - Cost Tracking by @ishaan-jaff in #14296
• [Feat]Cancel upstream on client disconnect by @eycjur in #14295
• Change DEFAULT_NUM_WORKERS_LITELLM_PROXY default to number CPUs by @dttran-glo in #14242
• Add model to azure error message by @ishaan-jaff in #14294
• [docs]: added loom for claude code by @mubashir1osmani in #14223
• Bug fix for openai.gpt-oss when using reasoning_effort parameter by @eycjur in #14300
• fix: correct output pricing for gemini-2.5-flash-image-preview by @tremlin in #14287
• feat: Add support for reasoning_effort='minimal' for Gemini models by @TobiMayr in #14262
• Add additionalProperties to vertex ai Schema definition by @Eitan1112 in #14252
• [BUG] Responses : add structured output for sdk by @kankute-sameer in #14206
• bug(auth): support for ES256/ES384/ES512 and EdDSA JWT verification by @iabhi4 in #14118
• Revert "feat: add redis ssl and username support" by @ishaan-jaff in #14303
• Revert "[Feat]Cancel upstream on client disconnect" by @ishaan-jaff in #14304

New Contributors

• @22mSqRi made their first contribution in #14241
• @onlylhf made their first contribution in #14028
• @btpemercier made their first contribution in #11319
• @tremlin made their first contribution in #14287
• @TobiMayr made their first contribution in #14262
• @Eitan1112 made their first contribution in #14252

Full Changelog: v1.76.2-nightly...v1.76.3.rc.1

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.76.3.rc.1

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results