BerriAI/litellm v1.70.0-nightly

on GitHub

What's Changed

• Capture Gemini reasoning tokens usage in streaming mode by @Dima-Mediator in #10789
• Anthropic unified web search + tool cost tracking support by @krrishdholakia in #10846
• avoid error AttributeError: 'Message' object has no attribute 'audio' by @igtm in #10814
• fix: bedrock-sonnet-caching-cost by @shibaboy in #10752
• Fix/issue 10113 embeddings use non default tokenizer by @camfarineau in #10629
• Add models import command by @msabramo in #10581
• Add documentation on in-memory and disk caching by @user202729 in #10522
• Bedrock pixtral fix by @ajac-zero in #10439
• added fix to resolve delete API by @jugaldb in #10845
• build(deps): bump undici from 6.21.1 to 6.21.3 in /docs/my-website by @dependabot in #10855
• [Refactor] Make Pagerduty a free feature by @ishaan-jaff in #10857
• [Refactor] Move enterprise_routes within litellm_enterprise by @ishaan-jaff in #10860
• refactor: make send_management_endpoint_alert a free feature by @ishaan-jaff in #10863
• [Feat - Guardrails] Expose /apply_guardrail endpoint for directly calling guardrail by @ishaan-jaff in #10867
• [CI/CD] Test: fix mapped tests by @ishaan-jaff in #10870
• [Feat] UI: Add support for Deleting Guardrail on UI by @ishaan-jaff in #10869
• [UI] Allow adding Bedrock, Presidio, Lakera, AIM guardrails on UI by @ishaan-jaff in #10874
• fix(proxy/_types.py): add missing comma for /v2/rerank by @krrishdholakia in #10861
• fix(streaming_handler.py): patch vertex ai meta llama streaming error by @krrishdholakia in #10878
• build(dependencies)📦: Add numpydoc to dependencies in pyproject.toml by @ericmjl in #10736
• [Feat]: Configure LiteLLM to Parse User Headers from Open Web UI by @damgem in #9802
• Feat/support anonymize in aim guardrail by @hxdror in #10757
• Gemini Multimodal Live API support by @krrishdholakia in #10841
• Feat log retention cleanup by @jugaldb in #10872
• Model filter added on logs tab. (#10877) by @ishaan-jaff in #10891
• fix: fix novita ai docs by @jasonhp in #10882
• [Feat] Add Tracing for guardrails in StandardLoggingPayload, Langfuse by @ishaan-jaff in #10890
• [UI] - Add Guardrail Tracing to LiteLLM SpendLogs by @ishaan-jaff in #10893
• Fix example commands in reliability.md by @wwwillchen in #10894
• Feat: add guardrail tracing to OTEL, Arize phoenix by @ishaan-jaff in #10896
• Key hash filter added by @NANDINI-star in #10903
• [Feat] UI: Allow clicking into a guardrail by @ishaan-jaff in #10904
• [Feat] Add Lakera v2 Guardrail Support by @ishaan-jaff in #10880
• Key alias filter on logs by @NANDINI-star in #10905
• [Feat] handle multi-pod deployment for SpendLogs Retention by @jugaldb in #10895
• [UI] Fixes for using Lakera Guardrail on LiteLLM UI by @ishaan-jaff in #10906
• add constant to config settings.md by @jugaldb in #10908
• Remove extra test file by @jugaldb in #10889
• [UI] Allow editing guardrails by @ishaan-jaff in #10907

New Contributors

• @Dima-Mediator made their first contribution in #10789
• @igtm made their first contribution in #10814
• @shibaboy made their first contribution in #10752
• @camfarineau made their first contribution in #10629
• @ajac-zero made their first contribution in #10439
• @damgem made their first contribution in #9802
• @hxdror made their first contribution in #10757
• @wwwillchen made their first contribution in #10894

Full Changelog: v1.69.3-nightly...v1.70.0-nightly

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.0-nightly

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.70.0-nightly

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results