What's Changed
- [Feat]- Proxy Add OpenAI Content Moderation Pre call hook by @ishaan-jaff in #3802
- Athina callback now handles streaming mode by @vivek-athina in #3071
- [Fix] async_post_call_streaming_hook not triggered on proxy server by @ishaan-jaff in #3797
- [Feat] Add Lakera AI Prompt Injection Detection by @ishaan-jaff in #3803
- [Feat] Add Cost Tracking for
vertex_ai/imagegeneration@006by @ishaan-jaff in #3809 - [Feat] LiteLLM Proxy - Require Enterprise License Key for Premium Guardrail Features by @ishaan-jaff in #3806
- Added script for auto updating the pricing with the pricings of openrouter.ai by @vietpham1911 in #3630
- build(model_prices_and_context_window.json): update Anyscale model list by @danielbichuetti in #3807
- fix(factory.py): Ollama vision fix. by @miesgre in #3792
- feat(databricks.py): adds databricks support - completion, embeddings, async, streaming by @krrishdholakia in #3808
New Contributors
- @vietpham1911 made their first contribution in #3630
- @danielbichuetti made their first contribution in #3807
- @miesgre made their first contribution in #3792
Full Changelog: v1.38.1...v1.38.2
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.38.2
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 22 | 22.735412932242443 | 1.4295949356664561 | 1.4295949356664561 | 428 | 428 | 17.413049000026604 | 110.43518400003904 |
| /health/liveliness | Failed ❌ | 21 | 23.353646324823114 | 15.568556063414373 | 15.568556063414373 | 4661 | 4661 | 16.87973199989301 | 1207.890811000027 |
| /health/readiness | Failed ❌ | 21 | 23.329244846860398 | 15.638699740164363 | 15.638699740164363 | 4682 | 4682 | 16.783239999995203 | 453.8819480000029 |
| Aggregated | Failed ❌ | 21 | 23.314873260464807 | 32.63685073924519 | 32.63685073924519 | 9771 | 9771 | 16.783239999995203 | 1207.890811000027 |