What's Changed
- build(deployment.yaml): Fix port + allow setting database url in helm chart by @krrishdholakia in #5587
- [Feat] support using "callbacks" for prometheus by @ishaan-jaff in #5599
- Properly use
allowed_fails_policywhen it has fields with a value of 0 by @eladsegal in #5604 - [Feat-Proxy] Allow using key based logging for success and failure by @ishaan-jaff in #5606
- [Fix - Otel logger] Set a max queue size of 100 logs for OTEL by @ishaan-jaff in #5576
- [Feat] Tag Routing - Allow setting default deployments by @ishaan-jaff in #5601
- LiteLLM Minor Fixes and Improvements (09/07/2024) by @krrishdholakia in #5580
- LiteLLM Minor Fixes and Improvements (09/09/2024) by @krrishdholakia in #5602
Full Changelog: v1.44.22...v1.44.23-stable
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.44.23-stable
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 130.0 | 153.73045870969366 | 6.342069904674704 | 0.0 | 1898 | 0 | 106.94551199992475 | 2384.351408000043 |
| Aggregated | Passed ✅ | 130.0 | 153.73045870969366 | 6.342069904674704 | 0.0 | 1898 | 0 | 106.94551199992475 | 2384.351408000043 |