What's Changed
- fix - allow non master key to access llm_utils_routes by @ishaan-jaff in #3710
- fix(bedrock_httpx.py): move anthropic bedrock calls to httpx by @krrishdholakia in #3708
- [Feat] Admin UI - use
base_modelfor Slack Alerts by @ishaan-jaff in #3713 - [Admin UI] show max input tokens on UI by @ishaan-jaff in #3714
- fix(proxy_server.py): fix setting model id for db models by @krrishdholakia in #3715
Full Changelog: v1.37.14...v1.37.16-stable
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.16-stable
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 9 | 11.476119168084793 | 1.5694795946554096 | 1.5694795946554096 | 470 | 470 | 7.414411000013388 | 79.26186800000323 |
| /health/liveliness | Failed ❌ | 8 | 10.598768169575871 | 15.497776167650544 | 15.497776167650544 | 4641 | 4641 | 6.259225999997398 | 982.9722660000186 |
| /health/readiness | Failed ❌ | 8 | 10.86070013134743 | 15.915190953463153 | 15.915190953463153 | 4766 | 4766 | 6.304408999994848 | 815.6038820000049 |
| Aggregated | Failed ❌ | 8 | 10.766908566366695 | 32.98244671576911 | 32.98244671576911 | 9877 | 9877 | 6.259225999997398 | 982.9722660000186 |