What's Changed
- Controll fallback prompts client-side by @krrishdholakia in #7334
- [Bug fix ]: Triton /infer handler incompatible with batch responses by @ishaan-jaff in #7337
- Litellm dev 12 20 2024 p3 by @krrishdholakia in #7339
- Litellm dev 2024 12 20 p1 by @krrishdholakia in #7335
- (fix) LiteLLM Proxy fix GET
/files/{file_id:path}/content"endpoint by @ishaan-jaff in #7342 - (Bug fix) Azure cost calculation -
dall-e-3by @ishaan-jaff in #7343
Full Changelog: v1.55.8...v1.55.9
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.9
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 240.0 | 270.2192842992925 | 6.152704904591068 | 0.0 | 1841 | 0 | 213.0105499999786 | 2430.5650640000067 |
| Aggregated | Passed ✅ | 240.0 | 270.2192842992925 | 6.152704904591068 | 0.0 | 1841 | 0 | 213.0105499999786 | 2430.5650640000067 |