What's Changed
- [Feat] Add Exception mapping for Azure ContentPolicyViolationError by @ishaan-jaff in #3449
- Add return_exceptions to litellm.batch_completion by @ffreemt in #3397
- fix(caching.py): fix redis caching ping check by @krrishdholakia in #3447
- change max_tokens type to int by @TanaroSch in #1530
- Revert "Add return_exceptions to litellm.batch_completion" by @ishaan-jaff in #3450
- fix(factory.py): support 'function' openai message role for anthropic by @krrishdholakia in #3448
- [Feat] Return model, api_base and first 100 chars of messages in Azure Exceptions by @ishaan-jaff in #3451
- [FEAT] router set custom num retries for ContentPolicyViolationErrorRetries, RateLimitErrorRetries, BadRequestErrorRetries etc by @ishaan-jaff in #3456
- [Feat] return num_retries in litellm.Router exceptions by @ishaan-jaff in #3457
- [Feat] Set a Retry Policy per model group by @ishaan-jaff in #3460
- Fix OpenMeter sync logger by @tothandras in #3452
- feat(openai.py): add support for openai assistants by @krrishdholakia in #3455
- gunicorn version bump by @RoniGurvichCycode in #3463
- Fix Ollama streamed tool calls. Set finish_reason to tool_calls for all tool_calls responses by @jackmpcollins in #3469
- Allowing extra headers for bedrock by @themrzmaster in #3299
New Contributors
- @ffreemt made their first contribution in #3397
- @TanaroSch made their first contribution in #1530
- @tothandras made their first contribution in #3452
- @RoniGurvichCycode made their first contribution in #3463
Full Changelog: v1.35.38-stable...v1.36.0
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 81 | 88.20979605555442 | 1.623255922227879 | 0.0 | 486 | 0 | 75.38953300002049 | 1264.5359969999959 |
| /health/liveliness | Passed ✅ | 65 | 68.12845653229724 | 15.253929623075564 | 0.0 | 4567 | 0 | 63.39287800000193 | 1385.0202130000184 |
| /health/readiness | Passed ✅ | 65 | 68.59345058785526 | 15.511112145733067 | 0.0033400327617857596 | 4644 | 1 | 63.46367399999053 | 1491.452105999997 |
| Aggregated | Passed ✅ | 65 | 69.35759579210092 | 32.38829769103651 | 0.0033400327617857596 | 9697 | 1 | 63.39287800000193 | 1491.452105999997 |