What's Changed
- Improve mocking in
test_proxy_server.pyby @msabramo in #3406 - Disambiguate invalid model name errors by @msabramo in #3403
- fix - revert init langfuse client on slack alerts by @ishaan-jaff in #3409
- Add Llama3 tokenizer and allow custom tokenizers. by @Priva28 in #3393
New Contributors
Full Changelog: v1.35.36...1.35.36.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 93 | 95.97100865434687 | 1.536506060425004 | 0.0 | 460 | 0 | 86.2270879999869 | 695.056527999995 |
| /health/liveliness | Passed ✅ | 78 | 80.44838701432926 | 15.385101987646888 | 0.0 | 4606 | 0 | 73.7866689999862 | 1317.1868050000057 |
| /health/readiness | Passed ✅ | 78 | 79.91031910375341 | 15.485308904631127 | 0.0 | 4636 | 0 | 73.8222589999964 | 1110.1436970000123 |
| Aggregated | Passed ✅ | 78 | 80.92724942630346 | 32.40691695270302 | 0.0 | 9702 | 0 | 73.7866689999862 | 1317.1868050000057 |