What's Changed
- Add sync iterator by @turmind in #2794
- fix(proxy_server.py): allow user to set in-memory + redis ttl by @krrishdholakia in #2789
- Fix max_tokens type in main.py by @phact in #2790
New Contributors
Full Changelog: v1.34.20...v1.34.21
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 82 | 99.44767344857672 | 1.5262768715973858 | 0.0 | 457 | 0 | 76.33821500002114 | 1827.514999000016 |
| /health/liveliness | Passed ✅ | 66 | 68.59761310860081 | 15.376321043401234 | 0.0 | 4604 | 0 | 63.56413200001043 | 1420.0152670000534 |
| /health/readiness | Passed ✅ | 66 | 68.43111482729716 | 15.122498193857686 | 0.0 | 4528 | 0 | 63.35147300001154 | 1183.1644380000057 |
| Aggregated | Passed ✅ | 66 | 69.9892674372716 | 32.0250961088563 | 0.0 | 9589 | 0 | 63.35147300001154 | 1827.514999000016 |