What's Changed
- [Docs] update gunicorn instructions - Uvicorn perf is significantly better on K8s by @ishaan-jaff in #2667
- [Docs] Add Docs on deploying to EKS Cluster + K8 by @ishaan-jaff in #2668
- feat(router.py): enable pre-call checks by @krrishdholakia in #2669
- (docs) best prod practices by @ishaan-jaff in #2670
- fix(proxy_server.py): fix model info check by @krrishdholakia in #2666
Full Changelog: v1.33.8...v1.33.9
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 54 | 60.228549854209994 | 1.6267984265796072 | 0.0 | 487 | 0 | 49.45159799999033 | 794.4530399999792 |
| /health/liveliness | Passed ✅ | 40 | 43.17110106318068 | 15.650001290606694 | 0.0 | 4685 | 0 | 38.61002399997915 | 1231.5367759999845 |
| /health/readiness | Passed ✅ | 40 | 42.72228626508686 | 15.222423880745934 | 0.0 | 4557 | 0 | 38.460548000017525 | 1388.6105939999993 |
| Aggregated | Passed ✅ | 40 | 43.814715877274416 | 32.499223597932236 | 0.0 | 9729 | 0 | 38.460548000017525 | 1388.6105939999993 |