What's Changed
- (feat) use
@google-cloud/vertexaijs sdk with litellm by @ishaan-jaff in #6873 - (chore) fix new .js tests running for vertex.js by @ishaan-jaff in #6875
- Bump cross-spawn from 7.0.3 to 7.0.6 in /ui/litellm-dashboard by @dependabot in #6865
- (Perf / latency improvement) improve pass through endpoint latency to ~50ms (before PR was 400ms) by @ishaan-jaff in #6874
- LiteLLM Minor Fixes & Improvements (11/23/2024) by @krrishdholakia in #6870
- Litellm dev 11 23 2024 by @krrishdholakia in #6881
- docs - have 1 section for routing +load balancing by @ishaan-jaff in #6884
- (QOL improvement) Provider budget routing - allow using 1s, 1d, 1mo, 2mo etc by @ishaan-jaff in #6885
- (feat) - provider budget improvements - ensure provider budgets work with multiple proxy instances + improve latency to ~90ms by @ishaan-jaff in #6886
Full Changelog: v1.52.14...v1.52.15
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.52.15
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 280.0 | 454.59782761891177 | 5.830264051408934 | 0.010023376592680114 | 1745 | 3 | 139.27931299997454 | 5766.263976999994 |
| Aggregated | Failed ❌ | 280.0 | 454.59782761891177 | 5.830264051408934 | 0.010023376592680114 | 1745 | 3 | 139.27931299997454 | 5766.263976999994 |