What's Changed
- Litellm Minor Fixes & Improvements (10/12/2024) by @krrishdholakia in #6179
- build(config.yml): add codecov to repo by @krrishdholakia in #6172
- ci(config.yml): add local_testing tests to codecov coverage check by @krrishdholakia in #6183
- ci(config.yml): add further testing coverage to codecov by @krrishdholakia in #6184
- docs(configs.md): document all environment variables by @krrishdholakia in #6185
- (feat) add components to codecov yml by @ishaan-jaff in #6207
- (refactor) caching use LLMCachingHandler for async_get_cache and set_cache by @ishaan-jaff in #6208
- (feat) prometheus have well defined latency buckets by @ishaan-jaff in #6211
- (refactor caching) use LLMCachingHandler for caching streaming responses by @ishaan-jaff in #6210
- bump @getcanary/web@1.0.9 by @yujonglee in #6187
- (refactor caching) use common
_retrieve_from_cachehelper by @ishaan-jaff in #6212
Full Changelog: v1.49.2...v1.49.3
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.49.3
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 270.0 | 313.92829846432517 | 5.997591789487422 | 0.0 | 1794 | 0 | 231.2673659999973 | 3079.4730589999517 |
| Aggregated | Failed ❌ | 270.0 | 313.92829846432517 | 5.997591789487422 | 0.0 | 1794 | 0 | 231.2673659999973 | 3079.4730589999517 |