What's Changed
- (azure): Enable stream_options for Azure OpenAI. (#6024) by @krrishdholakia in #6029
- (load testing) add vertex_ai embeddings load test by @ishaan-jaff in #6004
- (feat proxy) add key based logging for GCS bucket by @ishaan-jaff in #6031
- (feat) add nvidia nim embeddings by @ishaan-jaff in #6032
- (contributor PRs) oct 3rd, 2024 by @ishaan-jaff in #6034
- fix(factory.py): bedrock: merge consecutive tool + user messages by @krrishdholakia in #6028
- (feat) openai prompt caching (non streaming) - add prompt_tokens_details in usage response by @ishaan-jaff in #6039
Full Changelog: v1.48.10...v1.48.11
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.48.11
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 140.0 | 160.94989321263134 | 6.350767242529357 | 0.0 | 1900 | 0 | 112.36911899993629 | 2553.2338090000053 |
| Aggregated | Passed ✅ | 140.0 | 160.94989321263134 | 6.350767242529357 | 0.0 | 1900 | 0 | 112.36911899993629 | 2553.2338090000053 |