What's Changed
- Prometheus - custom metrics support + other improvements by @krrishdholakia in #7489
- (feat) POST
/fine_tuning/jobssupport passing vertex specific hyper params by @ishaan-jaff in #7490 - (Feat) - LiteLLM Use
UsernamePasswordCredentialfor Azure OpenAI by @ishaan-jaff in #7496 - (docs) Add docs on load testing benchmarks by @ishaan-jaff in #7499
- (Feat) Add support for reading secrets from Hashicorp vault by @ishaan-jaff in #7497
- Litellm dev 12 30 2024 p2 by @krrishdholakia in #7495
- Refactor Custom Metrics on Prometheus - allow setting k,v pairs on all metrics via config.yaml by @krrishdholakia in #7498
- (fix) GCS bucket logger - apply
truncate_standard_logging_payload_contenttostandard_logging_payloadand ensure GCS flushes queue on fails by @ishaan-jaff in #7500 - Litellm dev 01 01 2025 p3 by @krrishdholakia in #7503
Full Changelog: v1.56.6...v1.56.6.dev1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.56.6.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 230.0 | 255.89973974836954 | 6.151774848433542 | 0.003343355895887794 | 1840 | 1 | 94.9865199999067 | 1259.9916519999965 |
| Aggregated | Passed ✅ | 230.0 | 255.89973974836954 | 6.151774848433542 | 0.003343355895887794 | 1840 | 1 | 94.9865199999067 | 1259.9916519999965 |