BerriAI/litellm v1.10.4

on GitHub

Note: Proxy Server on 1.10.4 has a bug for non OpenAI LLMs - Fixed on 1.10.11

Updates Proxy Server

• Use custom callbacks on the proxy https://docs.litellm.ai/docs/proxy/logging
• Set timeout and stream_timeout per model https://docs.litellm.ai/docs/proxy/load_balancing#custom-timeouts-stream-timeouts---per-model
• Stability: Added testing for reading config.yaml on the proxy
• NEW /model/new + /model/info endpoints - Add new models + Get model info without restarting proxy.
• Custom user auth - #898 (comment)
• Key Security -> keys now stored as just hashes in the db
• user id accepted + passed to OpenAI/Azure

litellm Package

• Specify kwargs for Redis Cache 9ba1765
• Fixes for Sagemaker + Palm Streaming
• Support for async custom callbacks - https://docs.litellm.ai/docs/observability/custom_callback#async-callback-functions
• Major improvements to stream chunk builder - support for parallel tool calling, system fingerprints, etc.
• Fixes for azure / openai streaming (return complete response object)
• Support for loading keys from azure key vault - https://docs.litellm.ai/docs/secret#azure-key-vault

What's Changed

• docs: adds gpt-3.5-turbo-1106 in supported models by @rishabgit in #958
• (feat) Allow installing proxy dependencies explicitly with pip install litellm[proxy] by @PSU3D0 in #966
• Mention Neon as a database option in docs by @Manouchehri in #977
• fix system prompts for replicate by @nbaldwin98 in #970

New Contributors

• @rishabgit made their first contribution in #958
• @PSU3D0 made their first contribution in #966
• @nbaldwin98 made their first contribution in #970

Full Changelog: v1.7.11...v1.10.4