What's Changed
1. Improved litellm.Router logging for fallbacks
Example Log for one call
LiteLLM Router - INFO: get_available_deployment for model: azure/gpt-4-fast, No deployment available
LiteLLM Router - INFO: litellm.completion(model=None) Exception No models available.
LiteLLM Router - INFO: get_available_deployment for model: azure/gpt-4-basic, No deployment available
LiteLLM Router - INFO: litellm.completion(model=None) Exception No models available.
LiteLLM Router - INFO: get_available_deployment for model: openai-gpt-4, Selected deployment: {'model_name': 'openai-gpt-4', 'litellm_params': {'model': 'gpt-3.5-turbo', 'api_key': 'sk-PhEM****', 'tpm': 2000}, 'tpm': 2000, 'model_info': {'id': '5a4b95fa-c018-4767-85c2-c4851c57cf34'}} for model: openai-gpt-4
LiteLLM Router - INFO: litellm.completion(model=gpt-3.5-turbo) 200 OKHow to use in python
router = litellm.Router(
model_list=model_list,
fallbacks=fallbacks_list,
set_verbose=True,
debug_level="DEBUG" # optional, default=INFO
)2. Improvements to Usage Based Routing - litellm.Router
Before making the first call, we check if any of the deployments have TPM to make the call, Thanks @georgeseifada for this!
3. [Feat] Add typehints for litellm.Router by @ishaan-jaff in #1529
Full Changelog: v1.18.6...v1.18.7