LiteLLM Gateway Setup Intermediate

LiteLLM is the leading open-source LLM gateway that provides a unified OpenAI-compatible API for 100+ LLM providers. It handles provider-specific API translations, making it easy to switch between models without changing application code.

Installation & Deployment

• Deploy LiteLLM as a Docker container or Kubernetes service. Use the proxy server mode for production gateway deployments.
• Configure a PostgreSQL database for persistent storage of API keys, usage data, and configuration.

Provider Configuration

• Configure model providers in the LiteLLM config file. Map model aliases to specific provider endpoints for flexible routing.
• Set up credentials for each provider: OpenAI, Anthropic, Azure OpenAI, Google Vertex AI, AWS Bedrock, and self-hosted models.
• Use model aliases to abstract provider details from application teams. They request "gpt-4-equivalent" rather than specific provider models.

Authentication Setup

• Generate virtual API keys for each team or project. Virtual keys track usage and enforce limits without exposing provider API keys.
• Integrate with SSO/OIDC for user-level authentication and authorization on the admin interface.
• Configure key-level permissions: which models each key can access, request rate limits, and budget caps.

Basic Routing

• Configure default models and fallback chains. If the primary model is unavailable, requests automatically route to the fallback.
• Set up model-specific parameters: max tokens, temperature defaults, and system prompt injection per virtual key.