Library to easily interface with LLM API providers
Project description
๐ LiteLLM
Call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]
LiteLLM Proxy Server (LLM Gateway) | Hosted Proxy (Preview) | Enterprise Tier
LiteLLM manages:
- Translate inputs to provider's
completion,embedding, andimage_generationendpoints - Consistent output, text responses will always be available at
['choices'][0]['message']['content'] - Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router
- Set Budgets & Rate limits per project, api key, model LiteLLM Proxy Server (LLM Gateway)
Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers
๐จ Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published.
Support for more providers. Missing a provider or LLM Platform, raise a feature request.
Usage (Docs)
[!IMPORTANT] LiteLLM v1.0.0 now requires
openai>=1.0.0. Migration guide here
LiteLLM v1.40.14+ now requirespydantic>=2.0.0. No changes required.
pip install litellm
from litellm import completion
import os
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-cohere-key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="openai/gpt-4o", messages=messages)
# anthropic call
response = completion(model="anthropic/claude-3-sonnet-20240229", messages=messages)
print(response)
Response (OpenAI Format)
{
"id": "chatcmpl-565d891b-a42e-4c39-8d14-82a1f5208885",
"created": 1734366691,
"model": "claude-3-sonnet-20240229",
"object": "chat.completion",
"system_fingerprint": null,
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "Hello! As an AI language model, I don't have feelings, but I'm operating properly and ready to assist you with any questions or tasks you may have. How can I help you today?",
"role": "assistant",
"tool_calls": null,
"function_call": null
}
}
],
"usage": {
"completion_tokens": 43,
"prompt_tokens": 13,
"total_tokens": 56,
"completion_tokens_details": null,
"prompt_tokens_details": {
"audio_tokens": null,
"cached_tokens": 0
},
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 0
}
}
Call any model supported by a provider, with model=<provider_name>/<model_name>. There might be provider-specific details here, so refer to provider docs for more information
Async (Docs)
from litellm import acompletion
import asyncio
async def test_get_response():
user_message = "Hello, how are you?"
messages = [{"content": user_message, "role": "user"}]
response = await acompletion(model="openai/gpt-4o", messages=messages)
return response
response = asyncio.run(test_get_response())
print(response)
Streaming (Docs)
liteLLM supports streaming the model response back, pass stream=True to get a streaming iterator in response.
Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
from litellm import completion
response = completion(model="openai/gpt-4o", messages=messages, stream=True)
for part in response:
print(part.choices[0].delta.content or "")
# claude 2
response = completion('anthropic/claude-3-sonnet-20240229', messages, stream=True)
for part in response:
print(part)
Response chunk (OpenAI Format)
{
"id": "chatcmpl-2be06597-eb60-4c70-9ec5-8cd2ab1b4697",
"created": 1734366925,
"model": "claude-3-sonnet-20240229",
"object": "chat.completion.chunk",
"system_fingerprint": null,
"choices": [
{
"finish_reason": null,
"index": 0,
"delta": {
"content": "Hello",
"role": "assistant",
"function_call": null,
"tool_calls": null,
"audio": null
},
"logprobs": null
}
]
}
Logging Observability (Docs)
LiteLLM exposes pre defined callbacks to send data to Lunary, Langfuse, DynamoDB, s3 Buckets, Helicone, Promptlayer, Traceloop, Athina, Slack, MLflow
from litellm import completion
## set env variables for logging tools
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key"
os.environ["HELICONE_API_KEY"] = "your-helicone-auth-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["ATHINA_API_KEY"] = "your-athina-api-key"
os.environ["OPENAI_API_KEY"]
# set callbacks
litellm.success_callback = ["lunary", "langfuse", "athina", "helicone"] # log input/output to lunary, langfuse, supabase, athina, helicone etc
#openai call
response = completion(model="anthropic/claude-3-sonnet-20240229", messages=[{"role": "user", "content": "Hi ๐ - i'm openai"}])
LiteLLM Proxy Server (LLM Gateway) - (Docs)
Track spend + Load Balance across multiple projects
The proxy provides:
๐ Proxy Endpoints - Swagger Docs
Quick Start Proxy - CLI
pip install 'litellm[proxy]'
Step 1: Start litellm proxy
$ litellm --model huggingface/bigcode/starcoder
#INFO: Proxy running on http://0.0.0.0:4000
Step 2: Make ChatCompletions Request to Proxy
[!IMPORTANT] ๐ก Use LiteLLM Proxy with Langchain (Python, JS), OpenAI SDK (Python, JS) Anthropic SDK, Mistral SDK, LlamaIndex, Instructor, Curl
import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:4000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
}
])
print(response)
Proxy Key Management (Docs)
Connect the proxy with a Postgres DB to create proxy keys
# Get the code
git clone https://github.com/BerriAI/litellm
# Go to folder
cd litellm
# Add the master key - you can change this after setup
echo 'LITELLM_MASTER_KEY="sk-1234"' > .env
# Add the litellm salt key - you cannot change this after adding a model
# It is used to encrypt / decrypt your LLM API Key credentials
# We recommned - https://1password.com/password-generator/
# password generator to get a random hash for litellm salt key
echo 'LITELLM_SALT_KEY="sk-1234"' > .env
source .env
# Start
docker-compose up
UI on /ui on your proxy server
Set budgets and rate limits across multiple projects
POST /key/generate
Request
curl 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data-raw '{"models": ["gpt-3.5-turbo", "gpt-4", "claude-2"], "duration": "20m","metadata": {"user": "ishaan@berri.ai", "team": "core-infra"}}'
Expected Response
{
"key": "sk-kdEXbIqZRwEeEiHwdg7sFA", # Bearer token
"expires": "2023-11-19T01:38:25.838000+00:00" # datetime object
}
Supported Providers (Docs)
| Provider | Completion | Streaming | Async Completion | Async Streaming | Async Embedding | Async Image Generation |
|---|---|---|---|---|---|---|
| openai | โ | โ | โ | โ | โ | โ |
| azure | โ | โ | โ | โ | โ | โ |
| aws - sagemaker | โ | โ | โ | โ | โ | |
| aws - bedrock | โ | โ | โ | โ | โ | |
| google - vertex_ai | โ | โ | โ | โ | โ | โ |
| google - palm | โ | โ | โ | โ | ||
| google AI Studio - gemini | โ | โ | โ | โ | ||
| mistral ai api | โ | โ | โ | โ | โ | |
| cloudflare AI Workers | โ | โ | โ | โ | ||
| cohere | โ | โ | โ | โ | โ | |
| anthropic | โ | โ | โ | โ | ||
| empower | โ | โ | โ | โ | ||
| huggingface | โ | โ | โ | โ | โ | |
| replicate | โ | โ | โ | โ | ||
| together_ai | โ | โ | โ | โ | ||
| openrouter | โ | โ | โ | โ | ||
| ai21 | โ | โ | โ | โ | ||
| baseten | โ | โ | โ | โ | ||
| vllm | โ | โ | โ | โ | ||
| nlp_cloud | โ | โ | โ | โ | ||
| aleph alpha | โ | โ | โ | โ | ||
| petals | โ | โ | โ | โ | ||
| ollama | โ | โ | โ | โ | โ | |
| deepinfra | โ | โ | โ | โ | ||
| perplexity-ai | โ | โ | โ | โ | ||
| Groq AI | โ | โ | โ | โ | ||
| Deepseek | โ | โ | โ | โ | ||
| anyscale | โ | โ | โ | โ | ||
| IBM - watsonx.ai | โ | โ | โ | โ | โ | |
| voyage ai | โ | |||||
| xinference [Xorbits Inference] | โ | |||||
| FriendliAI | โ | โ | โ | โ | ||
| Galadriel | โ | โ | โ | โ |
Contributing
To contribute: Clone the repo locally -> Make a change -> Submit a PR with the change.
Here's how to modify the repo locally: Step 1: Clone the repo
git clone https://github.com/BerriAI/litellm.git
Step 2: Navigate into the project, and install dependencies:
cd litellm
poetry install -E extra_proxy -E proxy
Step 3: Test your change:
cd litellm/tests # pwd: Documents/litellm/litellm/tests
poetry run flake8
poetry run pytest .
Step 4: Submit a PR with your changes! ๐
- push your fork to your GitHub repo
- submit a PR from there
Building LiteLLM Docker Image
Follow these instructions if you want to build / run the LiteLLM Docker Image yourself.
Step 1: Clone the repo
git clone https://github.com/BerriAI/litellm.git
Step 2: Build the Docker Image
Build using Dockerfile.non_root
docker build -f docker/Dockerfile.non_root -t litellm_test_image .
Step 3: Run the Docker Image
Make sure config.yaml is present in the root directory. This is your litellm proxy config file.
docker run \
-v $(pwd)/proxy_config.yaml:/app/config.yaml \
-e DATABASE_URL="postgresql://xxxxxxxx" \
-e LITELLM_MASTER_KEY="sk-1234" \
-p 4000:4000 \
litellm_test_image \
--config /app/config.yaml --detailed_debug
Enterprise
For companies that need better security, user management and professional support
This covers:
- โ Features under the LiteLLM Commercial License:
- โ Feature Prioritization
- โ Custom Integrations
- โ Professional Support - Dedicated discord + slack
- โ Custom SLAs
- โ Secure access with Single Sign-On
Code Quality / Linting
LiteLLM follows the Google Python Style Guide.
We run:
- Ruff for formatting and linting checks
- Mypy + Pyright for typing 1, 2
- Black for formatting
- isort for import sorting
If you have suggestions on how to improve the code quality feel free to open an issue or a PR.
Support / talk with founders
- Schedule Demo ๐
- Community Discord ๐ญ
- Our numbers ๐ +1 (770) 8783-106 / โญ+1 (412) 618-6238โฌ
- Our emails โ๏ธ ishaan@berri.ai / krrish@berri.ai
Why did we build this
- Need for simplicity: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI and Cohere.
Contributors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file litellm-1.57.5.tar.gz.
File metadata
- Download URL: litellm-1.57.5.tar.gz
- Upload date:
- Size: 6.3 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.0.1 CPython/3.8.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
981461d67dc851ee4dd204c238e9528cbeffb3001a7e9cf2d5c32982373335af
|
|
| MD5 |
cbc60b9e2ddb2ee8b4900822e7e2336b
|
|
| BLAKE2b-256 |
54b3636fe74190e68450df6a87a1694f280f0c1258919d8a7acdd0962feaebf1
|
File details
Details for the file litellm-1.57.5-py3-none-any.whl.
File metadata
- Download URL: litellm-1.57.5-py3-none-any.whl
- Upload date:
- Size: 6.6 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.0.1 CPython/3.8.20
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
522ea3cbe0e348947bd9473897d0d28df35fb8b0dc5d7e3540c441c01b5ac821
|
|
| MD5 |
6f375aa14ced2255bac15dbcf1fb9a18
|
|
| BLAKE2b-256 |
96f8ac93f8764a9c6a8663feb773849636d6fccdd0b1c4965f1e47dcc3060ce8
|
