Library to easily interface with LLM API providers
Project description
๐ LiteLLM
Call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, etc.]
OpenAI Proxy Server | Hosted Proxy (Preview) | Enterprise Tier
LiteLLM manages:
- Translate inputs to provider's
completion,embedding, andimage_generationendpoints - Consistent output, text responses will always be available at
['choices'][0]['message']['content'] - Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router
- Set Budgets & Rate limits per project, api key, model OpenAI Proxy Server
Jump to OpenAI Proxy Docs
Jump to Supported LLM Providers
๐จ Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published.
Support for more providers. Missing a provider or LLM Platform, raise a feature request.
Usage (Docs)
[!IMPORTANT] LiteLLM v1.0.0 now requires
openai>=1.0.0. Migration guide here
LiteLLM v1.40.14+ now requirespydantic>=2.0.0. No changes required.
pip install litellm
from litellm import completion
import os
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["COHERE_API_KEY"] = "your-cohere-key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion(model="command-nightly", messages=messages)
print(response)
Call any model supported by a provider, with model=<provider_name>/<model_name>. There might be provider-specific details here, so refer to provider docs for more information
Async (Docs)
from litellm import acompletion
import asyncio
async def test_get_response():
user_message = "Hello, how are you?"
messages = [{"content": user_message, "role": "user"}]
response = await acompletion(model="gpt-3.5-turbo", messages=messages)
return response
response = asyncio.run(test_get_response())
print(response)
Streaming (Docs)
liteLLM supports streaming the model response back, pass stream=True to get a streaming iterator in response.
Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
from litellm import completion
response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
for part in response:
print(part.choices[0].delta.content or "")
# claude 2
response = completion('claude-2', messages, stream=True)
for part in response:
print(part.choices[0].delta.content or "")
Logging Observability (Docs)
LiteLLM exposes pre defined callbacks to send data to Lunary, Langfuse, DynamoDB, s3 Buckets, Helicone, Promptlayer, Traceloop, Athina, Slack
from litellm import completion
## set env variables for logging tools
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key"
os.environ["HELICONE_API_KEY"] = "your-helicone-auth-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["ATHINA_API_KEY"] = "your-athina-api-key"
os.environ["OPENAI_API_KEY"]
# set callbacks
litellm.success_callback = ["lunary", "langfuse", "athina", "helicone"] # log input/output to lunary, langfuse, supabase, athina, helicone etc
#openai call
response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi ๐ - i'm openai"}])
OpenAI Proxy - (Docs)
Track spend + Load Balance across multiple projects
The proxy provides:
๐ Proxy Endpoints - Swagger Docs
Quick Start Proxy - CLI
pip install 'litellm[proxy]'
Step 1: Start litellm proxy
$ litellm --model huggingface/bigcode/starcoder
#INFO: Proxy running on http://0.0.0.0:4000
Step 2: Make ChatCompletions Request to Proxy
import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:4000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
}
])
print(response)
Proxy Key Management (Docs)
Connect the proxy with a Postgres DB to create proxy keys
# Get the code
git clone https://github.com/BerriAI/litellm
# Go to folder
cd litellm
# Add the master key - you can change this after setup
echo 'LITELLM_MASTER_KEY="sk-1234"' > .env
# Add the litellm salt key - you cannot change this after adding a model
# It is used to encrypt / decrypt your LLM API Key credentials
# We recommned - https://1password.com/password-generator/
# password generator to get a random hash for litellm salt key
echo 'LITELLM_SALT_KEY="sk-1234"' > .env
source .env
# Start
docker-compose up
UI on /ui on your proxy server
Set budgets and rate limits across multiple projects
POST /key/generate
Request
curl 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data-raw '{"models": ["gpt-3.5-turbo", "gpt-4", "claude-2"], "duration": "20m","metadata": {"user": "ishaan@berri.ai", "team": "core-infra"}}'
Expected Response
{
"key": "sk-kdEXbIqZRwEeEiHwdg7sFA", # Bearer token
"expires": "2023-11-19T01:38:25.838000+00:00" # datetime object
}
Supported Providers (Docs)
| Provider | Completion | Streaming | Async Completion | Async Streaming | Async Embedding | Async Image Generation |
|---|---|---|---|---|---|---|
| openai | โ | โ | โ | โ | โ | โ |
| azure | โ | โ | โ | โ | โ | โ |
| aws - sagemaker | โ | โ | โ | โ | โ | |
| aws - bedrock | โ | โ | โ | โ | โ | |
| google - vertex_ai | โ | โ | โ | โ | โ | โ |
| google - palm | โ | โ | โ | โ | ||
| google AI Studio - gemini | โ | โ | โ | โ | ||
| mistral ai api | โ | โ | โ | โ | โ | |
| cloudflare AI Workers | โ | โ | โ | โ | ||
| cohere | โ | โ | โ | โ | โ | |
| anthropic | โ | โ | โ | โ | ||
| empower | โ | โ | โ | โ | ||
| huggingface | โ | โ | โ | โ | โ | |
| replicate | โ | โ | โ | โ | ||
| together_ai | โ | โ | โ | โ | ||
| openrouter | โ | โ | โ | โ | ||
| ai21 | โ | โ | โ | โ | ||
| baseten | โ | โ | โ | โ | ||
| vllm | โ | โ | โ | โ | ||
| nlp_cloud | โ | โ | โ | โ | ||
| aleph alpha | โ | โ | โ | โ | ||
| petals | โ | โ | โ | โ | ||
| ollama | โ | โ | โ | โ | โ | |
| deepinfra | โ | โ | โ | โ | ||
| perplexity-ai | โ | โ | โ | โ | ||
| Groq AI | โ | โ | โ | โ | ||
| Deepseek | โ | โ | โ | โ | ||
| anyscale | โ | โ | โ | โ | ||
| IBM - watsonx.ai | โ | โ | โ | โ | โ | |
| voyage ai | โ | |||||
| xinference [Xorbits Inference] | โ | |||||
| FriendliAI | โ | โ | โ | โ |
Contributing
To contribute: Clone the repo locally -> Make a change -> Submit a PR with the change.
Here's how to modify the repo locally: Step 1: Clone the repo
git clone https://github.com/BerriAI/litellm.git
Step 2: Navigate into the project, and install dependencies:
cd litellm
poetry install -E extra_proxy -E proxy
Step 3: Test your change:
cd litellm/tests # pwd: Documents/litellm/litellm/tests
poetry run flake8
poetry run pytest .
Step 4: Submit a PR with your changes! ๐
- push your fork to your GitHub repo
- submit a PR from there
Enterprise
For companies that need better security, user management and professional support
This covers:
- โ Features under the LiteLLM Commercial License:
- โ Feature Prioritization
- โ Custom Integrations
- โ Professional Support - Dedicated discord + slack
- โ Custom SLAs
- โ Secure access with Single Sign-On
Support / talk with founders
- Schedule Demo ๐
- Community Discord ๐ญ
- Our numbers ๐ +1 (770) 8783-106 / โญ+1 (412) 618-6238โฌ
- Our emails โ๏ธ ishaan@berri.ai / krrish@berri.ai
Why did we build this
- Need for simplicity: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI and Cohere.
Contributors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file litellm-1.41.27.tar.gz.
File metadata
- Download URL: litellm-1.41.27.tar.gz
- Upload date:
- Size: 6.8 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.8.19
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c70996f665be3c40177622afd9c54408a7430ede0e590abd8e9585f0285d6b87
|
|
| MD5 |
624ad0bb8d61596b87a7bb9fd3492fd8
|
|
| BLAKE2b-256 |
3c3c483a311cfaf12c6e5252e07808662253f0cf0336de0232cee6693090a58e
|
File details
Details for the file litellm-1.41.27-py3-none-any.whl.
File metadata
- Download URL: litellm-1.41.27-py3-none-any.whl
- Upload date:
- Size: 7.1 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.8.19
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
df97e46517129826ae958e6571b22714e79f3eba76af2f0fd812f499b21b7d2c
|
|
| MD5 |
8efa97b15e35f914f91d5d7e440075ad
|
|
| BLAKE2b-256 |
a75a7764c5754343f2e1933a2dd01e01994f983d49fcbcca5d4d4ef845cbd6ba
|
