Library to easily interface with LLM API providers
Project description
๐ LiteLLM
Call all LLM APIs using the OpenAI format [Bedrock, Huggingface, Cohere, TogetherAI, Azure, OpenAI, etc.]
Evaluate LLMs โ OpenAI-Compatible Server
LiteLLM manages
- Translating inputs to the provider's completion and embedding endpoints
- Guarantees consistent output, text responses will always be available at
['choices'][0]['message']['content'] - Exception mapping - common exceptions across providers are mapped to the OpenAI exception types.
10/05/2023: LiteLLM is adopting Semantic Versioning for all commits. Learn more
10/16/2023: Self-hosted OpenAI-proxy server Learn more
Usage (Docs)
pip install litellm
from litellm import completion
import os
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["COHERE_API_KEY"] = "your-cohere-key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion(model="command-nightly", messages=messages)
print(response)
Streaming (Docs)
liteLLM supports streaming the model response back, pass stream=True to get a streaming iterator in response.
Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
from litellm import completion
response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
for chunk in response:
print(chunk['choices'][0]['delta'])
# claude 2
result = completion('claude-2', messages, stream=True)
for chunk in result:
print(chunk['choices'][0]['delta'])
Reliability - Fallback LLMs
Never fail a request using LiteLLM
from litellm import completion
# if gpt-4 fails, retry the request with gpt-3.5-turbo->command-nightly->claude-instant-1
response = completion(model="gpt-4",messages=messages, fallbacks=["gpt-3.5-turbo", "command-nightly", "claude-instant-1"])
# if azure/gpt-4 fails, retry the request with fallback api_keys/api_base
response = completion(model="azure/gpt-4", messages=messages, api_key=api_key, fallbacks=[{"api_key": "good-key-1"}, {"api_key": "good-key-2", "api_base": "good-api-base-2"}])
Logging Observability - Log LLM Input/Output (Docs)
LiteLLM exposes pre defined callbacks to send data to LLMonitor, Langfuse, Helicone, Promptlayer, Traceloop, Slack
from litellm import completion
## set env variables for logging tools
os.environ["PROMPTLAYER_API_KEY"] = "your-promptlayer-key"
os.environ["LLMONITOR_APP_ID"] = "your-llmonitor-app-id"
os.environ["OPENAI_API_KEY"]
# set callbacks
litellm.success_callback = ["promptlayer", "llmonitor"] # log input/output to promptlayer, llmonitor, supabase
#openai call
response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi ๐ - i'm openai"}])
Supported Provider (Docs)
| Provider | Completion | Streaming | Async Completion | Async Streaming |
|---|---|---|---|---|
| openai | โ | โ | โ | โ |
| azure | โ | โ | โ | โ |
| aws - sagemaker | โ | โ | โ | โ |
| aws - bedrock | โ | โ | โ | โ |
| cohere | โ | โ | โ | โ |
| anthropic | โ | โ | โ | โ |
| huggingface | โ | โ | โ | โ |
| replicate | โ | โ | โ | โ |
| together_ai | โ | โ | โ | โ |
| openrouter | โ | โ | โ | โ |
| google - vertex_ai | โ | โ | โ | โ |
| google - palm | โ | โ | โ | โ |
| ai21 | โ | โ | โ | โ |
| baseten | โ | โ | โ | โ |
| vllm | โ | โ | โ | โ |
| nlp_cloud | โ | โ | โ | โ |
| aleph alpha | โ | โ | โ | โ |
| petals | โ | โ | โ | โ |
| ollama | โ | โ | โ | โ |
| deepinfra | โ | โ | โ | โ |
| perplexity-ai | โ | โ | โ | โ |
| anyscale | โ | โ | โ | โ |
Contributing
To contribute: Clone the repo locally -> Make a change -> Submit a PR with the change.
Here's how to modify the repo locally: Step 1: Clone the repo
git clone https://github.com/BerriAI/litellm.git
Step 2: Navigate into the project, and install dependencies:
cd litellm
poetry install
Step 3: Test your change:
cd litellm/tests # pwd: Documents/litellm/litellm/tests
pytest .
Step 4: Submit a PR with your changes! ๐
- push your fork to your GitHub repo
- submit a PR from there
Support / talk with founders
- Schedule Demo ๐
- Community Discord ๐ญ
- Our numbers ๐ +1 (770) 8783-106 / โญ+1 (412) 618-6238โฌ
- Our emails โ๏ธ ishaan@berri.ai / krrish@berri.ai
Why did we build this
- Need for simplicity: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI and Cohere.
Contributors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file litellm-0.13.1.dev3.tar.gz.
File metadata
- Download URL: litellm-0.13.1.dev3.tar.gz
- Upload date:
- Size: 1.3 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.5.1 CPython/3.11.4 Darwin/22.5.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
cb408360d8f20fd31852a84e2bd5adc880128789e24568a8240e465bc7ba42ad
|
|
| MD5 |
77f18c9d8d7f25875a305ab4faf18c78
|
|
| BLAKE2b-256 |
f0ad6bc4321784208643f5d15b4c28bb2e8a605934c6cf6399be2af735453b46
|
File details
Details for the file litellm-0.13.1.dev3-py3-none-any.whl.
File metadata
- Download URL: litellm-0.13.1.dev3-py3-none-any.whl
- Upload date:
- Size: 1.4 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.5.1 CPython/3.11.4 Darwin/22.5.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
886574b515c35d92b05a8dce8b33e5728452e3b75b8e1f44ad8cdfd5fd551bca
|
|
| MD5 |
8ded2646f106a658596b51c30c45f910
|
|
| BLAKE2b-256 |
a775ca635caa8708c1ab1363799e9f5149790dbebf0c6585ea2d01a63aeb1e7a
|
