LLM Diff

Prompt regression testing for RAG pipelines. Like git diff, but for AI.

Why LLM Diff

• You changed a prompt. Did it get better? Find out in 2 minutes.
• Works with any LLM — OpenAI, Anthropic, Google Gemini, and more.
• Local-first. No accounts, no cloud, no data leaves your machine.
• One env var. Set your API key and you're done.

Quickstart

# 1. Install
pip install llmregress

# 2. Set your API key (pick any provider you already have)
export ANTHROPIC_API_KEY=your_key_here
# or: export OPENAI_API_KEY=your_key_here
# or: export GOOGLE_API_KEY=your_key_here

# 3. Copy an example test file
cp examples/rag_pipeline.yaml my_tests.yaml

# 4. Compare your prompts
llmregress compare my_tests.yaml

# 5. Open the web dashboard
llmregress serve
# → http://localhost:7331

I have a LangChain RAG app — how do I use this?

If your app looks like:

result = chain.invoke({"question": q, "context": c})

Translate it into a YAML test case:

model: anthropic/claude-3-5-haiku-20241022
judge_model: openai/gpt-4o-mini
test_cases:
  - id: my_test
    input: "What is the default chunk size?"
    context: "LangChain's default chunk_size is 1000 characters..."
    reference_answer: >
      LangChain's RecursiveCharacterTextSplitter defaults to a chunk_size of 1000 characters
      and a chunk_overlap of 200 characters.
    criteria:
      - "Answer is factually correct"
      - "Response is concise (under 50 words)"
    prompt_v1: |
      You are a helpful assistant. Context: {context}
      Question: {input}
    prompt_v2: |
      Answer only from context. Be concise.
      Context: {context}
      Question: {input}

Then run: llmregress compare my_tests.yaml

Providers & model strings

Change 1–2 lines in your YAML — no code changes. You can use any model from each provider family:

The model string format is always provider/model-name. Any model supported by LiteLLM works — just set the matching API key.

Reduce judge bias: use a different model family for judge_model than model. Example: Anthropic runner + OpenAI judge = cross-family, lowest self-preference bias.

CLI reference

Web dashboard

llmregress serve

Opens at http://localhost:7331. Features:

• Side-by-side output comparison per test case
• Live streaming — results appear as they complete
• Run history with score trend chart
• Click any past run while a new test is running — views are independent

Environment variables

Docker

docker-compose up
# Dashboard at http://localhost:7331

Mount your YAML test files into the container:

# docker-compose.yml — add a volume:
volumes:
  - ./my_tests:/workspace/tests

Contributing

1. Fork the repo
2. Create a branch: git checkout -b feat/my-feature
3. Run tests: pytest tests/ -v
4. Open a PR