Skip to content

Getting started

Installation

pip install llm-search-eval

or straight from a release tag:

pip install "git+https://github.com/sadrmanesh/llm-search-eval@v0.4.0"

Optional extras let the connectors reuse official clients: llm-search-eval[elasticsearch], [opensearch], and [algolia]. They are not needed for the default HTTP mode.

Or use the Docker image, sadrmanesh/llm-search-eval on Docker Hub or ghcr.io/sadrmanesh/llm-search-eval; see Docker.

To try the library without installing anything, open the browser demo on Hugging Face. It runs the package with Pyodide on the example data or on your own files. The example data is also published as the llm-search-eval-electronics dataset.

From source, for development:

git clone https://github.com/sadrmanesh/llm-search-eval
cd llm-search-eval
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Quick start

The repository ships a small bilingual electronics catalog with three systems to compare. Run it offline with the mock judge:

llm-search-eval run \
  --queries  examples/electronics/queries.jsonl \
  --products examples/electronics/products.jsonl \
  --qrels    examples/electronics/qrels.csv \
  --system   bm25=examples/electronics/runs/bm25.jsonl \
  --system   semantic=examples/electronics/runs/semantic.csv \
  --system   hybrid=examples/electronics/runs/hybrid.trec \
  --judge mock \
  --output-dir reports/

| System | ndcg@5 | ndcg@10 | precision@5 | recall@10 | mrr | map |
| :--- | ---: | ---: | ---: | ---: | ---: | ---: |
| `bm25` | 0.5305 | 0.5305 | 0.2200 | 0.4833 | 0.8000 | 0.4306 |
| `semantic` | **0.9865** | **0.9865** | **0.4200** | **0.9667** | **1.0000** | **0.9667** |
| `hybrid` | 0.9767 | 0.9767 | **0.4200** | **0.9667** | **1.0000** | 0.9583 |

The output directory holds report.md, report.json, the report.html dashboard, cost.json, and judgments.jsonl. See the sample report and the sample dashboard.

Switch to a real LLM judge, with a price and a budget:

export OPENAI_API_KEY=...
llm-search-eval run ... --judge openai=gpt-4o-mini --cache .judge-cache.jsonl \
  --price gpt-4o-mini=0.15,0.60 --max-cost 2

Check inputs before spending tokens:

llm-search-eval validate --queries ... --products ... --qrels ... --system ...

Run llm-search-eval run --help for every option.