Batch hallucination-rate evaluation and reporting over a dataset

[adaamko]

Problem

Everything in the repo today answers "is this one answer hallucinated?" or "how good is this model against gold labels?". Nothing answers the question production users actually have: "what fraction of my RAG system's answers contain hallucinations, and where do they concentrate?"

• HallucinationDetector.predict scores a single (context, question, answer) triple (lettucedetect/models/inference.py:27-46).
• The evaluation scripts require gold span annotations: scripts/evaluate.py scores against labeled HallucinationData, and scripts/evaluate_span_model.py scores against labeled HF datasets. They measure the detector, not the monitored system.

A small, self-contained CLI that runs a detector over a whole file of records and produces an aggregate report makes the library usable for monitoring, and is a good standalone contributor project — no core changes needed.

Current behavior

What exists to build on:

• HallucinationDetector.predict(context, answer, question, output_format="spans", min_confidence=...) — lettucedetect/models/inference.py:27-46; span dicts carry start/end/text/confidence (lettucedetect/detectors/transformer.py:210-228), plus category/subcategory when a taxonomy_head is configured (transformer.py:380-381).
• predict_prompt_batch exists (inference.py:62-80) but is a sequential loop for the transformer detector (transformer.py:438-441) and a 30-worker thread pool for the LLM detector (lettucedetect/detectors/llm.py:524-529) — fine to use as-is; do not optimize it here.
• scripts/span_eval_metrics.py is stdlib-only shared metric code — a precedent for keeping aggregation logic dependency-free and testable.

There is no dataset-level runner, no aggregation, and no report export.

What to do

1. Add scripts/report_hallucination_rates.py (descriptive verb-phrase name, matching the existing scripts/ naming) that reads JSONL or CSV records with context (string or list of strings), answer, optional question, and an optional grouping column (e.g. source).
2. Build the detector from CLI flags: --method (transformer / llm), --model-path, optional --taxonomy-head, --min-confidence, --lang — pass straight through to HallucinationDetector.
3. Run predict(..., output_format="spans") over all records with a progress indicator; collect per-record span lists. Support --limit N for quick runs.
4. Aggregate, in pure functions that take (record, spans) pairs and no model:
   • overall hallucination rate: fraction of answers with at least one flagged span;
   • per-group rates when a grouping column is present;
   • span-count histogram and span-confidence histogram;
   • per-category / per-subcategory counts when spans are typed;
   • top-N flagged examples (highest-confidence spans) for eyeballing.
5. Export a machine-readable JSON report and a human-readable Markdown summary (--out-json, --out-md).
6. Document the record format and one full command in the script docstring, mirroring the style of scripts/evaluate_span_model.py:1-14.

Acceptance

• python scripts/report_hallucination_rates.py --input sample.jsonl --method transformer --model-path KRLabsOrg/lettucedect-v2-mmbert-base --out-json report.json --out-md report.md runs end-to-end on a 10-row sample file.
• Counts reconcile: rows in = rows scored = denominator of every rate in the JSON report; every rate names its denominator.
• Aggregation functions are unit-tested with a stubbed detector (no model download, no network) in tests/test_report_hallucination_rates_pytest.py (the test_*_pytest.py pattern is required by tests/pytest.ini).
• python -m pytest tests/test_report_hallucination_rates_pytest.py -v passes and python tests/run_pytest.py stays green.

Non-goals

• No gold-label metrics — scripts/evaluate.py and scripts/evaluate_span_model.py already do that.
• No dashboard, HTML app, or server; JSON + Markdown files only.
• No changes to the detectors and no parallelization work.
• No new dependencies beyond what pip install -e ".[dev]" already brings (stdlib csv/json for IO).

Start here

git clone https://github.com/KRLabsOrg/LettuceDetect.git
cd LettuceDetect
pip install -e ".[dev]"
python tests/run_pytest.py   # should be green before you change anything

# The code to read first:
grep -n "def predict" lettucedetect/models/inference.py
sed -n '1,30p' scripts/evaluate_span_model.py   # CLI + docstring style to mirror
sed -n '1,25p' scripts/span_eval_metrics.py     # dependency-free aggregation precedent