Map repeated LLM span strings to distinct answer occurrences

[adaamko]

Problem

The generic LLM-judge path converts every returned substring with a fresh re.search() from the beginning of the answer. If the model returns the same text twice, both predictions receive the first occurrence's offsets:

answer = "Paris is mentioned; Paris is repeated."
LLMDetector._to_spans(["Paris", "Paris"], answer)
# [(start=0, end=5), (start=0, end=5)]

The second occurrence is never represented, and token output therefore flags only the first occurrence. Dict items can also collapse onto the same offsets while carrying different confidence or taxonomy metadata.

This is inconsistent with the native generative path: lettucedetect.prompts.generative.spans_to_offsets() already selects the first unused, non-overlapping verbatim occurrence. No model call is needed to reproduce or test the difference.

Deterministic convention

A text-only LLM response cannot identify which occurrence it intended when the same substring appears more than once. This issue does not pretend to recover that missing information. Instead, make the generic judge path follow the deterministic convention already used by the native path:

1. Returned items are interpreted in left-to-right answer order.
2. Each valid textual item reserves the first unused, non-overlapping exact occurrence.
3. Reserve the occurrence before applying is_hallucination and confidence filters, so filtering an earlier item cannot shift a later item's offsets.
4. Emit accepted items with their original confidence, reasoning, category, and subcategory metadata.
5. Skip and log an item when no unused occurrence remains.
6. A single ambiguous item keeps the existing first-occurrence fallback.

Update the generic judge prompt/response instructions to ask for spans in answer order and at most one item per distinct occurrence. Do not modify the frozen native-model prompt.

Scope

• Implement consistent non-overlapping localization in LLMDetector._to_spans().
• Reuse or extract the native locator if doing so keeps the metadata and filtering behavior clear; a large refactor is not required.
• Cover both direct span output and the end-to-end token projection with network-free tests and a fake/canned response.

Acceptance tests

• Two identical returned strings map to the first and second answer occurrences, with distinct offsets.
• Dict metadata remains attached to the corresponding occurrence.
• A rejected or low-confidence first item still reserves the first occurrence; an accepted second item keeps the second offset.
• More returned items than available occurrences never produce duplicate offsets.
• One returned item with multiple matches retains the documented first-match fallback.
• Overlapping candidates do not produce overlapping output spans.
• output_format="tokens" flags both repeated occurrences end to end.
• Existing unique-string, missing-string, confidence, taxonomy, and native-path tests remain green.

Non-goals

• Inferring the intended occurrence for one ambiguous text-only item.
• Fuzzy, normalized, or case-insensitive matching.
• Adding offsets or occurrence indices to the public LLM response schema.
• Retraining or changing the frozen native generative prompt.
• Occurrence-specific behavior in the optional verify=True second pass; verification currently identifies candidates by text and needs a separate design if that becomes important.

Start here

git clone https://github.com/KRLabsOrg/LettuceDetect.git
cd LettuceDetect
pip install -e ".[dev]"
python tests/run_pytest.py

rg -n "def _to_spans|def spans_to_offsets" lettucedetect tests