fix: reject empty scorer prompt sets - #408
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds validation to ensure that both the KeywordRate and KLDivergence scorers are initialized with at least one prompt, raising a ValueError otherwise. It also introduces corresponding unit tests to verify this behavior. I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
This also looks like a reasonable change to me, though I'm not sure if the unit tests are strictly necessary. |
|
They're cheap now that CI runs the tests directory, but I don't feel strongly — happy to drop them and keep just the validation if you'd rather keep the suite lean. |
yes I don't think that test is necessary, it did not changed anything affecting reproducibility. And tests are already passing for what cases we have as of now. |
Summary
Empty scorer prompt sets now fail during scorer initialization with a clear error instead of producing a later scoring crash.
KeywordRatepreviously accepted0prompts and could later divide by zero.KLDivergencecould proceed to baseline-logit collection with no prompts. Both built-in scorers now reject empty prompt sets immediately after loading them.Validation
PYTHONPATH=src uv run python -m unittest discover -s tests -p test_scorers.pyuv run ruff format --check src/heretic/scorers/keyword_rate.py src/heretic/scorers/kl_divergence.py tests/test_scorers.pyuv run ruff check --extend-select I src/heretic/scorers/keyword_rate.py src/heretic/scorers/kl_divergence.py tests/test_scorers.pyuv run ty check src/heretic/scorers/keyword_rate.py src/heretic/scorers/kl_divergence.py tests/test_scorers.pygit diff --check