fix(verify): stop counting source-disclaimer sentences as citations - #191
Merged
Merged
Conversation
Closes #186. When the synthesizer is transparent about which sources it discarded — "note that sources [3], [5], [7], [8], [9], [11], and [12] contain no information about ternary LLM accuracy and are not cited below." — extractCiteIds pulled seven ids out of that aside, each was recall-checked against the disclaimer sentence, each correctly failed (they ARE irrelevant), and all seven landed in unsupportedIds. The answer was penalised precisely for disclosing what it rejected. A live canary run scored supportRatio 0.52 with 7 of its 10 "unsupported citations" coming from that single sentence. The fix: exclude a sentence from citation accounting when BOTH 1. isSourceDisclaimer() — it contains a source noun (sources/references/ citations) AND an exclusion phrase (no information / not cited / irrelevant / does not address / …), and 2. EVERY cited id in it also fails the recall threshold. Requiring both is the safety property, and it is the whole design. Condition 2 means a sentence with even one supported cite is a real claim and is never skipped, so this cannot quietly bury a partially-hallucinated citation. Condition 1 is deliberately narrow — an exclusion CLAIM, not any negation — so a genuine negative finding like "the trial showed no improvement [4]" is untouched, having no source noun. Excluded sentences are RECORDED in report.disclaimers rather than dropped, and the CLI prints a line when any are excluded. An exclusion nobody can see is indistinguishable from a miscount, and the point of this change is to make the score honest, not quieter. Residual gap, stated plainly: a fully-unsupported claim that also names "sources" and negates them would be skipped. That phrasing is rare, and it still surfaces in `disclaimers` rather than vanishing. Tests (+6 in test/verify.test.mjs, 33 in that file, 859 suite-wide): - the disclaimer is excluded AND recorded - a disclaimer-shaped sentence with ANY supported cite still counts - a fully-unsupported ordinary claim is still flagged (the fix must not weaken the hallucinated-citation detection it sits beside) - isSourceDisclaimer needs both signals; "showed no improvement [4]" and "does not address long-context recall [4]" are NOT disclaimers - end-to-end: the issue's repro goes 0.13 -> 1.00 Verified the new tests FAIL with the skip disabled (2 fail) and pass with it, so they are regression guards rather than tautologies. Report shape change is additive; the only consumers (agent.ts, cli.ts) read .unsupported.
askalf
enabled auto-merge (squash)
July 28, 2026 16:11
sprayberry-reviewer
approved these changes
Jul 28, 2026
sprayberry-reviewer
left a comment
Collaborator
There was a problem hiding this comment.
This is an automated review from the Sprayberry Labs fleet code reviewer.
Verdict: No blocking issues found — approving.
What I checked
gh pr diff 191(src/cli.ts, src/verify.ts, test/verify.test.mjs)- CI rollup:
gh pr checks 191— CodeQL, actionlint, analyze, build(20), build(22) all pass - Traced the counting-loop refactor in
src/verify.tsline-by-line to confirm the new two-phase
(score-then-tally) approach is arithmetically equivalent to the old per-id inline increment for
every non-disclaimer sentence:totalCitations += citedIds.length; supportedCitations += citedIds.length - unsupportedIds.lengthmatches the sum of the old per-iteration+= 1/+= 1. - Confirmed the disclaimer-skip gate (
unsupportedIds.length === citedIds.length && isSourceDisclaimer(sentence)) requires every cite in the sentence to fail recall, so a sentence
with one genuine supported cite is never dropped — this is the safety property the PR claims, and
test/verify.test.mjsexercises it directly ("a disclaimer-shaped sentence with ANY supported cite still counts"). - Checked the only two consumers of the report shape (
src/agent.ts:825,src/cli.ts) — both read
.unsupported, which is unaffected in shape;disclaimersis a pure addition.
What's good
- The
isSourceDisclaimerregex pair (source-noun + exclusion-phrase, both required) is deliberately
narrow, and the test suite specifically guards the false-positive case ("The trial showed no
improvement over baseline [4]" / "does not address long-context recall [4]") that would otherwise
swallow a real negative finding. - Excluded sentences are recorded in
report.disclaimersand surfaced in the CLI output
(src/cli.ts:615-619,633) rather than silently dropped — auditable, not just quieter. - New tests were verified to fail with the skip disabled per the PR description, so they're regression
guards rather than tautologies.
Minor (non-blocking): r.disclaimers?.length in src/cli.ts:617 uses optional chaining even though
disclaimers is a non-optional field on VerificationReport — harmless defensiveness, not worth a
changes-requested.
Merged
askalf
added a commit
that referenced
this pull request
Jul 29, 2026
Ships the #186 fix (merged in #191): a source-disclaimer sentence no longer counts its bracketed ids as unsupported citations, so being transparent about discarded sources stops costing the answer its supportRatio. Patch, not minor — no API change. `VerificationReport` gains an additive `disclaimers` field; the only consumers (agent.ts, cli.ts) read `.unsupported`. Verified before opening: extract-release-notes.cjs resolves the new `## [0.32.1]` section (empty release notes was the #135 failure mode), package-lock version tracks package.json at 0.32.1, typecheck clean, 859 tests / 0 failures. Merging this fires auto-release.yml → tag v0.32.1 + GitHub release + Sigstore provenance → dispatches publish.yml → npm publish via OIDC trusted publishing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(verify): stop counting source-disclaimer sentences as citations
Closes #186.
When the synthesizer is transparent about which sources it discarded —
"note that sources [3], [5], [7], [8], [9], [11], and [12] contain no
information about ternary LLM accuracy and are not cited below."
— extractCiteIds pulled seven ids out of that aside, each was recall-checked
against the disclaimer sentence, each correctly failed (they ARE irrelevant),
and all seven landed in unsupportedIds. The answer was penalised precisely for
disclosing what it rejected. A live canary run scored supportRatio 0.52 with 7
of its 10 "unsupported citations" coming from that single sentence.
The fix: exclude a sentence from citation accounting when BOTH
citations) AND an exclusion phrase (no information / not cited /
irrelevant / does not address / …), and
Requiring both is the safety property, and it is the whole design. Condition 2
means a sentence with even one supported cite is a real claim and is never
skipped, so this cannot quietly bury a partially-hallucinated citation.
Condition 1 is deliberately narrow — an exclusion CLAIM, not any negation — so
a genuine negative finding like "the trial showed no improvement [4]" is
untouched, having no source noun.
Excluded sentences are RECORDED in report.disclaimers rather than dropped, and
the CLI prints a line when any are excluded. An exclusion nobody can see is
indistinguishable from a miscount, and the point of this change is to make the
score honest, not quieter.
Residual gap, stated plainly: a fully-unsupported claim that also names
"sources" and negates them would be skipped. That phrasing is rare, and it
still surfaces in
disclaimersrather than vanishing.Tests (+6 in test/verify.test.mjs, 33 in that file, 859 suite-wide):
weaken the hallucinated-citation detection it sits beside)
"does not address long-context recall [4]" are NOT disclaimers
Verified the new tests FAIL with the skip disabled (2 fail) and pass with it,
so they are regression guards rather than tautologies. Report shape change is
additive; the only consumers (agent.ts, cli.ts) read .unsupported.