Skip to content

fix(verify): stop counting source-disclaimer sentences as citations - #191

Merged
askalf merged 1 commit into
masterfrom
fix/186-disclaimer-sentences-not-citations
Jul 28, 2026
Merged

fix(verify): stop counting source-disclaimer sentences as citations#191
askalf merged 1 commit into
masterfrom
fix/186-disclaimer-sentences-not-citations

Conversation

@askalf

@askalf askalf commented Jul 28, 2026

Copy link
Copy Markdown
Owner

fix(verify): stop counting source-disclaimer sentences as citations

Closes #186.

When the synthesizer is transparent about which sources it discarded —

"note that sources [3], [5], [7], [8], [9], [11], and [12] contain no
information about ternary LLM accuracy and are not cited below."

— extractCiteIds pulled seven ids out of that aside, each was recall-checked
against the disclaimer sentence, each correctly failed (they ARE irrelevant),
and all seven landed in unsupportedIds. The answer was penalised precisely for
disclosing what it rejected. A live canary run scored supportRatio 0.52 with 7
of its 10 "unsupported citations" coming from that single sentence.

The fix: exclude a sentence from citation accounting when BOTH

  1. isSourceDisclaimer() — it contains a source noun (sources/references/
    citations) AND an exclusion phrase (no information / not cited /
    irrelevant / does not address / …), and
  2. EVERY cited id in it also fails the recall threshold.

Requiring both is the safety property, and it is the whole design. Condition 2
means a sentence with even one supported cite is a real claim and is never
skipped, so this cannot quietly bury a partially-hallucinated citation.
Condition 1 is deliberately narrow — an exclusion CLAIM, not any negation — so
a genuine negative finding like "the trial showed no improvement [4]" is
untouched, having no source noun.

Excluded sentences are RECORDED in report.disclaimers rather than dropped, and
the CLI prints a line when any are excluded. An exclusion nobody can see is
indistinguishable from a miscount, and the point of this change is to make the
score honest, not quieter.

Residual gap, stated plainly: a fully-unsupported claim that also names
"sources" and negates them would be skipped. That phrasing is rare, and it
still surfaces in disclaimers rather than vanishing.

Tests (+6 in test/verify.test.mjs, 33 in that file, 859 suite-wide):

  • the disclaimer is excluded AND recorded
  • a disclaimer-shaped sentence with ANY supported cite still counts
  • a fully-unsupported ordinary claim is still flagged (the fix must not
    weaken the hallucinated-citation detection it sits beside)
  • isSourceDisclaimer needs both signals; "showed no improvement [4]" and
    "does not address long-context recall [4]" are NOT disclaimers
  • end-to-end: the issue's repro goes 0.13 -> 1.00

Verified the new tests FAIL with the skip disabled (2 fail) and pass with it,
so they are regression guards rather than tautologies. Report shape change is
additive; the only consumers (agent.ts, cli.ts) read .unsupported.

Closes #186.

When the synthesizer is transparent about which sources it discarded —

  "note that sources [3], [5], [7], [8], [9], [11], and [12] contain no
   information about ternary LLM accuracy and are not cited below."

— extractCiteIds pulled seven ids out of that aside, each was recall-checked
against the disclaimer sentence, each correctly failed (they ARE irrelevant),
and all seven landed in unsupportedIds. The answer was penalised precisely for
disclosing what it rejected. A live canary run scored supportRatio 0.52 with 7
of its 10 "unsupported citations" coming from that single sentence.

The fix: exclude a sentence from citation accounting when BOTH

  1. isSourceDisclaimer() — it contains a source noun (sources/references/
     citations) AND an exclusion phrase (no information / not cited /
     irrelevant / does not address / …), and
  2. EVERY cited id in it also fails the recall threshold.

Requiring both is the safety property, and it is the whole design. Condition 2
means a sentence with even one supported cite is a real claim and is never
skipped, so this cannot quietly bury a partially-hallucinated citation.
Condition 1 is deliberately narrow — an exclusion CLAIM, not any negation — so
a genuine negative finding like "the trial showed no improvement [4]" is
untouched, having no source noun.

Excluded sentences are RECORDED in report.disclaimers rather than dropped, and
the CLI prints a line when any are excluded. An exclusion nobody can see is
indistinguishable from a miscount, and the point of this change is to make the
score honest, not quieter.

Residual gap, stated plainly: a fully-unsupported claim that also names
"sources" and negates them would be skipped. That phrasing is rare, and it
still surfaces in `disclaimers` rather than vanishing.

Tests (+6 in test/verify.test.mjs, 33 in that file, 859 suite-wide):
  - the disclaimer is excluded AND recorded
  - a disclaimer-shaped sentence with ANY supported cite still counts
  - a fully-unsupported ordinary claim is still flagged (the fix must not
    weaken the hallucinated-citation detection it sits beside)
  - isSourceDisclaimer needs both signals; "showed no improvement [4]" and
    "does not address long-context recall [4]" are NOT disclaimers
  - end-to-end: the issue's repro goes 0.13 -> 1.00

Verified the new tests FAIL with the skip disabled (2 fail) and pass with it,
so they are regression guards rather than tautologies. Report shape change is
additive; the only consumers (agent.ts, cli.ts) read .unsupported.
@askalf
askalf enabled auto-merge (squash) July 28, 2026 16:11

@sprayberry-reviewer sprayberry-reviewer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an automated review from the Sprayberry Labs fleet code reviewer.

Verdict: No blocking issues found — approving.

What I checked

  • gh pr diff 191 (src/cli.ts, src/verify.ts, test/verify.test.mjs)
  • CI rollup: gh pr checks 191 — CodeQL, actionlint, analyze, build(20), build(22) all pass
  • Traced the counting-loop refactor in src/verify.ts line-by-line to confirm the new two-phase
    (score-then-tally) approach is arithmetically equivalent to the old per-id inline increment for
    every non-disclaimer sentence: totalCitations += citedIds.length; supportedCitations += citedIds.length - unsupportedIds.length matches the sum of the old per-iteration += 1 / += 1.
  • Confirmed the disclaimer-skip gate (unsupportedIds.length === citedIds.length && isSourceDisclaimer(sentence)) requires every cite in the sentence to fail recall, so a sentence
    with one genuine supported cite is never dropped — this is the safety property the PR claims, and
    test/verify.test.mjs exercises it directly ("a disclaimer-shaped sentence with ANY supported cite still counts").
  • Checked the only two consumers of the report shape (src/agent.ts:825, src/cli.ts) — both read
    .unsupported, which is unaffected in shape; disclaimers is a pure addition.

What's good

  • The isSourceDisclaimer regex pair (source-noun + exclusion-phrase, both required) is deliberately
    narrow, and the test suite specifically guards the false-positive case ("The trial showed no
    improvement over baseline [4]" / "does not address long-context recall [4]") that would otherwise
    swallow a real negative finding.
  • Excluded sentences are recorded in report.disclaimers and surfaced in the CLI output
    (src/cli.ts:615-619,633) rather than silently dropped — auditable, not just quieter.
  • New tests were verified to fail with the skip disabled per the PR description, so they're regression
    guards rather than tautologies.

Minor (non-blocking): r.disclaimers?.length in src/cli.ts:617 uses optional chaining even though
disclaimers is a non-optional field on VerificationReport — harmless defensiveness, not worth a
changes-requested.

@askalf
askalf merged commit 0617a19 into master Jul 28, 2026
5 checks passed
@askalf
askalf deleted the fix/186-disclaimer-sentences-not-citations branch July 28, 2026 16:21
@askalf askalf mentioned this pull request Jul 29, 2026
askalf added a commit that referenced this pull request Jul 29, 2026
Ships the #186 fix (merged in #191): a source-disclaimer sentence no longer
counts its bracketed ids as unsupported citations, so being transparent about
discarded sources stops costing the answer its supportRatio.

Patch, not minor — no API change. `VerificationReport` gains an additive
`disclaimers` field; the only consumers (agent.ts, cli.ts) read `.unsupported`.

Verified before opening: extract-release-notes.cjs resolves the new
`## [0.32.1]` section (empty release notes was the #135 failure mode),
package-lock version tracks package.json at 0.32.1, typecheck clean,
859 tests / 0 failures.

Merging this fires auto-release.yml → tag v0.32.1 + GitHub release + Sigstore
provenance → dispatches publish.yml → npm publish via OIDC trusted publishing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Citation verifier counts disclaimer sentences as citations, penalizing source transparency

2 participants