Skip to content

fix: honor contract reliability weights in session summaries#3

Open
CTWalk wants to merge 1 commit into
qualixar:mainfrom
CTWalk:agent/honor-session-reliability-weights
Open

fix: honor contract reliability weights in session summaries#3
CTWalk wants to merge 1 commit into
qualixar:mainfrom
CTWalk:agent/honor-session-reliability-weights

Conversation

@CTWalk

@CTWalk CTWalk commented Jul 22, 2026

Copy link
Copy Markdown

Summary

  • Pass the loaded contract's reliability.weights into the session-level Theta calculation.
  • Preserve the existing default weights when a contract has no reliability section.
  • Add a regression test at the SessionMonitor boundary.

Why

The metrics documentation says that each Theta component is weighted according to the contract's reliability.weights configuration. ContractSpec parses and validates those weights, and compute_theta already accepts them, but SessionMonitor._session_summary_internal() currently omits the argument. compute_theta therefore falls back to ReliabilityWeights() even when the contract declares different values.

This also affects the shipped healthcare-triage.yaml example, which declares non-default reliability weights (0.40 / 0.25 / 0.20 / 0.15).

The change mirrors the existing optional-config handoff used for drift configuration in the same class. It does not alter the Theta formula, default weights, YAML shape, deployment threshold, or public result types.

Reproduction

With compliance weight 1.0, all other weights 0.0, and one of two hard constraints failing, the combined compliance input is 0.75.

Before this change:

session_summary().theta                       = 0.7375
compute_theta(..., weights=contract.weights) = 0.75

After this change, both paths return 0.75. Contracts without a reliability section remain unchanged.

Verification

  • uv run pytest — 445 passed
  • uv run ruff check src/ tests/ — passed
  • uv run basedpyright src/ — 0 errors (22 existing warnings)
  • git diff --check — passed

Thank you for maintaining AgentAssert. I kept this change focused on carrying the already-supported configuration through the session summary path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants