Skip to content

test: add operation-aware artifact agreement coverage - #55

Merged
basauth merged 1 commit into
mainfrom
test/operation-aware-artifact-agreement
Jul 20, 2026
Merged

test: add operation-aware artifact agreement coverage#55
basauth merged 1 commit into
mainfrom
test/operation-aware-artifact-agreement

Conversation

@basauth

@basauth basauth commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Add comprehensive agreement tests for the operation-aware kernel artifacts produced by the merged response and audit assembly functions:

  • EvaluationTrace
  • OperationAwareDecisionResponse
  • AuditEvidence

This PR proves that one evaluation produces one coherent story across all three artifacts.

It adds no production behavior.

Scope

The PR adds:

tests/operation_aware/test_artifact_agreement.py

and narrowly updates the PR 32 roadmap entry.

No file under src/ changes.

Agreement Matrix

The tests verify agreement across every shared field:

  • request ID
  • correlation ID
  • evaluation status
  • authorization outcome
  • failure reason
  • bundle ID
  • bundle version
  • trace ID
  • reason code
  • explanation

Agreement is checked across:

response ↔ trace
AuditEvidence ↔ trace
response ↔ AuditEvidence

Optional-field presence is part of agreement. A value on one artifact and None on another is detected as a mismatch.

Independent Vocabulary Mapping

The tests define independent, test-owned mappings from trace enums to the decisions-owned response and audit enums:

EvaluationStatus
    → OperationAwareEvaluationStatus

TraceOutcome
    → OperationAwareDecisionOutcome

TraceFailureReason
    → OperationAwareFailureReason

The mappings are not imported from the production assembler.

Exhaustiveness tests prove complete source and target coverage.

Response Forms

Both supported response forms are covered.

Reference-only response:

response.trace_id == trace.trace_id
response.evaluation_trace is None

Embedded response:

response.trace_id == trace.trace_id
response.evaluation_trace == trace

All top-level agreement fields remain consistent in either form.

Matched-Rule Agreement

AuditEvidence.matched_rule_ids is independently projected from ordered trace rule evidence.

Only MATCHED results are included.

The tests prove:

  • one matched rule for allow;
  • multiple matched rules for deny precedence;
  • trace ordering is preserved;
  • default deny produces an empty list;
  • not applicable produces an empty list;
  • failed evaluation produces an empty list.

A reversed matched-rule order is detected as disagreement.

Evidence Provenance

The tests verify that request-owned evidence references are preserved in AuditEvidence:

  • identity evidence reference;
  • adapter evidence reference.

Coverage includes:

  • both present;
  • both absent;
  • identity only;
  • adapter only.

Response and trace models remain free of these request-only fields.

Evaluation States

Complete agreement is proven for:

  • completed allow;
  • completed explicit deny;
  • completed default deny;
  • completed not applicable;
  • failed policy validation.

The failed semantic bundle-validation case correctly remains:

evaluation_status = failed
outcome = null
failure_reason = policy_validation_failure

It is never converted to deny and never uses the superseded invalid_policy_bundle classification for the duplicate-rule scenario.

All six governed failure reasons are covered.

Negative Mutation Matrix

The suite includes exactly 24 test-only disagreement mutations covering:

  • request identity;
  • correlation identity;
  • evaluation status;
  • outcome;
  • failure reason;
  • bundle ID;
  • bundle version;
  • trace ID;
  • reason code;
  • explanation;
  • embedded trace;
  • matched-rule content;
  • matched-rule ordering;
  • optional-field presence.

Every mutation is detected.

Assertions identify the mismatched field without requiring a production agreement validator.

Serialization

Agreement is preserved through:

  • model_dump(mode="json");
  • model_dump(mode="json", exclude_none=True);
  • model_dump_json();
  • nested embedded-trace serialization.

Contract-required nullable fields remain present.

Determinism and Immutability

Tests prove:

  • repeated assembly produces equal artifacts;
  • agreement results are deterministic;
  • requests are not mutated;
  • traces are not mutated;
  • responses are not mutated;
  • audit evidence is not mutated;
  • nested rule evidence and evidence references remain unchanged.

Boundaries

The new test module does not:

  • load YAML fixtures;
  • import snapshot helpers;
  • invoke OperationAwareEvaluationEngine;
  • invoke policy evaluation;
  • import enforcement or adapters;
  • construct gateway audit events;
  • add a production agreement function;
  • modify public exports.

Complete canonical fixture equality remains PR 37.

Gateway-owned enforcement facts remain outside kernel artifact agreement.

Validation

  • focused agreement suite: 109 passed
  • related response, audit, trace, and assembly regressions: 636 passed
  • operation-aware suite: 2,998 passed, 86 skipped
  • v0.1 regression suite: 263 passed
  • full repository suite: 3,715 passed, 86 skipped
  • Ruff lint: passed
  • Ruff formatting: passed
  • mypy strict validation: passed
  • git diff --check: passed

@basauth
basauth merged commit 69f15c0 into main Jul 20, 2026
3 checks passed
@basauth
basauth deleted the test/operation-aware-artifact-agreement branch July 20, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants