Skip to content

test: add canonical operation-aware conformance - #62

Merged
basauth merged 1 commit into
mainfrom
test/operation-aware-canonical-conformance
Jul 24, 2026
Merged

test: add canonical operation-aware conformance#62
basauth merged 1 commit into
mainfrom
test/operation-aware-canonical-conformance

Conversation

@basauth

@basauth basauth commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Add canonical end-to-end conformance coverage for the operation-aware authorization path in basis-core.

This PR executes all five canonical basis-schemas v0.2.2 compatibility scenarios through the real OperationAwareEnforcementPoint and verifies exact agreement across the kernel-owned result artifacts:

  • OperationAwareDecisionResponse
  • EvaluationTrace
  • AuditEvidence

The work completes the canonical-conformance milestone after the required evidence-provenance reconciliation was completed and the corrected v0.2.2 compatibility snapshot was vendored.

Canonical scenarios

The test suite covers:

  • allow-basic
  • deny-precedence
  • default-deny
  • not-applicable
  • invalid-policy-bundle

Each scenario is loaded from the active immutable basis-schemas v0.2.2 snapshot and evaluated using the real typed request, policy, engine, enforcement, trace, response, and audit paths.

Coverage

Allow basic

Verifies:

  • final outcome is allow;
  • evaluation status matches the canonical artifact;
  • top-level explanation remains null;
  • bundle identity is preserved;
  • matched-rule evidence preserves the authored reason code and explanation;
  • response, trace, and audit evidence exactly match the canonical fixtures.

Deny precedence

Verifies:

  • final outcome is deny;
  • both matched ALLOW and DENY rules remain represented;
  • the matched but non-decisive ALLOW rule retains its authored evidence;
  • deny precedence determines the aggregate outcome without erasing matched evidence;
  • matched-rule ordering is deterministic;
  • response, trace, and audit evidence exactly match the canonical fixtures.

Default deny

Verifies:

  • final outcome is deny;
  • the canonical default-deny reason is returned;
  • the candidate rule is represented as not_matched;
  • authored rule rationale is not projected onto the non-matching rule;
  • no matched-rule IDs are introduced;
  • response, trace, and audit evidence exactly match the canonical fixtures.

Not applicable

Verifies:

  • outcome is NOT_APPLICABLE;
  • evaluation status is completed;
  • top-level explanation remains null;
  • no matched rules are introduced;
  • trustworthy typed bundle identity remains present;
  • response, trace, and audit evidence exactly match the canonical fixtures.

Invalid policy bundle

Verifies:

  • evaluation status is failed;
  • outcome remains null;
  • failure reason is policy_validation_failure;
  • no aggregate authorization outcome is invented;
  • no matched-rule IDs are introduced;
  • trustworthy typed bundle identity remains present;
  • response, trace, and audit evidence exactly match the canonical fixtures.

Execution-path integrity

The tests use the real:

OperationAwareEnforcementPoint
→ OperationAwareEvaluationEngine
→ response assembly
→ evaluation-trace assembly
→ AuditEvidence assembly

The conformance test does not:

  • bypass the enforcement point;
  • recreate production evaluation logic;
  • use a test-double evaluator;
  • use model_construct();
  • modify or normalize canonical fixtures;
  • require basis-core to generate gateway-owned artifacts.

GatewayAuditEvent remains outside the kernel’s direct conformance boundary.

Evidence-provenance semantics

The canonical tests confirm the reconciled behavior now implemented across the ecosystem:

  • top-level explanation remains null when no governed stage supplies one;
  • matched rules retain authored rationale;
  • non-matching and skipped rules omit authored rationale;
  • matched non-decisive evidence remains present under deny precedence;
  • bundle identity is preserved whenever a trustworthy typed PolicyBundle exists, including NOT_APPLICABLE and typed semantic validation failures.

Changes

  • Added tests/operation_aware/test_canonical_vectors.py.
  • Updated the operation-aware implementation roadmap to record the milestone as implementation-complete and pending PR merge.
  • Updated the restored snapshot-release self-guard from v0.2.1 to the active v0.2.2 compatibility release.

No production code, fixtures, snapshot helpers, dependencies, or package metadata were changed.

Validation

  • Canonical conformance tests: 16 passed
  • Related operation-aware tests: 300 passed
  • Full operation-aware suite: 3098 passed, 86 skipped
  • Full repository suite: 3930 passed, 86 skipped
  • ruff check .: passed
  • ruff format --check src tests: passed
  • mypy src: passed
  • git diff --check: passed

Follow-Up

After this PR merges, the canonical operation-aware conformance milestone is complete.

The next sequenced roadmap item is PR 38.

@basauth
basauth merged commit b159ae8 into main Jul 24, 2026
3 checks passed
@basauth
basauth deleted the test/operation-aware-canonical-conformance branch July 24, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants