Skip to content

orb: run the backtest replay inside the CoCo runtime class and persist its attestation envelope #8537

Description

@JSONbored

Problem

With the confidential runtime class live (blocked-by) and the attestation-evidence envelope schema shipped (blocked-by), the actual capability lands here: execute the existing backtest replay inside the attested environment and persist the evidence with the run, so a REGRESSED/OK verdict over a private corpus carries proof of what executed. Today persistThresholdBacktestRuns (src/services/threshold-backtest-run.ts) writes runs into audit_events (THRESHOLD_BACKTEST_EVENT_TYPE = "calibration.threshold_backtest_run") with metadata.comparison + corpus checksum + SHAs; nothing carries execution-environment evidence.

Requirements

  • A job image that runs the threshold comparison (the pure core behind runThresholdBacktestAdvisory) and the logic replay (scripts/backtest-logic-check.ts's core) over an exported, checksummed corpus snapshot — no live DB access from inside the confidential VM; corpus in, verdict + envelope out.
  • The job computes reportData via buildAttestationReportData({ corpusChecksum, headSha, baseSha }) (envelope module) and binds it into the SNP attestation report's user-data field, so the attestation cryptographically commits to exactly which corpus + code revision ran.
  • The emitted AttestationEnvelope validates against validateAttestationEnvelope and is persisted under the run's audit_events metadata (additive field; existing readers — scripts/backtest-track-record.ts, src/review/public-rule-precision.ts, src/services/rule-calibration-trend.ts — must be byte-identical in behavior when the field is absent).
  • Verification happens outside the job (KBS/verifier from the runtime-class issue); the persisted envelope's verification status reflects the verifier's result, "unverified" when verification was skipped.
  • Advisory discipline unchanged: attested or not, verdicts remain advisory/config-gated exactly per Decide + implement whether a REGRESSED backtest verdict should ever gate a merge (Phase 2) #8105 — this issue adds evidence, never authority.

Deliverables

  • Job image + manifest running in the confidential RuntimeClass end to end on a real corpus snapshot
  • Envelope persisted with a real run; absent-field compatibility covered by tests
  • Redacted end-to-end evidence (attest → verify → persisted envelope) recorded here

Links & Resources

  • Parent epic; the CoCo runtime-class + envelope sub-issues (both blocked-by)
  • src/services/threshold-backtest-run.ts, scripts/backtest-corpus-export.ts, scripts/backtest-logic-check.ts

Boundaries

No gating changes, no new authority for attested runs, no tenant data in any public artifact. The interim Cloudflare path keeps running the unattested advisory check unchanged.

maintainer-only — infra + persistence-shape authority.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions