adversarial-contract-gate

Self-evaluation is a trap. Grade the build against a contract — with a separate evaluator.

Lecture → code · Run it · Plug in a real model

     

   

---

An agent that grades its own work ships bugs it literally can't see. This repo runs the same build two ways — self-review, and an independent adversarial evaluator graded against a negotiated contract — and shows the first one shipping a defect the second one catches.

Built from the one slide the speakers said to remember:

"self-evaluation, very much a trap. Just use an adversarial evaluator." — Anthropic, Build Agents That Run for Hours (39:00)

Full timestamped map of lecture → code: LECTURE.md.

Run it

No install. No API key. No network.

git clone https://github.com/Archive228/adversarial-contract-gate
cd adversarial-contract-gate
./run.sh

Same build. Self-review passed its own two happy-path cases and shipped. The adversarial evaluator ran the whole contract, caught three defects, and the builder repaired them (see the demo above).

The shape

spec ──► negotiate() ──► contract (testable assertions, incl. the skipped edges)
                              │
build ──┬──► self_review(build, cases-the-builder-thought-of)  → blind, ships defects
        └──► adversarial_evaluate(build, FULL contract) → failures → repair → repeat

file	what it is
src/contract.py	negotiate(spec) → a Contract of concrete assertions, edges included
src/evaluators.py	self_review (builder's own cases) vs adversarial_evaluate (full contract)
src/generator.py	offline scripted builder: v1 skips edges, v2 repairs them
src/harness.py	the two tracks; adversarial fails closed at the round budget

Plug in a real model

The generator is build(failures) -> source. A model-backed builder reads the evaluator's failure list and writes the fix — the same repair loop, live:

class ClaudeGenerator:
    def build(self, failures):
        prompt = "Implement normalize(s) in Python."
        if failures:
            prompt += "\nYour last build failed:\n" + "\n".join(failures) + "\nFix all of them."
        return strip_fences(call_model(prompt))
    self_declared = [("  Alice ", "alice"), ("BOB", "bob")]

Keep the evaluator a separate call with its own context — that separation is the point (26:47).

Tests

python3 -m unittest discover -s tests -v   # 9 tests

They test the graders and both tracks: self-review is blind to un-declared cases, the adversary catches v1 and passes v2, and it fails closed when the builder never repairs.

Honest limits

• The generator here is scripted (two fixed builds) so the demo is offline and deterministic. Swap in a model behind the same build(failures) -> source interface for real work.
• exec() runs the build in-process — fine for a scripted demo, not for untrusted model output. Sandbox it (container/VM) before pointing a real model at it.
• The adversary is only as good as the contract. A weak contract passes weak builds; the value is in negotiating assertions that cover the edges.

License

MIT — see LICENSE. All code here is original; nothing is vendored.

---

The lecture moment this repo is built from:

_{Anthropic — Build Agents That Run for Hours · 39:00}

Built from a real lecture, quoted verbatim with timestamps in LECTURE.md — not a rehash of a rehash.
If it made the idea click, ⭐ it.