🔁  loop-graph-harness

Build a real agent pipeline in three layers — a loop, a graph of loops, and the harness they run in.

Field Guide  ·  Run it  ·  The three layers  ·  Evidence

The niche keeps splitting agent building into three separate "engineering" disciplines — loop engineering, graph engineering, harness engineering — and selling a course for each. They're three layers of one stack. Clone this, run it, read ~600 lines, and you have the whole thing, with a timestamped lecture line and an honest verdict behind every design choice.

⚡ Run it (no install, no key, no network)

git clone https://github.com/Archive228/loop-graph-harness
cd loop-graph-harness
./run.sh

The demo fans three large "sheets" through the pipeline and prints an honest proof at every layer:

FAN-OUT DECISION: FANOUT - parallel + (overflow or high-value) - worth the 3-10x token cost
  (total material to process: 25,997 bytes across 3 sheets)

LOOP results (one worker per sheet):
  PASS  sheet-A: builds      total=9210000s  attempts=2  <- attempt 1 undercounted a multi-unit entry, loop retried
  PASS  sheet-B: tests       total=6315000s  attempts=2  <- attempt 1 undercounted a multi-unit entry, loop retried
  PASS  sheet-C: deploys     total=7500000s  attempts=1

HARNESS event log (each child holds KILObytes; the parent grows by BYTES):
  spawn #1: child_ctx=8544B  parent +73B (sum sheet: sheet-A: builds)
  spawn #2: child_ctx=10043B  parent +72B (sum sheet: sheet-B: tests)
  spawn #3: child_ctx=7545B  parent +74B (sum sheet: sheet-C: deploys)
  spawn #4: child_ctx=106B  parent +6B (adversarially verify the merged result)

GRAPH merge: {'grand_total_seconds': 23025000, 'sheets': 3, 'expected_sheets': 3}
INDEPENDENT verify (fresh context): ACCEPTED
HITL checkpoint: APPROVED (auto - plug a human in prod)

GRADER CALIBRATION: 100% agreement with human labels | trustworthy=True

TRACE (11 events): {'loop.attempt': 5, 'spawn': 4, 'verify': 1, 'hitl': 1}
cost: 26238B across 4 spawns (budget 1000000B)

ISOLATION: 25,997 bytes of sheets processed, but the main agent's context is 225 bytes
  -> 115x smaller than the raw material. Only distilled results crossed the boundary.

The three layers

HARNESS  (src/harness.py)   ── clean-context spawn · one general tool · sandbox boundary · cost meter · tracer
  └── GRAPH  (src/graph.py) ── fan-out N workers · collect · independent verify · HITL checkpoint
        └── LOOP (src/loop.py) ── gather → act → verify → repeat
              worker (src/workers.py) ── the honest, jagged demo task

  budget.py  ── the fan-out DECISION rule (a graph is a cost, not a default) + a cost ceiling
  trace.py   ── structured observability of every attempt / spawn / verify / checkpoint
  evals.py   ── grader calibration: audit the verifier against human labels

🔁 Layer 1 — the loop (src/loop.py)

gather → act → verify → repeat, until a program says it's done.

"the three parts to an agent loop: first gather context, second taking action, and the third is verifying the work." — masterclass 22:20

The verifier is written before the action — a golden total, known before any parser exists. Attempt 1 uses a parser with a real, common bug (reads only the first unit of each entry), so it's exactly right on single-unit sheets and silently returns a wrong value on multi-unit ones — no exception, just a plausible wrong number that review would miss. The verifier catches the mismatch and the loop retries. That's the "jagged edge" (Karpathy 10:35) as a reproducible bug.

🕸️ Layer 2 — the graph (src/graph.py)

Several loop-workers, fanned out and back, plus an independent checker — but a graph is a cost, not a default (budget.advise_fanout() decides).

"read and summarize sheet one, sheet two, sheet three... then they return their results, and the agent maybe spins off more sub agents again." — 79:12

A graph node is a loop; the graph is only the wiring. The verifier runs in a fresh context and sees only the artifact — no loyalty to the work (67:25).

⚙️ Layer 3 — the harness (src/harness.py)

The runtime everything executes inside: clean-context spawn(), one general tool, a spawn budget.

"we take care of all of this for you in the harness so you can think about: what sub agents do I need to spin off?" — 80:02

The one rule that makes isolation real: a spawned worker gets a new, empty context — it never inherits the parent's (69:12). The run above proves it with a real ratio: ~26,000 bytes of sheets are fanned through the workers, but the main agent's context ends at 225 bytes — 115x smaller — because each worker held its own kilobytes and handed back only a small total.

Validated against the literature

Every design choice was cross-checked against primary sources (Anthropic engineering posts, OpenAI/Google framework docs, peer-reviewed papers) and adversarially verified. Verdicts are honest — including where our own claims don't hold up:

choice	verdict	source
verify-first; rule-based grader, LLM-judge only where needed	✅ supported	Anthropic, Demystifying Evals
sub-agent gets its own clean context, returns a small summary	✅ supported	Anthropic, Multi-Agent Research System
independent verifier, no loyalty to the work	✅ supported	Anthropic, Building Multi-Agent Systems
a graph is a costed choice, not a default	✅ strongly supported	Cemri et al., NeurIPS 2025
one general tool (bash) beats many narrow tools	⚠️ contested — folklore; narrow typed tools can cut mis-invocation	no primary source survived review
sandbox is the mandatory outer layer	⚠️ near-certain, under-cited	mainstream security practice
fail closed on budget	🟡 partial — pair with verifier-hardening, not just retry limits	Wang et al. (Qwen Team), The Verification Horizon

Two findings drove the design:

• Verification is now harder than generation for coding agents (Wang et al. (Qwen Team), arXiv:2606.26300) — every verifier is only a proxy for intent that gets gamed under optimization. So evals.py audits the grader; a retry budget alone isn't enough.
• Multi-agent is costly and failure-prone: 3–10× tokens (Anthropic) and 41–86.7% task failure across 7 SOTA frameworks (Cemri et al., NeurIPS 2025). So budget.advise_fanout() makes the graph a decision — single-agent is the default, fan-out is earned.

Full method and per-claim verdicts: FIELD_GUIDE.md.

Tests

python3 -m unittest discover -s tests -v    # 18 tests

They pin the pipeline, not the numbers: the loop really fails attempt 1 and recovers, the harness never leaks a full sheet into the parent context, the independent verifier can reject a bad merge, the fan-out rule prefers single-agent when a graph isn't justified, the cost ceiling fails closed, and grader calibration catches a brittle false-negative.

Honest limits

• The generator is scripted, so the whole thing runs offline and deterministically. Swap src/workers.py's act for a real model call and nothing else changes — the loop, graph, and harness don't know or care what produced the candidate.
• spawn() runs workers in-process. That models isolation; it doesn't enforce it. For real model output, put the worker behind a container/VM — the harness is where that boundary goes (14:17).
• Fan-out is sequential here for a deterministic demo. Real parallelism is the hard part the harness exists to solve (78:37).

MIT — all code original, nothing vendored. · Built from public Anthropic & Karpathy talks; see LECTURE.md.