test: add agent evals#229
Draft
angeloashmore wants to merge 13 commits into
Draft
Conversation
Plain vitest evals that drive a real Claude Code agent (via the Agent SDK) against the built CLI in a generic, hermetic config. Adds the `agent` fixture, an `toHaveRun` matcher, an LLM-judge matcher with a never-regress ratchet, and a per-run reporter that records efficiency metrics and the agent's prismic calls. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Merge the agent fixture, matchers, judge, config isolation, and result recording into eval/it.ts; delete claude.ts, report.ts, and setup.ts. Move the judge ratchet from a read-back of results.jsonl into an inline threshold literal that vitest -u rewrites like a snapshot, so results.jsonl is write-only. Trim recorded fields to effectiveness and effort (turns, tokens, duration); drop cost, which swings with provider pricing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Judge uses the SDK's native structured output instead of hand-parsing JSON; score is range-checked to [0, 1] - Create a throwaway Claude config dir per agent/judge call and clean it up; set persistSession: false so runs stay ephemeral - Detect -u from vitest's update state; the worker's process.argv is empty - Fix raiseLiteral rewriting the wrong literal on the callsite line, and support inline bars - Fix the evals npm script so vitest also receives RUN_EVALS Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each eval now runs EVAL_TRIALS times (default 3) via it.for; a custom reporter prints per-eval pass rates with infra failures excluded and appends per-trial rows (pass, cost, tokens, turns, commands) to results.jsonl. The judge is binary like agent-eval's: prose criteria, pass/fail verdicts, no scores, no ratcheting bars. The full eval catalog is implemented as it.skip drafts; only the rich-text smoke eval runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`node --run evals:report` prints one block per eval with one line per run (pass rate, cost, mean tokens/turns/duration) and a latest-run footer with best-of and per-trial rates. Rows now carry a per-run id so the digest groups runs exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rows record the git SHA that produced them, shown in the digest so rate changes point at a commit range. The file keeps the newest 100 runs; the digest shows the last 10 per eval. results.jsonl merges with the union driver so branch runs never conflict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The judge now calls the Messages API directly with structured outputs instead of spawning a Claude Code session. The agent fixture writes its own task.meta via the built-in task context, trial rows no longer carry a parsed trial number, and eval env vars are parsed with a zod schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Enable 26 validated evals with per-trial runs and skip 3 that fail consistently, with the reason noted on each. Trim duplicate evals and merge the two slice component evals into one. Each eval now gets its own repository via an isolatedRepo fixture, letting evals run 8-way concurrent without cross-test interference. Global teardown sweeps orphaned repositories created by agents (e.g. prismic init) that the harness cannot name in advance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add four evals from the Next.js happy-path docs: init with an existing repository, page types vs custom types, commit-then-push flow, and production previews after a deploy. The last two fail today and stay enabled: evals are measurements, and their pass rates track CLI gaps. Un-skip the other agent-capability evals for the same reason; only the field rename eval stays skipped since the CLI cannot rename fields. toHaveRun now evaluates each shell segment separately and ignores --help invocations, so reading help no longer counts as running a command in either direction. Scope the results digest to the last EVAL_HISTORY runs (default 10) with a per-run footer, and document the results.jsonl schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adopts main's isolateRepo fixture API from #231: drops the serial test project split, moves the exec fixture into eval/it.ts, and switches the eval repo isolation to isolateRepo: true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Evals only shell out to git; the generic runner's options plumbing was left over from the pre-#231 CLI fixture. throwOnError surfaces setup failures that previously passed silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves:
Description
Adds an eval suite that measures how well AI agents (like Claude Code) can use the CLI.
Each eval gives an agent a task in a temp project. The agent runs the real CLI against a real repo, and the test checks the outcome. A few notes on the design:
prismic init).node --run evals:report(scope withEVAL_HISTORY).node --run evals) and refuse to run outside a container or CI, because the agent runs with permissions bypassed.Checklist
Preview
How to QA 1
Inside a disposable container with
.env.test.localavailable:🤖 Generated with Claude Code
Footnotes
Please use these labels when submitting a review:
⚠️ #issue: Strongly suggest a change.
❓ #ask: Ask a question.
💡 #idea: Suggest an idea.
🎉 #nice: Share a compliment. ↩