robot-eval-bench

Benchmarking task-success evaluation for VLMs: can a vision-language model tell whether a robot actually did the task?

The hardest question in robot learning isn't "can the policy move." It's "did it actually do the thing." A policy can pick up a cup with picture-perfect motion and still put it down in the wrong spot, and if your evaluation can't tell those two runs apart, every number downstream is fiction. Before you can trust a policy, you have to trust the thing grading it.

This repo is the open dataset and code behind that question. We put five frontier models to work as robot-policy judges, scoring whether an episode accomplished its task from keyframes and from video, across 16 open-source LeRobot datasets. Everything needed to reproduce the numbers, and to benchmark a new model against the same ground truth, is here.

Read the write-up: kiteml.com/blog/robot-eval-bench · Built by Kite.

Results

Accuracy is the fraction of episodes the judge labeled correctly (abstentions count as incorrect, since the judge was asked and didn't answer).

Judge	Keyframes	Video
Gemini 3.1 pro	0.88	0.59
Gemini 3.6 flash	0.94	0.63
Claude Opus 5	0.76	0.73
GPT 5.6 Sol	0.90	0.92
Kimi 3	0.86	0.84

Cost per episode (USD), macro-averaged across datasets:

Judge	Keyframes	Video
Gemini 3.1 pro	$0.0058	$0.0005
Gemini 3.6 flash	$0.0058	$0.0008
Claude Opus 5	$0.0095	$0.0551
GPT 5.6 Sol	$0.0078	$0.0538
Kimi 3	$0.0019	$0.0081

Two things flip depending on the model

Accuracy. More frames should mean better judgment, so video should beat keyframes everywhere. It doesn't. The reasoning models (GPT-5.6 Sol, Kimi 3, Claude Opus 5) hold up or improve on video, while both Gemini judges are excellent on keyframes and then crater on the full clip (Flash drops from 0.94 to 0.63). The right approach is a property of the judge you picked, not a universal law.

Cost. For Gemini, video is the cheap option: one natively-encoded clip costs almost nothing, while a few full-resolution keyframes cost about ten times more. For the frame-sampling models it flips, because they read video as ~16 separate frames, so Opus and GPT jump 5-7x from keyframes to video. Kimi 3 is the cheapest reasoning judge either way, and sits with Gemini 3.6 Flash on the efficient frontier of the keyframes cost/accuracy trade-off.

What's in here

data/
  episodes.jsonl      one row per (dataset x model x approach x episode) judgment
  results.json        per (model, approach) accuracy + cost + per-dataset breakdown
  ground_truth.json   the reproducible labeled set: per-episode labels + the config
                      (dataset, seed, negatives) that regenerates it
  datasets.json       the 16 datasets, task strings, and per-dataset config
src/
  selection.py        episode selection + how negatives are synthesized (+ frame sampling)
  cost.py             the token-based cost model and the price table
  judges.py           the shared prompt and exactly how each model was called
  evaluate.py         scoring, and the entry point for adding a new model
  charts.py           regenerates the figures from data/results.json
charts/               the generated figures

Reproduce it

No API keys or dataset downloads needed to check the headline numbers, they're recomputed from the shipped per-episode predictions:

python src/evaluate.py          # re-scores every model from data/episodes.jsonl
pip install matplotlib
python src/charts.py            # regenerates the figures

The details

Each episode evaluated lives in data/episodes.jsonl: its dataset, the judge and approach, the ground-truth label, the source (demo success or synthetic negative), the failure mode for negatives, the model's prediction, whether it was correct, and its confidence and latency.

How episodes are selected is in src/selection.py. The open-source episodes uploaded to HuggingFace are almost entirely success examples, so we build a class-balanced set: each demo is a success, and one matched negative is synthesized per demo by corrupting it (truncate, mismatch, shuffle, reverse). Each negative carries its failure mode. The exact set is determined by (dataset, seed, negatives), stored per dataset in data/ground_truth.json, so it's fully reproducible.

How cost is calculated is in src/cost.py. Every call records its input/output tokens; dollar cost comes from an editable price table after the fact. The per-episode figure divides total cost by every episode the judge was called on, abstentions included, then macro-averages across datasets. Native-video Gemini runs predate token logging, so they fall back to a recorded cost_usd.

How each model was called is documented in src/judges.py: the identical strict-JSON prompt, tolerant verdict parsing, and a MODELS table with each model's ID, provider, API, whether it takes native video, its max-output-tokens field, and whether it accepts a custom temperature (reasoning models don't). Keyframes = 4 evenly-spaced stills (first and last included); video = a native MP4 for Gemini, ~16 dense frames for everyone else.

A note on the ground truth. Some LeRobot labels are wrong. aloha_static_towel ships a description about cleaning up a spilled liquid that never appears in the footage, when the robot is really placing a paper towel next to a tipped-over can. Grade a judge against the wrong task and it fails every episode, so a handful of task strings were corrected before benchmarking. Corrections are flagged with task_overridden in data/datasets.json.

Add your own model

The ground truth, episode selection, and scoring are all model-agnostic, so a new model grades against exactly the same episodes and labels every other model saw:

1. Implement judges.Judge.judge(task, frames, video_path=...) for your model (see the MODELS table for the calling conventions of the existing five).
2. For each dataset in data/ground_truth.json, reconstruct the labeled set with selection.build_labeled_set(demos, negatives=cfg["negatives"], seed=cfg["seed"]) using that dataset's config. Load the demos with any LeRobot loader.
3. Encode per approach (episode.sample_frames(4) for keyframes; native MP4 or 16 frames for video), call your judge, and evaluate.score(predictions, labels).

License

MIT, see LICENSE. The underlying episodes are from the LeRobot datasets on HuggingFace under their respective licenses; this repo ships derived labels and judgments, not raw episode video.