Skip to content

Add AIME26 benchmark - #4

Open
taivu1998 wants to merge 1 commit into
marin-community:mainfrom
taivu1998:tdv/aime26-benchmark
Open

Add AIME26 benchmark#4
taivu1998 wants to merge 1 commit into
marin-community:mainfrom
taivu1998:tdv/aime26-benchmark

Conversation

@taivu1998

Copy link
Copy Markdown

Summary

  • Add an AIME26 custom benchmark adapter following the existing AIME25 prompt, repeat, boxed-answer extraction, and is_equiv scoring pattern.
  • Vendor eval/chat_benchmarks/AIME26/data/aime26.json with 30 AIME 2026 problems from MathArena, mapped as I-1 through I-15 and II-1 through II-15.
  • Document AIME26 in the README benchmark list and add a focused stdlib-only data integrity test.

Data Source and License

  • Source dataset: https://huggingface.co/datasets/MathArena/aime_2026
  • License reported by the source dataset: CC BY-NC-SA 4.0.
  • The vendored JSONL keeps the same lightweight local schema used by AIME25: id, problem, solution, answer, and url.
  • solution is intentionally left blank because this PR only vendors problem statements and final answers.

Validation

  • uvx --with pytest pytest tests/test_aime26_data.py passed: 3 tests.
  • python3 -m py_compile eval/chat_benchmarks/AIME26/eval_instruct.py tests/test_aime26_data.py passed.
  • Local AIME26 data invariants passed: 30 rows, ordered unique IDs, expected answer sequence, numeric answers in the AIME 0-999 domain, blank solutions, source URL present on every row.
  • Live source parity was checked against freshly downloaded MathArena/aime_2026 rows: local problems and answers match exactly.
  • The first 15 rows were checked against MathArena/aime_2026_I and match exactly.
  • A stubbed TaskManager(task_list=["AIME26"]) discovery and scoring smoke test passed.
  • git diff --check passed.

Notes

  • Full uv run python ... import validation is blocked before reaching AIME26 by the existing project metadata issue at pyproject.toml: fschat @ file:eval/chat_benchmarks/MTBench.
  • I did not find a public official MAA problem/answer source during the source research pass, so this PR should be treated as MathArena-sourced rather than MAA-officially verified.

@taivu1998 taivu1998 changed the title Add AIME26 benchmark from MathArena Add AIME26 benchmark Jul 6, 2026
@taivu1998
taivu1998 marked this pull request as ready for review July 6, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant