Skip to content

Add AMC24 and AMC25 text-ready benchmarks - #5

Open
taivu1998 wants to merge 2 commits into
marin-community:mainfrom
taivu1998:tdv/amc24-amc25
Open

Add AMC24 and AMC25 text-ready benchmarks#5
taivu1998 wants to merge 2 commits into
marin-community:mainfrom
taivu1998:tdv/amc24-amc25

Conversation

@taivu1998

@taivu1998 taivu1998 commented Jul 6, 2026

Copy link
Copy Markdown

Summary

Adds AMC24 and AMC25 as committed, local Evalchemy chat benchmarks:

  • vendors reviewed AMC 12 A/B text-ready subsets for 2024 and 2025
  • includes source-figure rows only when the prompt text fully specifies the problem without needing the diagram
  • adds source manifests with canonical MAA/AoPS provenance, Hugging Face mirror cross-checks, row counts, manually included self-contained figure rows, and remaining exclusions
  • adds a shared AMC benchmark helper for prompt rendering, repeated generation, boxed-answer extraction, and scoring
  • adds lightweight AMC answer utilities so normalization/scoring can be tested without requiring the full lm_eval stack
  • documents AMC24/AMC25 in the README benchmark list

The dataset-generation script and planning files are intentionally excluded from this PR; only the committed benchmark data, runtime adapters, shared helpers, tests, and README changes are included.

Data Scope

  • AMC24 includes 48 text-ready rows.
  • AMC25 includes 49 text-ready rows.
  • Remaining excluded rows:
    • AMC24: 2024-amc12a-18, 2024-amc12a-22
    • AMC25: 2025-amc12a-05
  • Rows with source figures are marked with has_figure=true plus figure_handling=source_figure_omitted_text_self_contained when the diagram is redundant for the converted text prompt.
  • Runtime benchmark execution uses local committed JSONL only, with no network fetch.

Validation

  • uvx --with pytest pytest tests/test_amc_benchmarks.py -q
  • uvx ruff check eval/chat_benchmarks/amc_common.py eval/chat_benchmarks/amc_utils.py eval/chat_benchmarks/AMC24/eval_instruct.py eval/chat_benchmarks/AMC25/eval_instruct.py tests/test_amc_benchmarks.py
  • uvx ruff format --check eval/chat_benchmarks/amc_common.py eval/chat_benchmarks/amc_utils.py eval/chat_benchmarks/AMC24/eval_instruct.py eval/chat_benchmarks/AMC25/eval_instruct.py tests/test_amc_benchmarks.py
  • python3 -m py_compile eval/chat_benchmarks/amc_common.py eval/chat_benchmarks/amc_utils.py eval/chat_benchmarks/AMC24/eval_instruct.py eval/chat_benchmarks/AMC25/eval_instruct.py tests/test_amc_benchmarks.py
  • structured data-contract check for row counts, exact exclusions, manual inclusion IDs, and figure-handling metadata

Note: uv run pytest tests/test_amc_benchmarks.py -q still fails before collection on the existing project metadata issue: fschat @ file:eval/chat_benchmarks/MTBench uses a relative file dependency that uv cannot resolve while building the package.

@taivu1998
taivu1998 marked this pull request as ready for review July 6, 2026 06:36
@taivu1998
taivu1998 marked this pull request as draft July 6, 2026 06:49
@taivu1998
taivu1998 marked this pull request as ready for review July 6, 2026 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant