Skip to content

Fix nested boxed answer parsing 🤖🤖🤖 - #254

Open
cupkk wants to merge 1 commit into
NVIDIA:mainfrom
cupkk:fix/nested-boxed-metrics
Open

Fix nested boxed answer parsing 🤖🤖🤖#254
cupkk wants to merge 1 commit into
NVIDIA:mainfrom
cupkk:fix/nested-boxed-metrics

Conversation

@cupkk

@cupkk cupkk commented Jul 31, 2026

Copy link
Copy Markdown

PR description

AIME25 and Math500 currently extract a boxed answer by splitting at the first closing brace. Nested LaTeX such as \boxed{\frac{1}{2}} is therefore truncated to \frac{1, causing a correct model answer to be scored as incorrect.

Add a shared balanced-brace parser and use it in both benchmarks. AIME25 keeps its existing behavior of selecting the last boxed answer, while Math500 continues to select the first. The focused tests cover nested LaTeX, first/last selection, missing answers, and unmatched braces.

Local verification:

python -m unittest tests/test_evaluation_utils.py
python -m compileall -q evaluation/benchmarks/utils.py evaluation/benchmarks/aime25/calculate_metrics.py evaluation/benchmarks/math500/calculate_metrics.py tests/test_evaluation_utils.py
git diff --check

The three focused tests pass. The full test suite and make style were not run because this minimal environment intentionally did not install PyTorch, Transformers, datasets, or the development toolchain.

AI assistance: Codex helped inspect the repository, draft the focused implementation, and prepare the tests. I reviewed every changed line and verified the behavior and remote diff.

Checklist

Before submitting a PR, please make sure:

  • Tests are working (make test)

  • Code is formatted correctly (make style, on errors try fix with make format)

  • Copyright header is included

  • All commits are signed-off using git commit -s

  • (new press) mypress_press.py is in the presses directory (not applicable)

  • (new press) MyPress is in __init__.py (not applicable)

  • (new press) README.md is updated with a 1 liner about the new press in the Available presses section (not applicable)

  • (new press) New press is in the default_presses list in tests/default_presses.py (not applicable)

  • (new press) A docstring is provided that follows the same structure as the existing ones (not applicable)

🤖🤖🤖

Signed-off-by: Haijie Zhi <133995660+cupkk@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 31, 2026 01:40
@copy-pr-bot

copy-pr-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cupkk
cupkk force-pushed the fix/nested-boxed-metrics branch from 234974d to 61bbffb Compare July 31, 2026 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants