[PROPOSAL] harbor-style retry + error classification for the API-eval path - #19
Open
penfever wants to merge 1 commit into
Open
[PROPOSAL] harbor-style retry + error classification for the API-eval path#19penfever wants to merge 1 commit into
penfever wants to merge 1 commit into
Conversation
Design proposal (no code yet) to bring harbor's retry/error-classification discipline into evalchemy's API-eval path (eval/robust_api.py over lm-eval's get_batched_requests/amodel_call). Centerpiece: an infra-vs-model taxonomy so requests are handled by WHY they failed, not uniformly. INFRA_TRANSIENT/RATE_LIMIT -> retry with jittered backoff (429 honors Retry-After); MODEL_CONTEXT/MODEL_TRUNCATION -> no retry, score as a miss (never converges); FATAL_CONFIG (auth / bad payload) -> fail loud instead of masking a broken run as a fake 0.0 (the anti-masking refinement over #18). Mined from harbor: lite_llm.py retry_if_not_exception_type passthrough set, daytona per-class retry/wait callbacks, the scienceagentbench 429 jitter backoff, and the trial-level benign-vs-not-scored split. Includes: taxonomy table (error -> class -> policy -> outcome), the exact hooks in eval/robust_api.py, concrete backoff/politeness defaults (env-tunable), and a unit + integration + flag-off-parity test plan. Entirely within robust_api.py; no lm-eval fork, no pyproject/uv.lock change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design proposal for review — no implementation yet. Greenlight before any code.
Brings harbor's retry + error-classification discipline into evalchemy's API-eval path (
eval/robust_api.pyover lm-eval'sget_batched_requests/amodel_call, i.e. everyeval.serve_evalendpoint eval). Today that path retries/fails uniformly, with no idea why a request failed.Centerpiece — infra-vs-model taxonomy:
Retry-After.finish_reason=="length") → no retry, score as a miss — retrying never converges and wastes budget.Preserves #18's guarantee (one bad item never nukes the batch) while adding: don't waste budget on non-convergent items, and don't hide a broken run.
Mined from harbor (cited in the doc):
lite_llm.py'sretry_if_not_exception_typepassthrough set (context/output/auth),daytona/utils.py's per-class retry/wait callbacks + transient string patterns,scienceagentbench's 429 jitter backoff, andtrial/errors.py's benign-vs-not-scored split.Doc:
docs/retry_error_classification_plan.md— taxonomy table (error → class → policy → outcome), exact hooks inrobust_api.py, concrete env-tunable backoff/politeness defaults, and a unit + integration (fault-injecting proxy over the mlx smoke) + flag-off-parity test plan. Entirely withinrobust_api.py: no lm-eval fork, no pyproject/uv.lock change.Open questions for the reviewer are listed in §8 (FATAL default = abort vs flag; 429 budget/caps; adaptive cooldown phasing; truncation count-only vs retry-larger).