[eval] Keep endpoint benchmark extras torch-free - #55
Merged
Conversation
Six benchmark extras installed PyTorch for cleanup, no-grad wrappers, data containers, or eager vendored local-model imports even when evaluating a served endpoint. Make those imports optional and keep FastChat's local adapters on the vllm extra. Expand lean and isolated-extra checks to exercise every benchmark without torch.
|
Working on this in loom: https://loom.oa.dev/s/hts94mwg |
Import LiveBench API and local-model symbols from their concrete modules. This removes the lazy module attribute shim while keeping the benchmark's endpoint imports torch-free.
rjpower
reviewed
Jul 27, 2026
|
FYI @penfever part of making evalchemy easy to do a |
Load local inference modules only when torch is installed so endpoint code can use the same adapter registry and model aliases. Keep the missing local-inference extra failure explicit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Keep all per-benchmark endpoint and grading extras torch-free. Six benchmarks previously installed PyTorch: BigCodeBench and CruxEval only cleaned CUDA caches, alpaca_eval wrapped generation in
no_grad, MixEval used torch-backed data and RNG helpers, and LiveBench and MTBench reached torch-heavy vendored local-model adapters while selecting conversation templates or grading.Make those operations optional while retaining torch-backed cleanup, seeding, and model loading when local inference is installed. The
vllmextra now carries FastChat'sacceleratedependency. LiveBench endpoint mode uses the same model-specific adapter registry and alias normalization without importing its local torch loaders.Validate all 15 non-empty benchmark extras in isolated environments and dynamically import all 17 empty-extra benchmarks from the lean install. The remaining torch dependency is reachable only through
evalchemy[vllm].The audit record is available at https://loom.oa.dev/s/hts94mwg.