[e2e] Resolve the served model id from /v1/models - #54
Closed
weaverbot wants to merge 1 commit into
Closed
Conversation
The nightly e2e 404'd on every eval request after marin-core floated to 0.2.58.dev202607250745: marin-serve began launching vLLM with --served-model-name set to the resolved weights cache path, so /v1/models no longer advertised the requested id Qwen/Qwen3-0.6B and lm-eval's model=Qwen/Qwen3-0.6B requests returned NotFoundError. The runner had assumed the served name equals the HF id it asked to serve. serve_eval now reads /v1/models once the endpoint is ready and uses the advertised id as the request model, keeping the requested HF id only as the lm-eval tokenizer for length bookkeeping. It prefers the requested id when the endpoint advertises it, adopts the sole advertised id otherwise, and refuses an ambiguous mismatch, so the eval lane holds across both providers and future marin-serve renames. The served-name change is a marin-core regression tracked in marin-community/marin#7636. Part of #40
This was referenced Jul 25, 2026
Author
|
🤖 Closing — this will be fixed on the marin side. Root cause is marin-serve advertising the resolved weights cache path as the served model name (marin-community/marin#7636); with that fixed upstream, the /v1/models discovery workaround here isn't needed. Triage details in #40. |
Author
|
🤖 The marin-side fix (marin-community/marin#7637) is verified end-to-end on a TPU |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nightly e2e 404'd on every eval request after marin-core floated to
0.2.58.dev202607250745: marin-serve began launching vLLM with
--served-model-name set to the resolved weights cache path, so /v1/models no
longer advertised the requested id Qwen/Qwen3-0.6B and lm-eval's
model=Qwen/Qwen3-0.6B requests returned NotFoundError (run 30152121453). The TPU
served and cleanup stopped the Iris job; the failure was purely the model-name
mismatch, and the runner had assumed the served name equals the HF id it asked to
serve.
serve_eval now reads /v1/models once the endpoint reports ready and uses the
advertised id as the request model, keeping the requested HF id only as the
lm-eval tokenizer for length bookkeeping. It prefers the requested id when the
endpoint advertises it, adopts the sole advertised id otherwise, and refuses an
ambiguous mismatch. This holds for both providers and survives future marin-serve
renames.
The served-name change is a marin-core regression tracked in
marin-community/marin#7636; this change keeps the eval lane green regardless.
Part of #40