fix(debug-trace-server): stop rotating to fallbacks for not-yet-generated frontier witnesses - #164
fix(debug-trace-server): stop rotating to fallbacks for not-yet-generated frontier witnesses#164flyq wants to merge 3 commits into
Conversation
…ontier-fresh witnesses At the sync frontier, "witness not found" from the generator means "not generated yet", not "ask someone else": the fallback endpoints receive witnesses from the same generation pipeline and cannot be ahead of it. The fetcher now routes by freshness against the last remote head observed by the pipeline's latest_block_number polls: frontier-fresh blocks retry the generator alone under a bounded grace before falling back to the full endpoint chain, while deep catch-up blocks (where the generator may have pruned the witness) keep full failover from the first attempt. Adds RpcClient::get_witness_light_first_provider_only, the complement of the skip-based historical routing, by generalizing the witness retry loop's provider selection from a skip prefix to an index range. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude review status
🛠️ Review did not finish Attempted head This round did not publish: MODEL_ACTION_FAILED in phase review_retry. Anything listed below is from the last round that did. Re-run the workflow or push a new commit to try again. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1a8d50f558
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
… generator, cleanup pass The grace now requires the CLI-declared generator (generator_first threaded into TraceFetcher::new), matching the request path's can_skip_generator gate: without --witness-generator-endpoint no endpoint is special, and a no-generator multi-endpoint deployment keeps plain same-round failover instead of pinning frontier sync to endpoint 0 for 6s — pinned by a new no_declared_generator_keeps_plain_failover test, and the two routing-policy docs now cross-reference each other. fetch_witness returns just the LightWitness every caller kept anyway (MptWitness import moves into the test module), test_fetcher builds on TraceFetcher::new via functional update instead of restating its defaults, the twin provider-range tests in rpc_client share one two_endpoint_witness_client scaffold, and the generator-cannot-be-behind rationale now lives once in chain_sync with the rpc_client method doc trimmed to mechanism. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 989b1137ac
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…ne-exceeded metric The grace probe passed its budget as the RPC client's logical-call deadline, so a generator lagging past the grace incremented debug_trace_upstream_deadline_exceeded_total even when the fall-through then succeeded — polluting a counter that otherwise fires only when a request really fails. The grace is now a caller-side tokio timeout around a deadline-less single-provider probe: expiry is an internal routing budget the metric never sees, and rpc_client is untouched. The downed-generator test now runs with a counting RpcMetrics sink and pins deadline_exceeded == 0 across a grace expiry plus fallback success. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
At the sync frontier the chain-sync prefetch treated the generator's "witness not found" as a provider failure and rotated to the public fallback. But a frontier witness the generator doesn't have yet cannot exist on the fallbacks either — they receive witnesses from the same generation pipeline. The fetcher now gives the generator a short exclusive grace for frontier-fresh blocks (retrying it alone with round backoff) and only then falls back to the full endpoint chain; deep catch-up blocks keep today's full failover from the first attempt.
Motivation
A 7.7 h production log (v2.0.14, 2026-07-21) shows the race is the norm, not the exception: 18,766 sync fetches hit "witness not yet generated" at the generator (~68% of blocks) and 18,499 of them rotated to the public fallback, which also missed (~600 ms per wasted round trip) before round 1 retried the generator successfully. Twice in that window the fallback did not answer its miss at all and hung until the 20 s per-attempt cap (blocks 21832224 / 21841054), stalling tip advance ~21 s each — with the witness almost certainly ready seconds in. The fallback won the frontier race in only 267 of 18,766 cases (1.4%). Routing fresh misses back to the generator removes both the daily waste and the routine exposure to fallback stalls; the ~21 s worst case becomes ~1–2 s (generation lag) with the fallback untouched.
Fix
RpcClient::get_witness_light_first_provider_only(crates/stateless-common/src/rpc_client.rs) — the complement of the existing_from(skip)historical routing; the witness retry loop's provider selection generalizes from a skip prefix to an index range.TraceFetchermemoizes the last remote head observed bylatest_block_number(which the pipeline already polls) as the freshness anchor, andfetch_witnessroutes (bin/debug-trace-server/src/chain_sync.rs): blocks withinFRONTIER_FRESHNESS_BLOCKS(256) of that head give the generator aGENERATOR_WITNESS_GRACE(6 s) exclusive window, then fall through to the full chain; an unknown head or deeper blocks keep the full chain from the first attempt.--witness-generator-endpoint), mirroring the request path'scan_skip_generatorgate — single-endpoint and no-declared-generator deployments keep plain same-round failover untouched.Testing
Six new
chain_synctests over real fixture witnesses served by scripted jsonrpsee mocks: a fresh miss retries the generator without touching the fallback; a stale (deep catch-up) miss fails over immediately with no grace; an unknown head classifies as non-fresh; a deployment without a declared generator keeps plain same-round failover; a downed generator falls back right after the grace;latest_block_numberfeeds the anchor. Plus an rpc_client test pinning thatfirst_provider_onlynever rotates past provider 0 (mirror of the existing skip test). Full workspace: 301 tests,cargo fmt/clippy --workspace --all-targets --all-features/codespellall clean.Notes
The public endpoint's 20 s hang-on-miss (vs. its usual ~600 ms
-32603error) is upstream behavior worth checking on the megaeth.com side for the two windows above; this PR removes the sync path's routine exposure to it, but a fallback that stalls when genuinely needed is still bounded only by the 20 s per-attempt cap. The two constants (256 blocks / 6 s) are policy values documented in code with qualitative rationale; their derivation from the log is above.🤖 Generated with Claude Code