Skip to content

Fix: initialize dep-gen before orchestrator submission#1454

Open
Little-oil wants to merge 1 commit into
hw-native-sys:mainfrom
Little-oil:fix/dep-gen-init-before-submit
Open

Fix: initialize dep-gen before orchestrator submission#1454
Little-oil wants to merge 1 commit into
hw-native-sys:mainfrom
Little-oil:fix/dep-gen-init-before-submit

Conversation

@Little-oil

Copy link
Copy Markdown
Contributor

Summary

  • initialize the dep-gen buffer during leader pre_handshake_init, before the decoupled orchestrator can submit task 0
  • keep PMU and dump setup post-handshake so the barrier-free scheduler/orchestrator overlap remains intact
  • run the a2a3 and a5 dep-gen smoke with full-device worker counts to cover the initialization window

Root cause

The barrier-free scheduler initialization lets the orchestrator build the graph while scheduler threads are still handshaking. Dep-gen initialization was left in the post-handshake profiling path, so early dep_gen_aicpu_record_submit calls saw a null collector state and returned without recording the leading tasks or their dependencies.

Testing

  • pre-commit hooks on all changed files
  • a2a3sim dep-gen smoke (1 passed)
  • a5sim dep-gen smoke (1 passed)
  • a2a3sim L2 swimlane suite with dep-gen (2 passed)
  • a5sim L2 swimlane suite with dep-gen (2 passed)
  • original a2a3 DeepSeek V4 Flash MoE reproduction succeeds with this runtime worktree (requester confirmed)

Initialize the dep-gen buffer in the leader pre-handshake path before
hs_setup_done_ releases the decoupled orchestrator. This prevents early
task submissions from being silently skipped while scheduler threads
are still handshaking.

Keep PMU and dump initialization post-handshake, and run the a2a3/a5
dep-gen smoke with the full worker count so task zero remains covered
under the wider handshake window.
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Little-oil, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 51 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6a6537fe-2012-45eb-8216-489c21991109

📥 Commits

Reviewing files that changed from the base of the PR and between d0bc661 and e6a63a6.

📒 Files selected for processing (7)
  • src/a2a3/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp
  • src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp
  • src/a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp
  • src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp
  • src/common/platform/include/aicpu/dep_gen_collector_aicpu.h
  • tests/st/a2a3/tensormap_and_ringbuffer/dfx/dep_gen/test_dep_gen.py
  • tests/st/a5/tensormap_and_ringbuffer/dfx/dep_gen/test_dep_gen.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ChaoWao

ChaoWao commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Drive-by data point: this race also fires onboard on a2a3, not just in the sim smokes.

st-onboard-a2a3 on an unrelated PR (#1482, a Python-only RunHandle change) failed its dep_gen smoke step with exactly the signature this PR's root cause predicts — the leading task missing from the sidecar:

tests/st/a2a3/tensormap_and_ringbuffer/dfx/dep_gen/test_dep_gen.py::TestDepGen::test_run
AssertionError: tasks[] missing expected ids: {0}
  assert {0, 4294967296, ...} <= {4294967296, ...}

Run: https://github.com/hw-native-sys/simpler/actions/runs/30184309146/job/89746344476

Confirmed intermittent: a plain re-run of the same job on the same commit passed, and the same test passed 3/3 locally on a2a3 silicon under CI-equivalent flags (--device-num 4 -p no:xdist --require-pto-isa --enable-dep-gen). Consistent with the initialization window being timing-dependent rather than deterministic.

No action needed here — just noting that the onboard job is an additional (flaky) surface for the same bug, so this fix should quiet it too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants