fix(math-cuda): driver-independent cuda builds (cudarc pin + cubin AOT)#800
Merged
Conversation
a1552af to
9cfc866
Compare
e6188eb to
908b963
Compare
9cfc866 to
20edc0e
Compare
908b963 to
b4e0246
Compare
20edc0e to
1d15d4b
Compare
b4e0246 to
b0bd855
Compare
1d15d4b to
d14bac6
Compare
b0bd855 to
1f94122
Compare
d14bac6 to
23e62bd
Compare
Two driver-independence fixes so `--features cuda` builds AND runs on the team's GPU hardware without CUDARC_CUDA_VERSION env or a hand-picked toolkit. 1. cudarc symbol floor. Pin cudarc's CUDA version to `cuda-12080` in crypto/math-cuda/Cargo.toml (was `cuda-version-from-build-system` + `fallback-latest`). Auto-detect bound the newest symbol set the build toolkit knew (e.g. CUDA 13.1 -> cuDevSmResourceSplit, gated behind cuda-13010/13020), which cudarc eagerly resolves at init and panics on a driver that predates it (580.x = CUDA 13.0 max). Our cudarc surface is all CUDA-11-era, so the 12.8 symbol set (a strict subset every >=12.8 driver exports) resolves cleanly everywhere. Replaces the per-script sed pin; drops it from scripts/gpu_test.sh and adjusts the compat shim in scripts/bench_abba.sh (still rewrites pre-pin baseline shas). 2. PTX ISA version. AOT-compile kernels to native cubin (SASS) via `nvcc --cubin -arch=sm_XX` instead of `--ptx`, loaded through `Ptx::from_binary` (cuModuleLoadData). A cubin carries pre-compiled SASS for a real arch, so the driver loads it regardless of the toolkit's PTX ISA version -- no CUDA_ERROR_UNSUPPORTED_PTX_VERSION and no silent CPU fallback when the toolkit is newer than the driver. Build+run are co-located and the arch is detected from nvidia-smi, so the arch always matches; a too-old toolkit now fails loudly at nvcc build time. CPU-only and nvcc-less stub builds unaffected (empty cubin stub). README "GPU Tests" updated.
… no-op - backend(): warn once when GPU init fails, so a cubin arch mismatch (AOT cubins are arch-specific) doesn't silently disable the GPU with no signal. Stale ptx->cubin comment fixed. - bench_abba.sh: warn when CUDARC_PIN can't apply (post-pin sha, anchor gone) instead of silently no-opping.
…bench_abba_gpu pin guard/restore + cubin comment
bench_abba.sh and bench_abba_gpu.sh carried near-verbatim copies of the prove-time parsing, the CUDARC_PIN compat sed, and the paired-stats python — a stats fix applied to one would silently leave the other reporting different numbers for the same pairs. scripts/lib/bench_abba_common.sh now holds all three; the GPU bench gains the full analysis (exact Wilcoxon, stability diagnostics, verdicts) it previously lacked. README: state the cuda-12080 pin's driver floor (CUDA >= 12.8 / 570+) and that older drivers abort at CUDA init rather than CPU-fallback.
No automation calls it (the GPU bench CI runs bench_abba.sh), and its whole premise — the rigorous ABBA paired-t + Wilcoxon method — is overkill for the question it wrapped: a GPU-on-vs-off delta is 10-40%, which two runs and your eyes resolve. The ABBA machinery earns its keep only for the ~1% effects (bench_abba.sh's PR-vs-baseline deltas), so one bench script is enough.
f57a069 to
210c022
Compare
A cubin is arch-locked, so when nvcc is present but no GPU arch can be detected there is no safe default — the old sm_89 fallback produced a binary that loads on exactly one GPU model (Ada) and silently CPU-falls-back on every other, including lower-arch cards like the 3090 (sm_86, which an sm_89 cubin can't even run — cubins aren't backward-compatible). Panic with an actionable message (set CUDARC_NVCC_ARCH or build on the target host) instead. Only reachable with nvcc present + no visible GPU + no override; nvcc-absent hosts take the empty-stub path and never hit this, so CPU-only CI is unaffected.
The bench-tooling dedup + bench_abba_gpu.sh removal are split into #812 so this PR is purely the cuda build-robustness changes (which want a GPU test run), leaving the trivial script cleanup free to merge independently.
ColoCarletti
approved these changes
Jul 14, 2026
MauroToscano
added a commit
that referenced
this pull request
Jul 16, 2026
Brings in the 5 commits the branch was behind (#815 OOD-width guard, #828 streamed field bytes, #800 cuda build robustness, #826 fused/hoisted DEEP reconstruction, #823 LogUp forward accumulation + g·z OOD pruning). Conflict: - crypto/stark/src/tests/bus_tests/soundness_tests.rs (imports): kept both sides — `use crate::table::Table;` and `use crate::test_utils::{multi_prove_batched_ram, multi_prove_ram};`. Semantic adaptation of the batched round-4 verifier to the post-#826 API (main deleted `reconstruct_deep_composition_poly_evaluation`; #823 changed `step_2_verify_claimed_composition_polynomial`'s signature and added the `trace_ood_next_evaluations` field to `StarkProof`): - batched_synthetic_table_proof: add the new `trace_ood_next_evaluations` field. The batched proof carries the FULL (unpruned) OOD grid in `trace_ood_evaluations`, so the split next-row block is unused on the batched path — set it to an empty table. - batched_verify_round_4 step 2: pass the full OOD grid as `ood_full` plus `air.step_size()` to the widened `step_2_verify_claimed_composition_polynomial`. - batched_verify_round_4 DEEP: replace the two per-point `reconstruct_deep_composition_poly_evaluation` calls with a single `reconstruct_deep_composition_poly_evaluation_pair` (regular + symmetric), hoisting the query-invariant OOD/gamma sums out of the query loop via `compute_query_invariant_deep_terms` (#826). Precomputed and main base columns are now passed as two borrowed slices (no per-query concat). The batched path stays a faithful analog of the (pre-pruning) per-table path: the batched prover commits the full OOD grid and full-width trace-term coefficients, so `next_row_cols` is every column and no g·z pruning is applied — the reconstruction sums the whole grid, matching the batched prover's DEEP codeword exactly. The forward-accumulation constraint change is inherited through the shared AIR (round-3 OOD, boundary_constraints, compute_transition); the batched transcript still absorbs the full grid on both prover and verifier sides, so its Fiat-Shamir semantics are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes cuda builds driver-independent so a box whose driver is older than the build toolkit still runs on the GPU instead of silently falling back to CPU (or panicking at init). Two independent robustness fixes plus bench-script cleanup.
The GPU constraint-eval + trace-residency work this supported has already merged (#798, #799); this is the build-robustness follow-on, now standalone against
main.The two fixes
AOT cubin instead of PTX.
build.rsnow compiles each kernel to a native cubin (SASS) for the host GPU's real arch (nvcc --cubin -arch sm_XX, loaded viaPtx::from_binary) instead of emitting PTX for the driver to JIT. A toolkit's PTX ISA version is fixed by its CUDA version, and a driver older than that toolkit rejects the module at load withCUDA_ERROR_UNSUPPORTED_PTX_VERSION→ every kernel silently falls back to CPU. A cubin carries pre-compiled SASS, so the driver loads it directly regardless of toolkit version. Trade-off: a cubin is arch-specific — we build and run on the same GPU box in every flow and detect the arch fromnvidia-smi, so this is exactly right;CUDARC_NVCC_ARCHoverrides for cross-compiling.Pin cudarc to
cuda-12080. Replacescuda-version-from-build-system+fallback-latest, which bind the newest symbol set the build toolkit knows (e.g. a 13.1 toolkit pulls incuDevSmResourceSplit); cudarc resolves those eagerly at init, so a driver that predates them panics. Our cudarc surface is entirely CUDA-11-era, so the 12.8 set is a strict subset every ≥12.8 driver exports. This replaces the fragile per-scriptsedpin.Driver floor: the pin requires a driver of CUDA ≥ 12.8 (branch 570+; any Blackwell-capable driver qualifies). On an older driver cudarc aborts at init rather than CPU-falling-back — documented in the README.
Also in this PR
nvccis present but no GPU arch can be detected and noCUDARC_NVCC_ARCHis set, the build hard-fails rather than guessing. A cubin is arch-locked, so a guessed arch (the oldsm_89fallback) would load on exactly one GPU model and silently CPU-fall-back on every other card — including lower ones like the 3090 (sm_86), which can't even run ansm_89cubin. A hard error is loud and fixable (setCUDARC_NVCC_ARCHor build on the target host); a guess is neither. (nvcc-absent hosts take the empty-stub path and never reach this, so CPU-only CI is unaffected.)The ABBA bench-script cleanup that originally rode along here is split into #812 — trivial, no build or runtime impact.
Validation
Validated on RTX 5090 boxes with 580 and 590 drivers (13.1 toolkit) — exactly the driver-older-than-toolkit case where the previous PTX path would have failed to load; the cubin path runs with default env, no
CUDARC_CUDA_VERSIONorCUDA_HOMEjuggling.make lintgreen on both the default andcudafeature sets.