Skip to content

fix(math-cuda): driver-independent cuda builds (cudarc pin + cubin AOT)#800

Merged
MauroToscano merged 8 commits into
mainfrom
feat/gpu-cuda-build
Jul 16, 2026
Merged

fix(math-cuda): driver-independent cuda builds (cudarc pin + cubin AOT)#800
MauroToscano merged 8 commits into
mainfrom
feat/gpu-cuda-build

Conversation

@MauroToscano

@MauroToscano MauroToscano commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Visual explainer (the GPU residency work at a glance — what changed, what existed before, and the numbers): https://gistpreview.github.io/?a6bc0786b46cefe50b330ccb4d8f77e9

What

Makes cuda builds driver-independent so a box whose driver is older than the build toolkit still runs on the GPU instead of silently falling back to CPU (or panicking at init). Two independent robustness fixes plus bench-script cleanup.

The GPU constraint-eval + trace-residency work this supported has already merged (#798, #799); this is the build-robustness follow-on, now standalone against main.

The two fixes

  1. AOT cubin instead of PTX. build.rs now compiles each kernel to a native cubin (SASS) for the host GPU's real arch (nvcc --cubin -arch sm_XX, loaded via Ptx::from_binary) instead of emitting PTX for the driver to JIT. A toolkit's PTX ISA version is fixed by its CUDA version, and a driver older than that toolkit rejects the module at load with CUDA_ERROR_UNSUPPORTED_PTX_VERSION → every kernel silently falls back to CPU. A cubin carries pre-compiled SASS, so the driver loads it directly regardless of toolkit version. Trade-off: a cubin is arch-specific — we build and run on the same GPU box in every flow and detect the arch from nvidia-smi, so this is exactly right; CUDARC_NVCC_ARCH overrides for cross-compiling.

  2. Pin cudarc to cuda-12080. Replaces cuda-version-from-build-system + fallback-latest, which bind the newest symbol set the build toolkit knows (e.g. a 13.1 toolkit pulls in cuDevSmResourceSplit); cudarc resolves those eagerly at init, so a driver that predates them panics. Our cudarc surface is entirely CUDA-11-era, so the 12.8 set is a strict subset every ≥12.8 driver exports. This replaces the fragile per-script sed pin.

Driver floor: the pin requires a driver of CUDA ≥ 12.8 (branch 570+; any Blackwell-capable driver qualifies). On an older driver cudarc aborts at init rather than CPU-falling-back — documented in the README.

Also in this PR

  • Fail-loud instead of silent-CPU. Warn-once at runtime when the GPU backend is unavailable; and at build time, when nvcc is present but no GPU arch can be detected and no CUDARC_NVCC_ARCH is set, the build hard-fails rather than guessing. A cubin is arch-locked, so a guessed arch (the old sm_89 fallback) would load on exactly one GPU model and silently CPU-fall-back on every other card — including lower ones like the 3090 (sm_86), which can't even run an sm_89 cubin. A hard error is loud and fixable (set CUDARC_NVCC_ARCH or build on the target host); a guess is neither. (nvcc-absent hosts take the empty-stub path and never reach this, so CPU-only CI is unaffected.)

The ABBA bench-script cleanup that originally rode along here is split into #812 — trivial, no build or runtime impact.

Validation

Validated on RTX 5090 boxes with 580 and 590 drivers (13.1 toolkit) — exactly the driver-older-than-toolkit case where the previous PTX path would have failed to load; the cubin path runs with default env, no CUDARC_CUDA_VERSION or CUDA_HOME juggling. make lint green on both the default and cuda feature sets.

@MauroToscano MauroToscano force-pushed the feat/gpu-cuda-build branch from a1552af to 9cfc866 Compare July 13, 2026 15:38
@MauroToscano MauroToscano force-pushed the feat/gpu-trace-residency branch from e6188eb to 908b963 Compare July 13, 2026 16:14
@MauroToscano MauroToscano force-pushed the feat/gpu-cuda-build branch from 9cfc866 to 20edc0e Compare July 13, 2026 16:14
@MauroToscano MauroToscano force-pushed the feat/gpu-trace-residency branch from 908b963 to b4e0246 Compare July 13, 2026 18:53
@MauroToscano MauroToscano force-pushed the feat/gpu-cuda-build branch from 20edc0e to 1d15d4b Compare July 13, 2026 18:53
@MauroToscano MauroToscano force-pushed the feat/gpu-trace-residency branch from b4e0246 to b0bd855 Compare July 13, 2026 19:06
@MauroToscano MauroToscano force-pushed the feat/gpu-cuda-build branch from 1d15d4b to d14bac6 Compare July 13, 2026 19:06
@MauroToscano MauroToscano marked this pull request as ready for review July 13, 2026 19:19
@MauroToscano MauroToscano force-pushed the feat/gpu-trace-residency branch from b0bd855 to 1f94122 Compare July 14, 2026 15:35
@MauroToscano MauroToscano force-pushed the feat/gpu-cuda-build branch from d14bac6 to 23e62bd Compare July 14, 2026 15:35
Base automatically changed from feat/gpu-trace-residency to main July 14, 2026 16:01
Two driver-independence fixes so `--features cuda` builds AND runs on the
team's GPU hardware without CUDARC_CUDA_VERSION env or a hand-picked toolkit.

1. cudarc symbol floor. Pin cudarc's CUDA version to `cuda-12080` in
   crypto/math-cuda/Cargo.toml (was `cuda-version-from-build-system` +
   `fallback-latest`). Auto-detect bound the newest symbol set the build
   toolkit knew (e.g. CUDA 13.1 -> cuDevSmResourceSplit, gated behind
   cuda-13010/13020), which cudarc eagerly resolves at init and panics on a
   driver that predates it (580.x = CUDA 13.0 max). Our cudarc surface is all
   CUDA-11-era, so the 12.8 symbol set (a strict subset every >=12.8 driver
   exports) resolves cleanly everywhere. Replaces the per-script sed pin;
   drops it from scripts/gpu_test.sh and adjusts the compat shim in
   scripts/bench_abba.sh (still rewrites pre-pin baseline shas).

2. PTX ISA version. AOT-compile kernels to native cubin (SASS) via
   `nvcc --cubin -arch=sm_XX` instead of `--ptx`, loaded through
   `Ptx::from_binary` (cuModuleLoadData). A cubin carries pre-compiled SASS
   for a real arch, so the driver loads it regardless of the toolkit's PTX
   ISA version -- no CUDA_ERROR_UNSUPPORTED_PTX_VERSION and no silent
   CPU fallback when the toolkit is newer than the driver. Build+run are
   co-located and the arch is detected from nvidia-smi, so the arch always
   matches; a too-old toolkit now fails loudly at nvcc build time.

CPU-only and nvcc-less stub builds unaffected (empty cubin stub). README
"GPU Tests" updated.
… no-op

- backend(): warn once when GPU init fails, so a cubin arch mismatch (AOT cubins
  are arch-specific) doesn't silently disable the GPU with no signal. Stale
  ptx->cubin comment fixed.
- bench_abba.sh: warn when CUDARC_PIN can't apply (post-pin sha, anchor gone)
  instead of silently no-opping.
…bench_abba_gpu pin guard/restore + cubin comment
bench_abba.sh and bench_abba_gpu.sh carried near-verbatim copies of the
prove-time parsing, the CUDARC_PIN compat sed, and the paired-stats python —
a stats fix applied to one would silently leave the other reporting different
numbers for the same pairs. scripts/lib/bench_abba_common.sh now holds all
three; the GPU bench gains the full analysis (exact Wilcoxon, stability
diagnostics, verdicts) it previously lacked.

README: state the cuda-12080 pin's driver floor (CUDA >= 12.8 / 570+) and
that older drivers abort at CUDA init rather than CPU-fallback.
No automation calls it (the GPU bench CI runs bench_abba.sh), and its whole
premise — the rigorous ABBA paired-t + Wilcoxon method — is overkill for the
question it wrapped: a GPU-on-vs-off delta is 10-40%, which two runs and your
eyes resolve. The ABBA machinery earns its keep only for the ~1% effects
(bench_abba.sh's PR-vs-baseline deltas), so one bench script is enough.
@MauroToscano MauroToscano force-pushed the feat/gpu-cuda-build branch from f57a069 to 210c022 Compare July 14, 2026 16:06
A cubin is arch-locked, so when nvcc is present but no GPU arch can be
detected there is no safe default — the old sm_89 fallback produced a binary
that loads on exactly one GPU model (Ada) and silently CPU-falls-back on every
other, including lower-arch cards like the 3090 (sm_86, which an sm_89 cubin
can't even run — cubins aren't backward-compatible). Panic with an actionable
message (set CUDARC_NVCC_ARCH or build on the target host) instead. Only
reachable with nvcc present + no visible GPU + no override; nvcc-absent hosts
take the empty-stub path and never hit this, so CPU-only CI is unaffected.
The bench-tooling dedup + bench_abba_gpu.sh removal are split into #812 so this
PR is purely the cuda build-robustness changes (which want a GPU test run),
leaving the trivial script cleanup free to merge independently.
@MauroToscano MauroToscano added this pull request to the merge queue Jul 16, 2026
Merged via the queue into main with commit 4c108a1 Jul 16, 2026
@MauroToscano MauroToscano deleted the feat/gpu-cuda-build branch July 16, 2026 15:52
MauroToscano added a commit that referenced this pull request Jul 16, 2026
Brings in the 5 commits the branch was behind (#815 OOD-width guard, #828
streamed field bytes, #800 cuda build robustness, #826 fused/hoisted DEEP
reconstruction, #823 LogUp forward accumulation + g·z OOD pruning).

Conflict:
- crypto/stark/src/tests/bus_tests/soundness_tests.rs (imports): kept both
  sides — `use crate::table::Table;` and
  `use crate::test_utils::{multi_prove_batched_ram, multi_prove_ram};`.

Semantic adaptation of the batched round-4 verifier to the post-#826 API
(main deleted `reconstruct_deep_composition_poly_evaluation`; #823 changed
`step_2_verify_claimed_composition_polynomial`'s signature and added the
`trace_ood_next_evaluations` field to `StarkProof`):

- batched_synthetic_table_proof: add the new `trace_ood_next_evaluations`
  field. The batched proof carries the FULL (unpruned) OOD grid in
  `trace_ood_evaluations`, so the split next-row block is unused on the
  batched path — set it to an empty table.
- batched_verify_round_4 step 2: pass the full OOD grid as `ood_full` plus
  `air.step_size()` to the widened `step_2_verify_claimed_composition_polynomial`.
- batched_verify_round_4 DEEP: replace the two per-point
  `reconstruct_deep_composition_poly_evaluation` calls with a single
  `reconstruct_deep_composition_poly_evaluation_pair` (regular + symmetric),
  hoisting the query-invariant OOD/gamma sums out of the query loop via
  `compute_query_invariant_deep_terms` (#826). Precomputed and main base
  columns are now passed as two borrowed slices (no per-query concat).

The batched path stays a faithful analog of the (pre-pruning) per-table path:
the batched prover commits the full OOD grid and full-width trace-term
coefficients, so `next_row_cols` is every column and no g·z pruning is
applied — the reconstruction sums the whole grid, matching the batched
prover's DEEP codeword exactly. The forward-accumulation constraint change is
inherited through the shared AIR (round-3 OOD, boundary_constraints,
compute_transition); the batched transcript still absorbs the full grid on
both prover and verifier sides, so its Fiat-Shamir semantics are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants