Skip to content

ci: Increase macrobenchmark JobSet timeout for FSDP strategies #987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yuxin00j wants to merge 3 commits into from
Closed

ci: Increase macrobenchmark JobSet timeout for FSDP strategies #987

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8c1b0fb
chore: Increase macrobenchmark timeout for FSDP strategies to 4 hours
yuxin00j Jul 24, 2026
aaa9669
style: fix trailing whitespace in lib.sh
yuxin00j Jul 24, 2026
f230d57
fix: restrict FSDP timeout increase to run-workload step
yuxin00j Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions cloudbuild/macrobenchmarks/scripts/lib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,12 +70,18 @@ shared_workload_helm_args() {

# Poll a JobSet until it reports Completed (return 0) or Failed/timeout (record
# the failure in the ledger, dump diagnostics, return 1). Shared by the
# seed-checkpoint and run-workload steps so the 240x30s poll lives in one place.
# seed-checkpoint and run-workload steps. Default is 240x30s (2h), FSDP gets 480x30s (4h).
# Usage: wait_for_jobset <jobset-name> <step-id>
wait_for_jobset() {
local jobset="$1" step="$2" complete failed
echo "Waiting for JobSet $jobset to complete..."
for _ in $(seq 1 240); do

local max_iterations=240
if [[ "$step" == "run-workload" ]] && [[ "${_TRAINING_STRATEGY}" == "fsdp_full" || "${_TRAINING_STRATEGY}" == "fsdp_sharded" ]]; then
max_iterations=480
fi
Comment thread
yuxin00j marked this conversation as resolved.

echo "Waiting for JobSet $jobset to complete (max_iterations=${max_iterations})..."
for _ in $(seq 1 $max_iterations); do
complete=$(kubectl get jobset "$jobset" -o jsonpath='{.status.conditions[?(@.type=="Completed")].status}' 2>/dev/null || echo "")
failed=$(kubectl get jobset "$jobset" -o jsonpath='{.status.conditions[?(@.type=="Failed")].status}' 2>/dev/null || echo "")
if [ "$complete" = "True" ]; then echo "JobSet $jobset completed."; return 0; fi
Expand Down
Loading