ci: Increase macrobenchmark JobSet timeout for FSDP strategies

[yuxin00j]

This increases the wait_for_jobset polling limit in lib.sh from 2 hours to 4 hours specifically for fsdp_full and fsdp_sharded runs.

We observed that FSDP runs were frequently exceeding the default 2-hour timeout, causing scrape_metrics.sh to exit without uploading the partial CSV metrics to the results bucket. The overall Cloud Build limit of 6 hours safely accommodates this new 4-hour limit.

[gemini-code-assist]

Code Review

This pull request updates the wait_for_jobset function in cloudbuild/macrobenchmarks/scripts/lib.sh to dynamically increase the maximum polling iterations from 240 to 480 when the training strategy is fsdp_full or fsdp_sharded. The reviewer noted that applying this extended timeout to the seed-checkpoint step is risky because it could cause the build to hang for too long and exceed the overall Cloud Build limit, thereby skipping critical cleanup steps. It is recommended to restrict this timeout increase specifically to the run-workload step.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.68%. Comparing base (e6c107b) to head (f230d57).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #987   +/-   ##
=======================================
  Coverage   89.68%   89.68%           
=======================================
  Files          16       16           
  Lines        3579     3579           
=======================================
  Hits         3210     3210           
  Misses        369      369           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.