pkg/steps: pin bundle builds to amd64 nodes

[kaovilai]

Note

Responses generated with Claude

Summary

Bundle-source (src-bundle) and bundle image builds are single-arch by design and were created without a node selector, so on heterogeneous build clusters the build pod can land on an arm64 node. The build inputs (e.g. pipeline:src) are manifest lists that contain only an amd64 entry for single-arch repositories, so buildah then fails with:

error: error creating buildah builder: choosing an image from manifest list docker://image-registry.openshift-image-registry.svc:5000/<ci-op-ns>/pipeline@sha256:<digest>: no image found in manifest list for architecture "arm64", variant "v8", OS "linux"

This PR pins both bundle build sites (bundleSourceStep.run in pkg/steps/bundle_source.go and the IsBundleImage() branch in pkg/steps/project_image.go) to amd64 nodes via a shared pinBuildToSingleArchNode helper that merges a kubernetes.io/arch entry into the build's node selector (merge rather than replace, so a selector set earlier on the build would survive), matching what constructMultiArchBuilds already does for pipeline builds. A tripwire unit test (TestSingleArchBuildPinMatchesPipelineArchGuarantees) locks the pin to the pipeline-side guarantees it depends on.

Fixes #5328

Why the previous "no node selector" behavior is wrong

The removed comment said the build "will be scheduled on any available node no matter the architecture" — that was written when it was harmless, but since pipeline outputs became manifest lists (#3587, reworked in #5004), the content buildah pulls is resolved by the architecture of the node the pod happens to land on. For a single-arch repo the manifest list has no arm64 entry (amd64 is the only architecture built unless more are configured), so the build's correctness depends on scheduler placement: amd64 node → works, arm64 node → hard failure. All 5 in-run retry attempts can land on arm64 too (FetchImageContentFailed ×5). That is a flake by construction, not a valid "any node" invariant.

Evidence (fail → retry-pass on the same PR/job)

From openshift/oadp-operator#2313:

• pull-ci-...oadp-dev-4.22-e2e-test-cli-aws: FAILURE 2026-07-23 00:48 UTC (exact error above in build-log) → retry SUCCESS 02:34 UTC
• pull-ci-...oadp-dev-4.23-ci-index: FAILURE 2026-07-22 22:40 UTC (same error, then src-bundle failed after 5 attempts ... FetchImageContentFailed) → retry SUCCESS 22:58 UTC
• second ci-index pair the next day: FAILURE 13:19 UTC → retry SUCCESS 13:33 UTC

Farm-wide: search.ci, no image found in manifest list for architecture, build-logs, 24h → 93 job-runs in 24 hours, e.g. kubevirt/hyperconverged-cluster-operator#4421, assisted-service periodic, csi-operator periodic, plus openshift-kni/, medik8s/, lvm-operator.

Why pinning to amd64 cannot break anyone (including additional_architectures repos)

• amd64 is unconditional in every pipeline. additional_architectures is strictly additive — pkg/api/types.go: "AdditionalArchitectures is a list of additional architectures to build for. AMD64 architecture is included by default." There is no knob to remove amd64, and projectDirectoryImageBuildStep.ResolveMultiArch() inserts NodeArchitectureAMD64 unconditionally before adding configured capabilities. So a repo with additional_architectures: [arm64] produces a pipeline:src manifest list containing both amd64 and arm64 — the amd64-pinned bundle build resolves its amd64 entry and works exactly as for single-arch repos. The failing direction (node arch missing from the list) cannot occur in reverse, because no configuration produces a manifest list without an amd64 entry.
• No new scheduling requirement. Every job that builds src already requires amd64 capacity: constructMultiArchBuilds pins src-amd64 (and every other pipeline build) to kubernetes.io/arch: amd64 nodes. If a cluster could not schedule the pinned bundle build, it could not have built src in the first place.
• Bundles are amd64 artifacts by design. ci-operator's bundle/index tooling is amd64-only (index_generator.go: "At the moment, we support only amd64"), and bundle image content is architecture-independent metadata/YAML — the node arch never affects the produced bundle, only whether its inputs can be pulled.
• Guarded by a tripwire test. TestSingleArchBuildPinMatchesPipelineArchGuarantees (in bundle_source_test.go) asserts the invariants above against the real functions: the default constructMultiArchBuilds output targets the pinned arch, ResolveMultiArch() contains it even when only other architectures are configured, multi-arch pipelines still produce a build for it, and pinning merges into (not replaces) an existing node selector. If a future change breaks any link in that chain, this test fails and points back at pinBuildToSingleArchNode.

Test plan

• go build ./pkg/steps/...
• gofmt -l clean
• go test ./pkg/steps/... passes, including new TestSingleArchBuildPinMatchesPipelineArchGuarantees

/cc @droslean

Summary

• Updates the CI build steps that produce src-bundle and the corresponding bundle images so they are scheduled only on amd64 nodes (kubernetes.io/arch stable/amd64), even when the pipeline inputs are provided via manifest lists.
• Eliminates intermittent src-bundle failures caused by buildah pods landing on arm64 while consuming pipeline:src manifest lists that may only include amd64 images.
• Applies the amd64 pin consistently across bundle build paths by using a shared pinBuildToSingleArchNode helper (that merges with any existing NodeSelector), covering both:
  • bundleSourceStep.run in pkg/steps/bundle_source.go
  • the IsBundleImage() branch in pkg/steps/project_image.go
• Adds a unit test (TestSingleArchBuildPinMatchesPipelineArchGuarantees) to enforce that the pinned architecture stays aligned with the pipeline multi-arch manifest-list guarantees and that the selector pinning logic merges rather than replaces.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 196edf25-754e-4aba-9253-a34c69bf16b2

📥 Commits

Reviewing files that changed from the base of the PR and between 416c24f and 0d57bf8.

📒 Files selected for processing (4)

• pkg/steps/bundle_source.go
• pkg/steps/bundle_source_test.go
• pkg/steps/project_image.go
• pkg/steps/source.go

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• openshift/release (manual)
• openshift/ci-docs (manual)
• openshift/release-controller (manual)
• openshift/ci-chat-bot (manual)

🚧 Files skipped from review as they are similar to previous changes (4)

• pkg/steps/bundle_source.go
• pkg/steps/source.go
• pkg/steps/bundle_source_test.go
• pkg/steps/project_image.go

---

📝 Walkthrough

Walkthrough

Bundle source and bundle-image builds now pin their build pods to AMD64 while preserving existing node selector entries. Tests validate the helper against single- and multi-architecture pipeline build construction.

Changes

Bundle build architecture

Layer / File(s)	Summary
Pin bundle builds to AMD64 pkg/steps/source.go, pkg/steps/bundle_source.go, pkg/steps/project_image.go	Adds shared AMD64 node-selector pinning and applies it to bundle source and bundle-image builds before execution.
Validate architecture guarantees pkg/steps/bundle_source_test.go	Tests default and multi-architecture build selection, pinned architecture coverage, and preservation of existing node selector entries.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: droslean, copilot

🚥 Pre-merge checks | ✅ 16 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Test Coverage For New Features	⚠️ Warning	The new test covers the helper’s assumptions, but no test executes the bundle-source or bundle-image run paths, so removing the new pinning calls would not fail.	Add regression tests that build those steps and assert the created Build has kubernetes.io/arch=amd64 in its node selector.

✅ Passed checks (16 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly states the main change: pinning bundle builds to amd64 nodes.
Linked Issues check	✅ Passed	The changes implement the issue fix by pinning src-bundle and bundle image builds to amd64 nodes and preserving existing selectors.
Out of Scope Changes check	✅ Passed	The patch stays focused on bundle-build node selection plus a supporting unit test, with no unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Go Error Handling	✅ Passed	PASS: The patch adds no new error-handling issues; touched code returns wrapped errors, no errors are ignored, and the new helper guards nil node selectors.
Stable And Deterministic Test Names	✅ Passed	No Ginkgo titles were added; the new subtest names are static and deterministic.
Test Structure And Quality	✅ Passed	PASS: New test is a pure unit test with focused subtests, no cluster resources, no Eventually calls, and assertions/messages match local patterns.
Microshift Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; the only new test is a plain unit test, so MicroShift compatibility is not implicated.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; the change only adds a Go unit test in pkg/steps, so the SNO multi-node check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	PASS: The change only adds an amd64 arch nodeSelector to build pods and merges it with existing selectors; it doesn't target control-plane/worker roles or add affinity/PDB constraints.
Ote Binary Stdout Contract	✅ Passed	Touched files only add node-selector logic and a unit test; no main/init/TestMain/BeforeSuite stdout writes were introduced.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The added test is a plain unit test, not Ginkgo e2e, and it contains no IPv4 assumptions or external connectivity.
No-Weak-Crypto	✅ Passed	No weak-crypto code was added: the PR only pins build node selectors; touched files import no crypto libs and the only bytes.Equal is a test fixture file compare.
Container-Privileges	✅ Passed	Only Go step logic/tests changed; no container/K8s manifests or privilege fields (privileged, hostPID/Network/IPC, SYS_ADMIN, allowPrivilegeEscalation) were added.
No-Sensitive-Data-In-Logs	✅ Passed	The PR only adds node-selector pinning and comments/tests; no new logging paths expose secrets, PII, or internal data.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

This PR addresses intermittent src-bundle / bundle image build failures on heterogeneous (amd64 + arm64) build clusters by ensuring single-arch bundle-related OpenShift Builds are scheduled onto amd64 nodes, avoiding “no image found in manifest list for architecture arm64” failures when inputs only contain amd64 images.

Changes:

• Pin bundleSourceStep.run builds to amd64 via kubernetes.io/arch node selector.
• Pin bundle image builds in projectDirectoryImageBuildStep.run (the IsBundleImage() path) to amd64 via the same selector.
• Update comments to document the manifest-list / scheduling failure mode that this resolves.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
pkg/steps/project_image.go	Pins bundle image builds to amd64 nodes to avoid arm64 scheduling failures when consuming amd64-only manifest entries.
pkg/steps/bundle_source.go	Pins src-bundle (bundle-source) builds to amd64 nodes to prevent buildah failures caused by landing on arm64 nodes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kaovilai
Once this PR has been reviewed and has the lgtm label, please assign smg247 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[kaovilai]

Note

Responses generated with Claude

Retriggering infra flakes (test pods Pending >1h, 0/52 nodes available) and the unrelated breaking-changes credential error:

/test checkconfig
/test integration
/test frontend-checks
/test breaking-changes

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e
/test e2e-oo

[kaovilai]

Note

Responses generated with Claude

CI status: checkconfig, integration, frontend-checks passed on retest (the first runs died to pods unschedulable for 1h — build-farm capacity).

The remaining failures — breaking-changes, e2e, e2e-oo — all fail with the same credential-resolution error (only the mount path differs):

failed to resolve credentials: invalid credentials for step <step>: invalid field name "" at mount_path <path>: mount file name "" decodes to an empty string

This is repo-wide, not from this PR: PR #5278's e2e runs today fail identically (mount_path /tmp/pull-secret), and the e2e job history shows the last green run anywhere was 2026-07-22 15:25 UTC — every PR's run since fails the same way. breaking-changes failed identically twice here, so retesting won't help until the credential config is fixed. Will retest these jobs once another PR's runs go green.

[kaovilai]

Note

Responses generated with Claude

Root cause of the e2e/e2e-oo/breaking-changes failures identified: the mount file name "" decodes to an empty string error string was introduced by #5317 (pkg/gsm-validation/mount_filename.go, merged 2026-07-22 11:58 UTC), and every ci-tools multi-stage job on every PR has failed credential resolution since. Reported to the author in #5317 (comment). Will retest these three jobs here once a fix or revert lands.

[kaovilai]

Note

Responses generated with Claude

#5331 merged and the fixed ci-operator image promoted — retesting the jobs broken by the credential validation regression:

/test breaking-changes
/test e2e
/test e2e-oo

[openshift-ci]

@kaovilai: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-oo	0d57bf8	link	false	/test e2e-oo

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[kaovilai]

Note

Responses generated with Claude

Post-#5331 retest results: breaking-changes ✅ and e2e ✅ (the credential regression is fixed).

e2e-oo still fails, but for a pre-existing reason unrelated to this PR: all four subtests die at config resolution (Ran for 0s) with

failed to generate integrated streams: failed to get integrated stream ocp/4.6: got unexpected http 400 status code from configresolver: not a valid integrated stream: ocp/4.6

— the e2e-oo fixture references the long-EOL ocp/4.6 integrated stream that configresolver no longer serves. Per job history, e2e-oo has failed on every run since at least February 2026, including #5072's runs which merged regardless. Happy to update the fixture in a follow-up if that's welcome.

[openshift-merge-bot]

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

[openshift-merge-bot]

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.