Refactor offline image management to use skopeo and OCI archives

[Kibakus]

/kind feature

What this PR does / why we need it:

This PR refactors the offline image and file management scripts to improve portability, reliability, and cross-platform support.

Main changes include:

• Replace docker/podman/nerdctl image operations with skopeo and OCI archives to preserve image metadata and manifest lists during offline transfers.
• Add architecture detection (amd64/arm64) and Linux platform selection to improve cross-platform compatibility.
• Use localhost instead of the system hostname when configuring and accessing the local registry.
• Improve error handling by enabling set -euo pipefail.
• Simplify runtime-specific logic in the offline scripts.
• Simplify nginx.conf and related configuration files used by the offline file server.

The previous offline image workflow relied on docker save and docker load, which do not preserve OCI manifest lists and platform descriptors associated with multi-architecture images. As a result, some images could not be pushed to the destination registry after being transferred through the offline workflow.

Using OCI archives with skopeo preserves manifest lists and platform metadata throughout the transfer process, allowing both single-platform and multi-platform images to be transferred and registered consistently.

In addition, the previous nginx configuration loaded default server definitions from the container image through conf.d and default.d include directives. Depending on the image contents, this could result in multiple server blocks being loaded simultaneously, causing unexpected behavior and configuration conflicts. The configuration has been simplified to ensure that only the intended offline file server configuration is used.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

During validation, issues were observed with Kubernetes images distributed as OCI manifest lists.

With the previous implementation, the following scenario could be reproduced:

Loaded image: registry.k8s.io/kube-apiserver:v1.33.11
The push refers to repository [localhost:5000/registry.k8s.io/kube-apiserver]
image with reference localhost:5000/registry.k8s.io/kube-apiserver:v1.33.11 was found but does not provide any platform

The issue occurs because docker save/docker load lose the OCI manifest list information required to determine which platform-specific manifest should be pushed to the destination registry.

The OCI-based workflow implemented in this PR preserves manifest lists and platform metadata, eliminating these failures and providing consistent behavior across supported environments.

The changes were validated using both single-platform images and multi-architecture images represented by OCI manifest lists.

Does this PR introduce a user-facing change?:

Improved offline image and file management workflows by switching image handling to skopeo with OCI archives, preserving multi-architecture image metadata, adding cross-platform architecture support, using localhost for local registry configuration, simplifying nginx configuration, and improving offline runtime reliability.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Kibakus
Once this PR has been reviewed and has the lgtm label, please assign vannten for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @Kibakus. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[yankay]

/ok-to-test

[Kibakus]

@yankay

https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/jobs/14898252406

Uploading cache.zip to http://minio.minio.svc.cluster.local:9000/runner-cache/project/9659330/pre-commit-2-4-protected
FATAL: received: 503 Service Unavailable. Request failed with code: SlowDownWrite, message: Resource requested is unwritable, please reduce your request rate

[Kibakus]

The failing CI jobs appear unrelated to this change. Waiting for review.

[yankay]

HI @Kibakus

Thanks for checking. The MinIO 503 happened after the job had already failed. The actual failure is end-of-file-fixer: it added missing trailing newlines to the two offline scripts.

Could you please run pre-commit run --all-files locally and push the fix?

[Kibakus]

Thanks! fixed!

[Copilot]

Pull request overview

This PR refactors the contrib/offline tooling to improve offline portability and multi-arch image handling by switching image transfer logic to skopeo with OCI artifacts, and simplifying local registry / nginx configuration.

Changes:

• Switch offline container image collection/registration to skopeo copy with oci-archive to preserve manifest lists and platform metadata.
• Add basic OS/arch detection and use localhost:5000 for local registry configuration.
• Harden scripts with set -euo pipefail and simplify nginx configuration by removing default includes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
contrib/offline/registries.conf	Update insecure registry entry to localhost:5000.
contrib/offline/nginx.conf	Remove includes that could unintentionally load extra server blocks.
contrib/offline/manage-offline-files.sh	Refactor into create/serve modes; add skopeo-based nginx image handling and stricter bash settings.
contrib/offline/manage-offline-container-images.sh	Replace runtime pull/save/push flow with skopeo OCI archive workflow; add platform detection and stricter bash settings.
contrib/offline/docker-daemon.json	Update insecure registry entry to localhost:5000.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Kibakus / name: Denis (3406420)
• ✅ login: Kibakus / name: Kibakus (09f0370, 6c2763f, d492f1e)

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Kibakus
Once this PR has been reviewed and has the lgtm label, please assign vannten for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Kibakus]

Hi @yankay @ErikJiang , just a gentle ping on this when you have a chance.

[Kibakus]

Hi @yankay,

I wanted to follow up on this PR. It's been waiting for about a week now.

The Kubespray CI Pipeline check is still showing "Waiting for status to be reported", while the other checks have already completed.

Is there anything I should do on my side to trigger it again, or is this something that needs attention from a maintainer?

[yankay]

HI @Kibakus

Thanks for following up, and thanks for the work on this PR.

I think we should discuss the compatibility impact before moving forward. This PR introduces a fairly large refactor of the offline workflow, and one breaking change is switching the image handling path from the existing container runtimes (docker/podman/nerdctl) to skopeo.

That means users would need to install skopeo themselves, including in offline or restricted environments. In practice, offline installation of skopeo can be difficult because it has many package dependencies, and the upstream podman-container-tools/skopeo project does not provide standalone binary releases. See:

• https://github.com/podman-container-tools/skopeo/blob/main/install.md
• Consider distributing statically built binaries as part of release podman-container-tools/skopeo#391

Docker can have similar installation concerns, but it is much more widely used and users generally have more existing experience installing and operating it.

So my main concern is that this is a big user-facing change: target machines may not have skopeo, and requiring it could make offline deployment harder for some users. Could we keep the existing runtime-based path as a fallback, make skopeo optional, or split this refactor into smaller PRs?

[ErikJiang]

nit: The nginx:alpine image is downloaded during create , but only used in serve .
For users who only need offline files without running an nginx server, this adds unnecessary download and package bloat.
Suggest moving the nginx image download into serve_offline_files() where it actually belongs.

[Kibakus]

@yankay Check pls, good?

[Kibakus]

@yankay
Сan you look please?

[Kibakus]

@yankay
Сan you look please?

[ErikJiang]

@Kibakus good job !

/lgtm

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 9 comments.

[Kibakus]

Quick question: do you think I should go through the new Copilot recommendations? As far as I remember, I already addressed the earlier comments, and the PR has already received /lgtm. Or do you think it's worth reviewing the latest suggestions as well?

[Kibakus]

@ErikJiang, @yankay
Сan you look please?

[yankay]

HI @Kibakus
Thanks for the follow-up. I think this still needs a few fixes before approval.

The main blocker is that the shared registry config change can break the existing offline workflow. docker-daemon.json and registries.conf now hard-code localhost:5000, but the existing manage-offline-container-images.sh register still defaults DESTINATION_REGISTRY to $(hostname):5000 and previously relied on replacing the HOSTNAME placeholder. With this PR, Docker/Podman can be configured for localhost:5000 while the push still goes to $(hostname):5000.

There is also an offline correctness issue with the skopeo-created docker archives. Both registry:latest and nginx:alpine are copied as docker-archive:<path> without an explicit reference. That can load as an untagged image, so runtime run registry:latest or runtime run nginx:alpine may try to pull from the network and fail in an offline environment. Please include the reference in the archive destination, for example docker-archive:${IMAGE_DIR}/registry-latest.tar:registry:latest and docker-archive:${NGINX_IMAGE_ARCHIVE}:nginx:alpine.

A few smaller things:

• The PR says it preserves multi-architecture metadata, but the default path uses --override-os/--override-arch, so it saves one platform unless the first copy fails.
• The README still shows manage-offline-files.sh create, but that command does not exist in the legacy script.
• The new skopeo container-image script requires docker/podman/nerdctl even for create or pushing to an existing registry, where skopeo alone should be enough.

Once the two blockers are fixed, this should be much closer.

I also think the High and Medium findings from the Copilot review are worth checking before approval, especially the legacy workflow regression and the README/script behavior mismatch.

[kubernetes-prow]

New changes are detected. LGTM label has been removed.

[Kibakus]

@yankay
Thanks for the detailed review! I've addressed all of the issues you pointed out.

Blocker 1 - Legacy workflow regression (docker-daemon.json, registries.conf)

The shared config files were previously hard-coded to localhost:5000, which broke the existing manage-offline-container-images.sh register workflow because it relies on:

sed -i "s@HOSTNAME@$(hostname)@"

to substitute the actual hostname.

Both configuration files now include HOSTNAME:5000 as a placeholder (alongside localhost:5000), so the legacy register workflow continues to inject the real hostname while Docker/Podman still treats localhost:5000 as an insecure registry.

For consistency, the new skopeo-based register workflow now performs the same HOSTNAME → $(hostname) replacement.

Blocker 2 - Untagged Docker archives

The Docker archive destinations now include explicit image references, ensuring images are loaded with the correct tags instead of becoming untagged:

• manage-offline-skopeo-container-images.sh — docker-archive:...:registry:latest
• manage-offline-skopeo-files.sh — docker-archive:...:nginx:alpine

Additional fixes

• Removed the invalid manage-offline-files.sh create example from the README (the legacy script does not provide a create subcommand).
• manage-offline-skopeo-container-images.sh no longer requires docker/podman/nerdctl during startup. Runtime detection is now only performed in the register path where it is actually required, so create works with skopeo alone.

The changes are included in:

• contrib/offline/docker-daemon.json
• contrib/offline/registries.conf
• contrib/offline/manage-offline-skopeo-container-images.sh
• contrib/offline/manage-offline-skopeo-files.sh
• contrib/offline/README.md

Please let me know if you see anything else that should be adjusted. Thanks again for the review!

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

[Kibakus]

@yankay
Сan you look please?

[Kibakus]

@yankay
Сan you look please?

[Kibakus]

@yankay
Сan you look please?