ERA-1706 Add safe Docker app rebuild workflow

[shrkz1]

What changed

• pin Community and Enterprise Compose services to an explicit immutable ERAMBA_IMAGE_TAG
• add the rebuild-app operator command with repository, current-deployment and canonical-plan preflight checks
• validate the exact target image, architecture, edition, distribution and application version
• protect the existing application volume using two identity checks before removal
• stage startup through eramba, cron and triggers with phase-aware recovery and diagnostics
• add operator documentation, design/specification and implementation plans
• add CI coverage for the full mocked lifecycle and failure matrix

Why

ERA-1706 replaces the manual Docker repository and application rebuild procedure with a repeatable workflow. The Docker checkout does not decide which release or tag should be installed. The running Eramba application requests a canonical image-switch plan from the support backend, and this command verifies that plan against the live deployment before making changes.

Rollout prerequisites

The workflow can be enabled only after all of the following are available:

1. The support backend migration for Docker release metadata is deployed.
2. The relevant support releases have approved immutable docker_image_tag values.
3. The currently running Eramba image contains bin/cake image_switch_plan.
4. The installation is running from this Docker repository with its normal .env and Compose files.
5. A current, recoverable database backup exists before a real rebuild.

Older installations whose running image does not contain image_switch_plan must use the manual image-switch procedure once to reach a compatible baseline. There is intentionally no operator option for overriding the target tag.

Usage

Run the command from the Docker repository root.

Preview a Community rebuild

./rebuild-app --dry-run

The edition is detected from the currently running eramba container. Community images must use ghcr.io/eramba/eramba:.

A dry run performs the complete preflight and target validation. It may pull the exact Community image, or load the supplied Enterprise archive, and it creates a temporary container to inspect the target VERSION. It does not update .env, stop or replace deployment containers, or remove deployment volumes.

If the support backend returns required=false, the command prints "nothing to do" and exits successfully.

Run an interactive Community rebuild

./rebuild-app

Before mutation, the command prints:

• current image and application version
• target image and application version
• the exact application volume that will be replaced
• the data, logs, database and trigger-storage volumes that will be preserved

The operator must type yes once to approve the plan and again to confirm that a current recoverable backup exists.

The backup confirmation is an operator assertion. The command does not create or validate a database backup.

Run an Enterprise rebuild

Download the approved Enterprise image archive before starting:

./rebuild-app \
  --edition enterprise \
  --image-file eramba-enterprise-<tag>-amd64.tar

The archive must be a readable local file. The command loads it with docker load and requires it to provide the exact repository and immutable tag returned by the support plan. An Enterprise plan must use the archive distribution.

Use --dry-run with the same arguments to validate the archive without mutating the deployment:

./rebuild-app \
  --edition enterprise \
  --image-file eramba-enterprise-<tag>-amd64.tar \
  --dry-run

Update the Docker checkout first

./rebuild-app --update-repo

This option requires a clean checkout on an attached branch. It runs:

git pull --ff-only

The command then re-executes itself under the same rebuild lock and continues with the original arguments. A dirty checkout, detached HEAD or non-fast-forward update aborts before Docker preflight.

The option can be combined with the normal Community or Enterprise arguments.

Non-interactive execution

./rebuild-app --yes --backup-confirmed

Both flags are required for a non-interactive real rebuild:

• --yes skips the interactive plan prompt.
• --backup-confirmed explicitly asserts that a current recoverable backup exists.

For Enterprise:

./rebuild-app \
  --edition enterprise \
  --image-file eramba-enterprise-<tag>-amd64.tar \
  --yes \
  --backup-confirmed

Using --yes without --backup-confirmed is rejected before deployment preflight.

What preflight verifies

Before changing deployment state, the command verifies:

1. No other rebuild-app process owns the installation lock.
2. The deployment .env exists and contains at most one ERAMBA_IMAGE_TAG.
3. The running eramba container exists and its image identifies Community or Enterprise unambiguously.
4. Any explicit --edition matches the running installation.
5. The current image ID and application VERSION can be read.
6. Docker Compose configuration renders successfully.
7. Current HTTP readiness, configuration validation, system health and cron migration status all pass.
8. The running Eramba command returns a valid canonical image-switch plan.
9. The plan source version, current tag, edition and distribution match the live deployment.
10. The target tag is immutable and is not latest.
11. The target image has the exact repository/tag, matches the Docker host architecture and contains the planned application VERSION.
12. The identities of the application and preserved volumes can be captured.

Community targets are pulled from the registry. Enterprise targets are loaded from the supplied archive. Both are acquired and validated before downtime.

Rebuild sequence

After confirmation, the command:

1. Creates a private run directory under .rebuild-app/runs//.
2. Atomically sets ERAMBA_IMAGE_TAG in .env while preserving file permissions and unrelated values.
3. Stops and removes only triggers_caddy, cron and eramba.
4. Re-creates an unstarted eramba container and verifies the application volume name and creation timestamp a second time.
5. Removes only the twice-verified volume mounted at /var/www/eramba.
6. Starts the target eramba service.
7. Waits for HTTP readiness, then verifies the running version, configuration and system health.
8. Starts cron and verifies migration status.
9. Starts triggers_caddy and verifies its health.
10. Verifies final image IDs and all preserved volume identities.
11. Writes the successful run state.

MySQL and Redis stay running throughout the normal rebuild. The database volume, application data volume, application logs volume and trigger-storage volume are preserved.

Failure and recovery behavior

Before the migration boundary

The migration boundary begins immediately before the first target eramba start, because its entrypoint may run database migrations.

If a failure happens before that boundary, the command attempts to:

1. restore the previous ERAMBA_IMAGE_TAG
2. start the previous eramba, cron and triggers_caddy services
3. record the run as recovered

If restoring the previous tag fails, application services are not started with an unknown tag.

At or after the migration boundary

The command does not automatically downgrade application code after the target eramba start has been attempted, because database migrations may already have run.

Instead it:

• stops any dependent service that has not been verified
• keeps already verified services running
• keeps the target image tag in .env
• captures private diagnostics under the run directory
• exits with a non-zero status for operator review

Recovery may require resolving the target failure or restoring the database backup before returning to the previous application version.

The diagnostics include run state, selected Docker inspection data and service logs. They do not copy the deployment .env.

Validation

• npx --yes bats@1.12.0 tests/rebuild-app-options.bats tests/rebuild-app-plan.bats tests/rebuild-app-lifecycle.bats — 52/52 passed
• Bash syntax checks passed
• Community and Enterprise Compose renders passed
• GitHub workflow YAML parsing passed
• git diff --check passed

No real or destructive Docker rebuild was executed during validation.