fix(controller): stop a data-plane outage from revoking issued credentials

[duckhawk]

Symptom

The e2e run on the current main fails system-durability → keeps every replica on its own volume and node across a full data-plane restart. Everything about the volumes is fine — same PVC → same PV → same node, pods back on their pinned nodes, the stored object still readable — and then the last assertion catches it:

[FAILED] a restart must not look like a recreate: the backend keys must still be there
Expected <string>: GK1e67171c989402cec63e48a3
to equal <string>: GK89d24858ab7875a1873da8eb

The access key issued to the caller is not the one it had before the restart.

Cause

The controller log, taken while the Garage pods were down, says it plainly — it was deleting the issued keys, retry after retry, for both accesses on the store:

Reconciler error err="delete access key: Post \"http://system-garage…:3903/v2/DeleteKey?id=GK89d24858ab7875a1873da8eb\": dial tcp …: connect: connection refused"
Reconciler error err="delete access key: Post \"http://system-garage…:3903/v2/DeleteKey?id=GKd1221fbd39536666c02ca893\": dial tcp …: connect: connection refused"

Not the delete path: both BucketAccess objects were alive and back to Ready afterwards. It is the gate in reconcileNormal, which treated three very different situations as one:

if !claimBound(claim) || bucket == nil || cluster == nil ||
   bucketReadyState(bucket) != True || !authorized {
        // revoke the backend key + delete the credentials Secret

A restarting store flips its Bucket out of Ready. That is a health signal, but the gate read it as "the claim is no longer usable" and revoked: backend key deleted, credentials Secret deleted, status.accessKeyID cleared. Once the store came back, EnsureAccess found no recorded key and minted a fresh one — so a caller's credentials were silently rotated because of an outage the controller only had to wait out. The bigger the store, the longer the window: every access on it is affected for as long as the data plane is down.

Fix

Revocation is driven by intent, not by health:

situation	before	now
bucket exists, not Ready (restart, backend unreachable, still coming up)	revoke key + delete Secret	wait: AccessGranted=False/InProgress, key and Secret untouched, requeue
claim no longer bound to a bucket	revoke	revoke (unchanged)
bound bucket / store gone	revoke	revoke (unchanged)
namespace no longer authorized by a BucketClaimPolicy	revoke	revoke (unchanged)

The deny-by-default enforcement is untouched: losing authorization still revokes on the spot, and it is still re-checked on every reconcile of a granted access.

Tests

bucket_access_revoke_test.go, three unit tests over a recording driver:

• an unready bucket must not call the backend at all, must keep status.accessKeyID / status.secretRef and the credentials Secret, and must report False/InProgress — this one fails on the old gate exactly the way the cluster did (the backend key was revoked 1 time(s), status.accessKeyID = "");
• a Shared bucket with no policy still revokes, with DeniedByPolicy;
• a claim that is no longer Bound still revokes.

The system-durability e2e spec is the end-to-end check — it is what caught this.

Base

Found while running the e2e suite for the Commander registry_mode fix and unrelated to it, so it was originally stacked on #40 to get both into one e2e run. #40 is merged now, and this is a single commit rebased straight onto main — no dependency left, nothing to merge first.

Also in here

Two fixes to the test side, both found on the same run and squashed into the same commit:

• the e2e pod dump could only say phase=Pending ready=false, so the run that lost its three-hour budget to two Pending Garage pods never printed the scheduler's own answer. It now carries the node and, for anything unplaced, the PodScheduled reason and message, plus a node dump with role, Ready, unschedulable and taints;
• a Garage unit test dereferenced a Secret it had only checked with Errorf, so a missing node identity panicked instead of reporting what went wrong — staticcheck flags it, which is why the Go linter was red on one runner and green on another for the same tree. Split out as Fatalf.