fix(seaweedfs): manage S3 IAM as per-identity files over the filer gRPC API

[duckhawk]

Symptom

The Full profile (SeaweedFS) could not hold S3 credentials. Two e2e runs failed with mc: <ERROR> ... The access key ID you provided does not exist in our records for a key the credentials Secret still advertised, with no pod restart to explain the loss. Fixing that exposed the next layer: the store then never reached Ready at all, sitting on BackendReady=False msg="configuring S3 admin identity: IAM config did not persist as written (found=false, 0 of 1 identities present)" — the driver's own write was invisible to the driver's own read.

Cause

Reproduced against a stock SeaweedFS 4.39 weed server -filer -s3, which unravelled the whole chain — and invalidated the shared-file design this driver had, not just its transport.

1. The gateway and the filer's HTTP API disagree about where file content lives. The S3 gateway loads IAM through filer.ReadInsideFiler, which returns only the entry's inline Content and never reads file chunks. Of the filer's upload paths:

• multipart POST (the original code) stores inline content but leaves FileSize at 0 — the gateway sees the identities, but an HTTP GET serves an empty body. So the driver's read-modify-write always started from "empty" and clobbered the file;
• raw PUT round-trips correctly over HTTP but stores the payload as chunks — which the gateway cannot see at all, so authentication silently stays anonymous.

2. On 4.39 the single /etc/iam/identity.json is legacy input, not state. The first gateway to load it migrates every identity into per-file form under /etc/iam/identities/ and renames the original away. Any design that keeps read-modify-writing the legacy file races that migration — the verified write from this branch's first commits was deleted from under the verifier — and revocation through it is broken outright: the identity removed from the legacy file has already been migrated, and its per-file copy keeps working.

3. The filer gRPC port was not reachable. With the rewrite in place the Full store stalled on dial tcp 10.223.237.10:18888: i/o timeout. The filer pods listen on 18888 and the headless Service publishes it, but the main ClusterIP Service — the one every endpoint helper resolves to — declared only s3 and the filer HTTP port, so those dials were blackholed rather than refused. This one predates the IAM work: quota.go sets the per-bucket quota over the same target, so bucket quotas on a Full store could never have worked either.

Fix

The driver now manages per-identity files the way the gateway's own credential manager writes them: one <name>.json per identity under /etc/iam/identities/, over the filer gRPC API, payload in the entry's inline content, snake_case JSON keys matching the generated iam_pb.Identity tags the gateway decodes with.

That removes the shared-file hazard structurally — a reconcile can only ever touch the identity it owns — so the fail-closed mutateIdentities machinery, the admin-Secret bootstrap annotation and the per-cluster identity lock are gone rather than patched again.

What remains guarded:

• strict decoding — a non-identity payload is an error, never an empty identity to overwrite;
• write verification by read-back — an accepted write is not proof the credential is readable;
• revocation that cannot claim success without confirming the file is gone.

And the main Service now publishes the filer gRPC port, fixing both IAM and bucket quotas.

Validation

Against a stock SeaweedFS 4.39:

• the live lifecycle test (write admin, write app identity, rotate, assert the neighbour is untouched, revoke, repeat the delete) passes — kept in the tree behind LIVE_FILER_GRPC, skipped without it;
• the gateway enforces the result: unsigned request 403, issued key 200, revoked key InvalidAccessKeyId.

Tests

The in-process protocol fakes are gone in favour of driver-level behaviour tests over an identityFiles stub: a protocol fake here validated our expectations of the filer rather than the filer itself, and hid all of the above through several e2e cycles. Covered: an access touching only its own identity file, reuse vs. rotation, deletion removing only its own file, and the on-disk JSON matching the gateway's schema.

The Service regression test derives the ports from the endpoint helpers instead of restating the Service's list, so a helper that starts dialing a new port fails until the Service publishes it.

The full and full-highredundancy e2e specs are the end-to-end check.

Not in scope

The same full round-trip also failed once with Unable to write to one or more targets ... internal error before any credential was lost. That points at the volume servers rather than IAM and needs its own investigation.