Skip to content

fix(controller): bound every reconcile so one stuck call cannot wedge a controller - #38

Merged
duckhawk merged 3 commits into
mainfrom
fix/reconcile-deadline
Jul 29, 2026
Merged

fix(controller): bound every reconcile so one stuck call cannot wedge a controller#38
duckhawk merged 3 commits into
mainfrom
fix/reconcile-deadline

Conversation

@duckhawk

@duckhawk duckhawk commented Jul 28, 2026

Copy link
Copy Markdown
Member

What the run showed

On the e2e run for #37 the controller went silent mid-suite:

17:48:21  [Reconcile] start for ObjectStore "system"     ← last line in the log
   …                                                       nothing, for 33 minutes
17:51:53  (the Full ObjectStore is created)               ← never reconciled at all
18:21:55  the Full spec gives up after 30 minutes

Requeues are queue-internal and cannot be dropped, so the worker was inside a call that never returned. With MaxConcurrentReconciles: 1 that is the entire controller — system, e2e-osc-light and the brand-new e2e-osc-full all queued behind it — and all eight full / full-highredundancy specs failed for want of a cluster nobody was building. (The other controllers had nothing to do in that window, which is why the log is completely quiet rather than partially.)

Nothing in the module noticed: a wedged worker emits no error, no event, no log.

Fix

A reconcile context carries no deadline of its own, and these reconcilers talk to backends over HTTP, exec into data-plane pods and wait on the API server. Every Reconcile now runs under a deadline — RECONCILE_TIMEOUT_SECONDS, default 300s — so a call that stops answering ends as an error naming the object and is retried, instead of taking the controller out of service indefinitely.

Honest limits

This does not say which call hung; that evidence was not in the logs, and I am not going to guess at a culprit. What it does is make the next occurrence self-reporting: the deadline turns silence into a logged failure against a named object, and the requeue keeps the rest of the module moving.

The e2e diagnostics also print the leader lease now (holder + renew age). When a controller goes quiet, that is what separates "the leader is stuck" from "leadership moved to another pod" — the log alone cannot tell them apart, and this run cost a full diagnosis cycle to that ambiguity.

Stacked

Branched on #37 (which is branched on #36), so a single e2e run exercises the SeaweedFS IAM guard, the Heavy teardown and this bound together. Merge order: #36#37 → this.

Also in this commit

Two things the same diagnosis pulled in:

  • reconcileDelete returns only an error. Bounding the reconciles made the linter's point visible: the delete paths of the ObjectStore, Bucket and BucketAccess reconcilers always returned an empty ctrl.Result, so the value carried no information. The caller supplies the empty Result now. BucketClaim keeps its Result — its delete path really does requeue while a BucketAccess still references the claim.
  • The e2e diagnostics dump every store and bucket, not just the shared ones. When a profile spec's own store failed, the dump was silent about it: a Full store stuck below Ready printed the healthy system store's conditions and nothing about the object that actually failed, so its BackendReady message — the one sentence that says why — was never in the log. Both kinds are cluster-scoped and few, so all of them are listed with phase, conditions, reasons and messages. This is what finally surfaced the SeaweedFS diagnosis in fix(seaweedfs): manage S3 IAM as per-identity files over the filer gRPC API #36.

@duckhawk duckhawk added e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR and removed e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR labels Jul 28, 2026
@duckhawk
duckhawk force-pushed the fix/heavy-teardown-strands-cephobjectstore branch from ec1e6d8 to a83de7c Compare July 28, 2026 07:21
@duckhawk
duckhawk force-pushed the fix/reconcile-deadline branch from 6800aa1 to dab424e Compare July 28, 2026 07:22
@duckhawk duckhawk added e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR and removed e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR labels Jul 28, 2026
@duckhawk
duckhawk force-pushed the fix/heavy-teardown-strands-cephobjectstore branch from a83de7c to b3b4d22 Compare July 28, 2026 10:55
@duckhawk
duckhawk force-pushed the fix/reconcile-deadline branch from dab424e to a6ebdbe Compare July 28, 2026 10:55
@duckhawk duckhawk added e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR and removed e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR labels Jul 28, 2026
@duckhawk
duckhawk force-pushed the fix/heavy-teardown-strands-cephobjectstore branch from b3b4d22 to c6ddb0f Compare July 28, 2026 15:25
@duckhawk
duckhawk force-pushed the fix/reconcile-deadline branch from a6ebdbe to d8477cb Compare July 28, 2026 15:25
@duckhawk
duckhawk force-pushed the fix/heavy-teardown-strands-cephobjectstore branch from c6ddb0f to 3b20a76 Compare July 28, 2026 15:29
@duckhawk
duckhawk force-pushed the fix/reconcile-deadline branch from d8477cb to 6f26803 Compare July 28, 2026 15:29
@duckhawk duckhawk added e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR and removed e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR labels Jul 28, 2026
duckhawk added 3 commits July 29, 2026 05:42
…PC API

The Full profile could not hold S3 credentials. The e2e runs kept failing with "The
access key ID you provided does not exist in our records" for a key the credentials
Secret still advertised, gone from the backend with no pod restart to explain it —
and once that was addressed the store stopped reaching Ready at all, sitting on
"IAM config did not persist as written (found=false, 0 of 1 identities present)":
the driver's own write was invisible to the driver's own read.

Reproducing against a stock SeaweedFS 4.39 `weed server -filer -s3` unravelled the
chain, and it invalidates the shared-file design this driver had, not just its
transport:

  - the S3 gateway loads IAM through filer.ReadInsideFiler, which returns only the
    entry's inline Content and never reads file chunks. Of the filer's HTTP upload
    paths, multipart POST (what the driver used) stores inline content but leaves
    FileSize at 0 — the gateway sees the identities, but an HTTP GET serves an empty
    body, so every read-modify-write started from "empty" and clobbered the file;
    raw PUT round-trips over HTTP but stores chunks, which the gateway cannot see at
    all, so authentication silently stays anonymous;
  - on 4.39 the single /etc/iam/identity.json is legacy input, not state: the first
    gateway to load it MIGRATES every identity into per-file form under
    /etc/iam/identities/ and renames the original away. Managing that file races the
    migration, and revocation through it is broken outright — the identity removed
    from the legacy file has already been migrated, and its per-file copy keeps
    working.

The driver now manages per-identity files the way the gateway's own credential
manager writes them: one <name>.json per identity under /etc/iam/identities/, over
the filer gRPC API, payload in the entry's inline content, snake_case JSON keys
matching the generated iam_pb.Identity tags the gateway decodes with. That removes
the shared-file hazard structurally — a reconcile can only ever touch the identity
it owns — so the fail-closed read-modify-write machinery, the admin-Secret bootstrap
annotation and the per-cluster identity lock are gone rather than patched again.

What remains guarded: strict decoding, so a non-identity payload is an error and
never an empty identity to overwrite; write verification by read-back, because an
accepted write is not proof the credential is readable; and revocation that cannot
claim success without confirming the file is gone.

Reaching the filer needed one more fix. The main ClusterIP Service — the one every
endpoint helper resolves to — published only s3 and the filer HTTP port, so gRPC
dials to 18888 were blackholed rather than refused and surfaced as "dial tcp
10.223.237.10:18888: i/o timeout" a dial deadline later. This predates the IAM work:
quota.go sets the per-bucket quota over the same target, so bucket quotas on a Full
store could never have worked either. The regression test derives the ports from the
endpoint helpers instead of restating the Service's list, so a helper that starts
dialing a new port fails until the Service publishes it.

Validated against stock 4.39: the live lifecycle test (write admin, write an app
identity, rotate, assert the neighbour is untouched, revoke, repeat the delete)
passes — kept in the tree behind LIVE_FILER_GRPC, skipped without it — and the
gateway enforces the result: unsigned request 403, issued key 200, revoked key
InvalidAccessKeyId. The e2e specs that caught this (full round-trip,
full-highredundancy filer failover) now pass end to end.

The in-process protocol fakes are gone in favour of driver-level behaviour tests
over an identityFiles stub: a protocol fake here validated our expectations of the
filer rather than the filer itself, and hid all of the above through several e2e
cycles.

Not addressed here: the same Full round-trip also failed once with "Unable to write
to one or more targets ... internal error" before any credential was lost, which
points at the volume servers rather than IAM and needs its own investigation.

Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
…ding it

A Heavy store deleted with the default Retain policy left its CephObjectStore
behind on purpose, to keep the RGW pools. The e2e runs show what that costs: the
ElasticCluster never finishes terminating (its finalizer waits on volumes the
leftover keeps in use), and the suite's Heavy teardown spends its full 20-minute
budget waiting before giving up — about a third of the whole run, every run, ending
with a cluster stuck in Terminating.

Nothing else removes that CR either. Its ownerReference is no help: sds-elastic's
validating webhook rejects requests to vendored Rook resources from anyone it does
not know — the garbage collector included — which is also why the suite's own
workaround of deleting it directly was denied ("Direct modifications to Rook Ceph
resources are not allowed").

Keeping the CR was never what protected the data: preservePoolsOnDelete is, and it
is already set from the reclaim policy when the CR is rendered (Retain -> true). So
the driver now deletes the CephObjectStore under both policies, and Retain still
preserves the pools and every object in them.

Tests: the driver removes the store under both policies and tolerates an absent
one, and preservePoolsOnDelete still follows the reclaim policy — that field is now
the only thing standing between Retain and the destruction of every bucket's data,
so it is pinned explicitly. The e2e delete spec asserts the CR is gone after the
ObjectStore, and the Heavy cleanup drops its impossible direct-delete step for a
diagnostic that names the leftover if one ever appears again.

The CRD (both languages) and DESIGN documented the old contract and now describe
this one.

Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
… a controller

An e2e run shows the failure mode plainly: the ObjectStore worker entered a
reconcile at 17:48:21 and the controller logged nothing ever again — no reconcile
start for any store, for the next 33 minutes. Requeues are queue-internal and cannot
be dropped, so the worker was inside a call that never returned. With
MaxConcurrentReconciles at 1 that is the whole controller: the Full store created six
minutes later was never reconciled at all, and its eight specs failed for want of a
cluster that nobody was building. Nothing in the module noticed, because a wedged
worker produces no error, no event and no log.

A reconcile context carries no deadline of its own, and these reconcilers talk to
backends over HTTP, exec into data-plane pods and wait on the API server. Each
Reconcile now runs under a configurable deadline (RECONCILE_TIMEOUT_SECONDS, default
300s), so a call that stops answering ends as an error naming the object and is
retried, instead of silently taking the controller out of service.

This does not identify which call hung — that evidence was not in the logs. It makes
the next occurrence self-reporting, and the e2e diagnostics now also print the leader
lease (holder + renew age), which is what separates "the leader stopped working" from
"leadership moved" when a controller goes quiet.

Bounding the reconciles made the linter's point visible: the delete paths of the
ObjectStore, Bucket and BucketAccess reconcilers always returned an empty
ctrl.Result, so the value carried no information. They return an error now and the
caller supplies the empty Result. BucketClaim keeps its Result — its delete path
really does requeue while a BucketAccess still references the claim.

The e2e diagnostics also stop naming only the primary ObjectStore and Bucket. When a
profile spec's own store failed the dump was silent about it: a Full store stuck
below Ready printed the healthy `system` store's conditions and nothing about the
object that failed, so its BackendReady message — the one sentence that says why —
was never in the log, and reading around that gap cost a diagnosis cycle. Both kinds
are cluster-scoped and few, so the dump lists all of them with phase, conditions,
reasons and messages.

Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
@duckhawk
duckhawk force-pushed the fix/reconcile-deadline branch from 6f26803 to e5b9a74 Compare July 29, 2026 02:44
@duckhawk
duckhawk force-pushed the fix/heavy-teardown-strands-cephobjectstore branch from 3b20a76 to 86c3f55 Compare July 29, 2026 02:44
Base automatically changed from fix/heavy-teardown-strands-cephobjectstore to main July 29, 2026 03:18
@duckhawk
duckhawk merged commit e3d2ff0 into main Jul 29, 2026
21 of 25 checks passed
@duckhawk
duckhawk deleted the fix/reconcile-deadline branch July 29, 2026 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant