fix(controller): bound every reconcile so one stuck call cannot wedge a controller - #38
Merged
Conversation
duckhawk
force-pushed
the
fix/heavy-teardown-strands-cephobjectstore
branch
from
July 28, 2026 07:21
ec1e6d8 to
a83de7c
Compare
duckhawk
force-pushed
the
fix/reconcile-deadline
branch
from
July 28, 2026 07:22
6800aa1 to
dab424e
Compare
duckhawk
force-pushed
the
fix/heavy-teardown-strands-cephobjectstore
branch
from
July 28, 2026 10:55
a83de7c to
b3b4d22
Compare
duckhawk
force-pushed
the
fix/reconcile-deadline
branch
from
July 28, 2026 10:55
dab424e to
a6ebdbe
Compare
duckhawk
force-pushed
the
fix/heavy-teardown-strands-cephobjectstore
branch
from
July 28, 2026 15:25
b3b4d22 to
c6ddb0f
Compare
duckhawk
force-pushed
the
fix/reconcile-deadline
branch
from
July 28, 2026 15:25
a6ebdbe to
d8477cb
Compare
duckhawk
force-pushed
the
fix/heavy-teardown-strands-cephobjectstore
branch
from
July 28, 2026 15:29
c6ddb0f to
3b20a76
Compare
duckhawk
force-pushed
the
fix/reconcile-deadline
branch
from
July 28, 2026 15:29
d8477cb to
6f26803
Compare
…PC API
The Full profile could not hold S3 credentials. The e2e runs kept failing with "The
access key ID you provided does not exist in our records" for a key the credentials
Secret still advertised, gone from the backend with no pod restart to explain it —
and once that was addressed the store stopped reaching Ready at all, sitting on
"IAM config did not persist as written (found=false, 0 of 1 identities present)":
the driver's own write was invisible to the driver's own read.
Reproducing against a stock SeaweedFS 4.39 `weed server -filer -s3` unravelled the
chain, and it invalidates the shared-file design this driver had, not just its
transport:
- the S3 gateway loads IAM through filer.ReadInsideFiler, which returns only the
entry's inline Content and never reads file chunks. Of the filer's HTTP upload
paths, multipart POST (what the driver used) stores inline content but leaves
FileSize at 0 — the gateway sees the identities, but an HTTP GET serves an empty
body, so every read-modify-write started from "empty" and clobbered the file;
raw PUT round-trips over HTTP but stores chunks, which the gateway cannot see at
all, so authentication silently stays anonymous;
- on 4.39 the single /etc/iam/identity.json is legacy input, not state: the first
gateway to load it MIGRATES every identity into per-file form under
/etc/iam/identities/ and renames the original away. Managing that file races the
migration, and revocation through it is broken outright — the identity removed
from the legacy file has already been migrated, and its per-file copy keeps
working.
The driver now manages per-identity files the way the gateway's own credential
manager writes them: one <name>.json per identity under /etc/iam/identities/, over
the filer gRPC API, payload in the entry's inline content, snake_case JSON keys
matching the generated iam_pb.Identity tags the gateway decodes with. That removes
the shared-file hazard structurally — a reconcile can only ever touch the identity
it owns — so the fail-closed read-modify-write machinery, the admin-Secret bootstrap
annotation and the per-cluster identity lock are gone rather than patched again.
What remains guarded: strict decoding, so a non-identity payload is an error and
never an empty identity to overwrite; write verification by read-back, because an
accepted write is not proof the credential is readable; and revocation that cannot
claim success without confirming the file is gone.
Reaching the filer needed one more fix. The main ClusterIP Service — the one every
endpoint helper resolves to — published only s3 and the filer HTTP port, so gRPC
dials to 18888 were blackholed rather than refused and surfaced as "dial tcp
10.223.237.10:18888: i/o timeout" a dial deadline later. This predates the IAM work:
quota.go sets the per-bucket quota over the same target, so bucket quotas on a Full
store could never have worked either. The regression test derives the ports from the
endpoint helpers instead of restating the Service's list, so a helper that starts
dialing a new port fails until the Service publishes it.
Validated against stock 4.39: the live lifecycle test (write admin, write an app
identity, rotate, assert the neighbour is untouched, revoke, repeat the delete)
passes — kept in the tree behind LIVE_FILER_GRPC, skipped without it — and the
gateway enforces the result: unsigned request 403, issued key 200, revoked key
InvalidAccessKeyId. The e2e specs that caught this (full round-trip,
full-highredundancy filer failover) now pass end to end.
The in-process protocol fakes are gone in favour of driver-level behaviour tests
over an identityFiles stub: a protocol fake here validated our expectations of the
filer rather than the filer itself, and hid all of the above through several e2e
cycles.
Not addressed here: the same Full round-trip also failed once with "Unable to write
to one or more targets ... internal error" before any credential was lost, which
points at the volume servers rather than IAM and needs its own investigation.
Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
…ding it
A Heavy store deleted with the default Retain policy left its CephObjectStore
behind on purpose, to keep the RGW pools. The e2e runs show what that costs: the
ElasticCluster never finishes terminating (its finalizer waits on volumes the
leftover keeps in use), and the suite's Heavy teardown spends its full 20-minute
budget waiting before giving up — about a third of the whole run, every run, ending
with a cluster stuck in Terminating.
Nothing else removes that CR either. Its ownerReference is no help: sds-elastic's
validating webhook rejects requests to vendored Rook resources from anyone it does
not know — the garbage collector included — which is also why the suite's own
workaround of deleting it directly was denied ("Direct modifications to Rook Ceph
resources are not allowed").
Keeping the CR was never what protected the data: preservePoolsOnDelete is, and it
is already set from the reclaim policy when the CR is rendered (Retain -> true). So
the driver now deletes the CephObjectStore under both policies, and Retain still
preserves the pools and every object in them.
Tests: the driver removes the store under both policies and tolerates an absent
one, and preservePoolsOnDelete still follows the reclaim policy — that field is now
the only thing standing between Retain and the destruction of every bucket's data,
so it is pinned explicitly. The e2e delete spec asserts the CR is gone after the
ObjectStore, and the Heavy cleanup drops its impossible direct-delete step for a
diagnostic that names the leftover if one ever appears again.
The CRD (both languages) and DESIGN documented the old contract and now describe
this one.
Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
… a controller An e2e run shows the failure mode plainly: the ObjectStore worker entered a reconcile at 17:48:21 and the controller logged nothing ever again — no reconcile start for any store, for the next 33 minutes. Requeues are queue-internal and cannot be dropped, so the worker was inside a call that never returned. With MaxConcurrentReconciles at 1 that is the whole controller: the Full store created six minutes later was never reconciled at all, and its eight specs failed for want of a cluster that nobody was building. Nothing in the module noticed, because a wedged worker produces no error, no event and no log. A reconcile context carries no deadline of its own, and these reconcilers talk to backends over HTTP, exec into data-plane pods and wait on the API server. Each Reconcile now runs under a configurable deadline (RECONCILE_TIMEOUT_SECONDS, default 300s), so a call that stops answering ends as an error naming the object and is retried, instead of silently taking the controller out of service. This does not identify which call hung — that evidence was not in the logs. It makes the next occurrence self-reporting, and the e2e diagnostics now also print the leader lease (holder + renew age), which is what separates "the leader stopped working" from "leadership moved" when a controller goes quiet. Bounding the reconciles made the linter's point visible: the delete paths of the ObjectStore, Bucket and BucketAccess reconcilers always returned an empty ctrl.Result, so the value carried no information. They return an error now and the caller supplies the empty Result. BucketClaim keeps its Result — its delete path really does requeue while a BucketAccess still references the claim. The e2e diagnostics also stop naming only the primary ObjectStore and Bucket. When a profile spec's own store failed the dump was silent about it: a Full store stuck below Ready printed the healthy `system` store's conditions and nothing about the object that failed, so its BackendReady message — the one sentence that says why — was never in the log, and reading around that gap cost a diagnosis cycle. Both kinds are cluster-scoped and few, so the dump lists all of them with phase, conditions, reasons and messages. Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
duckhawk
force-pushed
the
fix/reconcile-deadline
branch
from
July 29, 2026 02:44
6f26803 to
e5b9a74
Compare
duckhawk
force-pushed
the
fix/heavy-teardown-strands-cephobjectstore
branch
from
July 29, 2026 02:44
3b20a76 to
86c3f55
Compare
Base automatically changed from
fix/heavy-teardown-strands-cephobjectstore
to
main
July 29, 2026 03:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the run showed
On the e2e run for #37 the controller went silent mid-suite:
Requeues are queue-internal and cannot be dropped, so the worker was inside a call that never returned. With
MaxConcurrentReconciles: 1that is the entire controller —system,e2e-osc-lightand the brand-newe2e-osc-fullall queued behind it — and all eightfull/full-highredundancyspecs failed for want of a cluster nobody was building. (The other controllers had nothing to do in that window, which is why the log is completely quiet rather than partially.)Nothing in the module noticed: a wedged worker emits no error, no event, no log.
Fix
A reconcile context carries no deadline of its own, and these reconcilers talk to backends over HTTP, exec into data-plane pods and wait on the API server. Every
Reconcilenow runs under a deadline —RECONCILE_TIMEOUT_SECONDS, default 300s — so a call that stops answering ends as an error naming the object and is retried, instead of taking the controller out of service indefinitely.Honest limits
This does not say which call hung; that evidence was not in the logs, and I am not going to guess at a culprit. What it does is make the next occurrence self-reporting: the deadline turns silence into a logged failure against a named object, and the requeue keeps the rest of the module moving.
The e2e diagnostics also print the leader lease now (holder + renew age). When a controller goes quiet, that is what separates "the leader is stuck" from "leadership moved to another pod" — the log alone cannot tell them apart, and this run cost a full diagnosis cycle to that ambiguity.
Stacked
Branched on #37 (which is branched on #36), so a single e2e run exercises the SeaweedFS IAM guard, the Heavy teardown and this bound together. Merge order: #36 → #37 → this.
Also in this commit
Two things the same diagnosis pulled in:
reconcileDeletereturns only an error. Bounding the reconciles made the linter's point visible: the delete paths of the ObjectStore, Bucket and BucketAccess reconcilers always returned an emptyctrl.Result, so the value carried no information. The caller supplies the empty Result now. BucketClaim keeps its Result — its delete path really does requeue while a BucketAccess still references the claim.systemstore's conditions and nothing about the object that actually failed, so itsBackendReadymessage — the one sentence that says why — was never in the log. Both kinds are cluster-scoped and few, so all of them are listed with phase, conditions, reasons and messages. This is what finally surfaced the SeaweedFS diagnosis in fix(seaweedfs): manage S3 IAM as per-identity files over the filer gRPC API #36.