fix(cephrgw): remove the CephObjectStore on teardown instead of stranding it - #37
Open
duckhawk wants to merge 1 commit into
Open
Conversation
duckhawk
force-pushed
the
fix/heavy-teardown-strands-cephobjectstore
branch
from
July 27, 2026 16:58
fa4bcb4 to
ec1e6d8
Compare
duckhawk
changed the base branch from
main
to
fix/seaweedfs-credential-propagation
July 27, 2026 16:58
duckhawk
force-pushed
the
fix/heavy-teardown-strands-cephobjectstore
branch
from
July 28, 2026 07:21
ec1e6d8 to
a83de7c
Compare
…ding it
A Heavy store deleted with the default Retain policy left its CephObjectStore
behind on purpose, to keep the RGW pools. The e2e runs show what that costs: the
ElasticCluster never finishes terminating (its finalizer waits on volumes the
leftover keeps in use), and the suite's Heavy teardown spends its full 20-minute
budget waiting before giving up — about a third of the whole run, every run, ending
with a cluster stuck in Terminating.
Nothing else removes that CR either. Its ownerReference is no help: sds-elastic's
validating webhook rejects requests to vendored Rook resources from anyone it does
not know — the garbage collector included — which is also why the suite's own
workaround of deleting it directly was denied ("Direct modifications to Rook Ceph
resources are not allowed").
Keeping the CR was never what protected the data: preservePoolsOnDelete is, and it
is already set from the reclaim policy when the CR is rendered (Retain -> true). So
the driver now deletes the CephObjectStore under both policies, and Retain still
preserves the pools and every object in them.
Tests: the driver removes the store under both policies and tolerates an absent
one, and preservePoolsOnDelete still follows the reclaim policy — that field is now
the only thing standing between Retain and the destruction of every bucket's data,
so it is pinned explicitly. The e2e delete spec asserts the CR is gone after the
ObjectStore, and the Heavy cleanup drops its impossible direct-delete step for a
diagnostic that names the leftover if one ever appears again.
The CRD (both languages) and DESIGN documented the old contract and now describe
this one.
Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
duckhawk
force-pushed
the
fix/heavy-teardown-strands-cephobjectstore
branch
from
July 28, 2026 10:55
a83de7c to
b3b4d22
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the e2e run showed
The
heavyblock is the slowest part of the suite by a wide margin — ~30 of 73 minutes — and most of it is spent doing nothing useful:heavy creates a Heavy ObjectStore ... and reaches Readyheavy deletes the Heavy access, bucket and clusterThose 20 minutes are
cleanupHeavyElasticwaiting for the ElasticCluster to disappear — and it never does:So every Heavy run burns its full teardown budget and leaves a cluster stuck in
Terminating.Cause
A Heavy store deleted with the default
Retainleft itsCephObjectStorebehind on purpose, to keep the RGW pools. While that CR exists Rook keeps the Ceph object store alive, so sds-elastic's finalizer waits on volumes the leftover keeps in use.And nothing else removes it. Its ownerReference is no help — sds-elastic's validating webhook rejects requests to vendored Rook resources from anyone it does not know, the garbage collector included. That is the same wall the suite's own workaround hit:
Fix
Keeping the CR was never what protected the data —
preservePoolsOnDeleteis, and it is already set from the reclaim policy when the CR is rendered (Retain→true). The driver now deletes theCephObjectStoreunder both policies:Retain: CR removed, pools and every object in them preserved (Rook's contract for that field), ElasticCluster free to terminate;Delete: unchanged — CR removed, pools destroyed.Tests
preservePoolsOnDeletestill follows the reclaim policy — with the CR now always deleted, that field is the only thing standing betweenRetainand the destruction of every bucket's data, so it is pinned explicitly;ObjectStore;The CRD (both languages) and DESIGN spelled out the old contract — "the CephObjectStore is not deleted" — and now describe this one.
Note
That the webhook also blocks the garbage collector is an inference from the denial above plus the 20-minute hang; the fix does not rest on it. Deleting our own resource in our own teardown is right regardless, and the data is protected by
preservePoolsOnDeleteeither way.Stacked on #36
This branch is rebased onto
fix/seaweedfs-credential-propagationand its base is set to that branch, so the diff here is only the Heavy commit while an e2e run on this PR exercises both fixes at once — the SeaweedFS IAM guard (full/full-highredundancy) and the Heavy teardown (heavy). Merge #36 first; GitHub will retarget this PR tomainautomatically.