Skip to content

fix(cephrgw): remove the CephObjectStore on teardown instead of stranding it - #37

Open
duckhawk wants to merge 1 commit into
fix/seaweedfs-credential-propagationfrom
fix/heavy-teardown-strands-cephobjectstore
Open

fix(cephrgw): remove the CephObjectStore on teardown instead of stranding it#37
duckhawk wants to merge 1 commit into
fix/seaweedfs-credential-propagationfrom
fix/heavy-teardown-strands-cephobjectstore

Conversation

@duckhawk

@duckhawk duckhawk commented Jul 27, 2026

Copy link
Copy Markdown
Member

What the e2e run showed

The heavy block is the slowest part of the suite by a wide margin — ~30 of 73 minutes — and most of it is spent doing nothing useful:

Spec Time
heavy creates a Heavy ObjectStore ... and reaches Ready 9.5 min (≈8 of it bringing up the ElasticCluster)
heavy deletes the Heavy access, bucket and cluster 20.3 min

Those 20 minutes are cleanupHeavyElastic waiting for the ElasticCluster to disappear — and it never does:

ElasticCluster e2e-osc-heavy-ec still terminating (finalizers=[sds-elastic.deckhouse.io/elastic-cluster])   ← ×20 min
warning: ElasticCluster teardown failed: timeout waiting for ElasticCluster to be gone: context deadline exceeded

So every Heavy run burns its full teardown budget and leaves a cluster stuck in Terminating.

Cause

A Heavy store deleted with the default Retain left its CephObjectStore behind on purpose, to keep the RGW pools. While that CR exists Rook keeps the Ceph object store alive, so sds-elastic's finalizer waits on volumes the leftover keeps in use.

And nothing else removes it. Its ownerReference is no help — sds-elastic's validating webhook rejects requests to vendored Rook resources from anyone it does not know, the garbage collector included. That is the same wall the suite's own workaround hit:

admission webhook "d8-sds-elastic-vendor-cr-validation.deckhouse.io" denied the request:
Direct modifications to Rook Ceph resources are not allowed. (requested by kubernetes-super-admin)

Fix

Keeping the CR was never what protected the data — preservePoolsOnDelete is, and it is already set from the reclaim policy when the CR is rendered (Retaintrue). The driver now deletes the CephObjectStore under both policies:

  • Retain: CR removed, pools and every object in them preserved (Rook's contract for that field), ElasticCluster free to terminate;
  • Delete: unchanged — CR removed, pools destroyed.

Tests

  • the driver removes the store under both reclaim policies and tolerates an already-absent one;
  • preservePoolsOnDelete still follows the reclaim policy — with the CR now always deleted, that field is the only thing standing between Retain and the destruction of every bucket's data, so it is pinned explicitly;
  • the e2e delete spec asserts the CR is gone after the ObjectStore;
  • the Heavy cleanup drops its impossible direct-delete step, replacing it with a diagnostic that names the leftover if one ever appears again.

The CRD (both languages) and DESIGN spelled out the old contract — "the CephObjectStore is not deleted" — and now describe this one.

Note

That the webhook also blocks the garbage collector is an inference from the denial above plus the 20-minute hang; the fix does not rest on it. Deleting our own resource in our own teardown is right regardless, and the data is protected by preservePoolsOnDelete either way.

Stacked on #36

This branch is rebased onto fix/seaweedfs-credential-propagation and its base is set to that branch, so the diff here is only the Heavy commit while an e2e run on this PR exercises both fixes at once — the SeaweedFS IAM guard (full / full-highredundancy) and the Heavy teardown (heavy). Merge #36 first; GitHub will retarget this PR to main automatically.

@duckhawk
duckhawk force-pushed the fix/heavy-teardown-strands-cephobjectstore branch from fa4bcb4 to ec1e6d8 Compare July 27, 2026 16:58
@duckhawk
duckhawk changed the base branch from main to fix/seaweedfs-credential-propagation July 27, 2026 16:58
@duckhawk duckhawk added e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR and removed e2e/commander/run Trigger the storage-e2e commaner pipeline for this PR labels Jul 27, 2026
@duckhawk
duckhawk force-pushed the fix/heavy-teardown-strands-cephobjectstore branch from ec1e6d8 to a83de7c Compare July 28, 2026 07:21
…ding it

A Heavy store deleted with the default Retain policy left its CephObjectStore
behind on purpose, to keep the RGW pools. The e2e runs show what that costs: the
ElasticCluster never finishes terminating (its finalizer waits on volumes the
leftover keeps in use), and the suite's Heavy teardown spends its full 20-minute
budget waiting before giving up — about a third of the whole run, every run, ending
with a cluster stuck in Terminating.

Nothing else removes that CR either. Its ownerReference is no help: sds-elastic's
validating webhook rejects requests to vendored Rook resources from anyone it does
not know — the garbage collector included — which is also why the suite's own
workaround of deleting it directly was denied ("Direct modifications to Rook Ceph
resources are not allowed").

Keeping the CR was never what protected the data: preservePoolsOnDelete is, and it
is already set from the reclaim policy when the CR is rendered (Retain -> true). So
the driver now deletes the CephObjectStore under both policies, and Retain still
preserves the pools and every object in them.

Tests: the driver removes the store under both policies and tolerates an absent
one, and preservePoolsOnDelete still follows the reclaim policy — that field is now
the only thing standing between Retain and the destruction of every bucket's data,
so it is pinned explicitly. The e2e delete spec asserts the CR is gone after the
ObjectStore, and the Heavy cleanup drops its impossible direct-delete step for a
diagnostic that names the leftover if one ever appears again.

The CRD (both languages) and DESIGN documented the old contract and now describe
this one.

Signed-off-by: v.oleynikov <vasily.oleynikov@flant.com>
@duckhawk
duckhawk force-pushed the fix/heavy-teardown-strands-cephobjectstore branch from a83de7c to b3b4d22 Compare July 28, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant