Skip to content

Retry object updates on conflict in receiver tests - #1352

Merged
stefanprodan merged 1 commit into
fluxcd:mainfrom
jojinkb:test-read-before-update
Jul 25, 2026
Merged

Retry object updates on conflict in receiver tests#1352
stefanprodan merged 1 commit into
fluxcd:mainfrom
jojinkb:test-read-before-update

Conversation

@jojinkb

@jojinkb jojinkb commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Fixes #473

Problem

Several receiver test steps mutate an object immediately after an Eventually block that keys off a condition. The reconciler's deferred patch (patch.SerialPatcher) commits its changes with multiple sequential API requests: the conditions patch first, then metadata/spec, then the remaining status fields (observedGeneration, lastHandledReconcileAt, webhookPath). The Eventually can therefore return as soon as the conditions patch lands, while the trailing status patch is still in flight. If that patch commits between the test's Get and Update, the Update fails with an optimistic concurrency conflict ("the object has been modified"), producing the intermittent patch errors described in the issue.

Changes

Converted the four remaining plain Get + mutate + Update sequences in internal/controller/receiver_controller_test.go to the read-modify-write retry idiom already used in other Flux controllers (e.g. kustomize-controller):

g.Eventually(func() error {
    if err := k8sClient.Get(context.Background(), client.ObjectKeyFromObject(receiver), resultR); err != nil {
        return err
    }
    resultR.Spec.Suspend = true
    return k8sClient.Update(context.Background(), resultR)
}, timeout, time.Second).Should(Succeed())

Sites changed:

  • TestReceiverReconciler_Reconcile / "fails with secret not found error"
  • TestReceiverReconciler_Reconcile / "handles reconcileAt"
  • TestReceiverReconciler_Reconcile / "finalizes suspended object"
  • TestReceiverReconciler_EventHandler / "doesn't update the URL on spec updates"

Audit of the remaining mutation sites in the test suite:

  • The raw client.RawPatch(types.MergePatchType, ...) calls in the receiver tests do not send a resourceVersion and cannot conflict — this is the workaround the issue refers to as the "other option".
  • The alert and provider tests read the object via Eventually/Get before patching, and patch.Helper issues its spec/metadata patch without an optimistic lock, so they cannot fail with conflict errors either.

No controller code is changed.

Testing

KUBEBUILDER_ASSETS=... go test ./internal/controller/ -run 'TestReceiverReconciler' -v   # PASS
KUBEBUILDER_ASSETS=... go test ./internal/controller/                                   # ok (all package tests)
KUBEBUILDER_ASSETS=... go test ./internal/controller/ -run 'TestReceiverReconciler_Reconcile' -count=5  # ok
go vet ./internal/controller/ && gofmt -l internal/controller/                          # clean

The receiver tests updated objects right after an Eventually block
that observes the Ready condition. The reconciler commits its changes
with multiple sequential patch requests (conditions first, then the
rest of the status), so a trailing status patch can still bump the
resourceVersion after the condition became visible. When that patch
lands between the test's Get and Update, the Update fails with an
optimistic concurrency conflict.

Re-read the object and retry the update inside an Eventually block so
transient conflicts with in-flight reconciles resolve instead of
failing the test.

Assisted-by: Claude Code/claude-fable-5
Signed-off-by: Jojin <jojin.kb@gmail.com>
@stefanprodan stefanprodan added the area/testing Testing related issues and pull requests label Jul 25, 2026

@stefanprodan stefanprodan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks @jojinkb

@stefanprodan
stefanprodan merged commit e888c96 into fluxcd:main Jul 25, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/testing Testing related issues and pull requests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensure objects are read before making changes in tests

2 participants