Skip to content

k8s-certs-renew: derive next run from calendar to stop forced monthly renewal#13362

Open
polarAli wants to merge 1 commit into
kubernetes-sigs:masterfrom
polarAli:fix/k8s-certs-renew-monthly-13072
Open

k8s-certs-renew: derive next run from calendar to stop forced monthly renewal#13362
polarAli wants to merge 1 commit into
kubernetes-sigs:masterfrom
polarAli:fix/k8s-certs-renew-monthly-13072

Conversation

@polarAli

@polarAli polarAli commented Jul 7, 2026

Copy link
Copy Markdown

What type of PR is this?

/kind bug

What this PR does / why we need it:

k8s-certs-renew.sh decided whether to renew control-plane certificates by reading the next scheduled run from the timer's NextElapseUSecRealtime property. systemd clears that property while the service the timer triggered is still running, so it is always empty during the script's own execution. The empty value made the script take the "skip expiry comparison, renew directly" branch and run kubeadm certs renew all on every timer fire — effectively renewing all control-plane certificates monthly (and restarting the control-plane pods) instead of only when they are close to expiry.

This computes the next elapse from the calendar spec with systemd-analyze calendar instead, which is independent of the timer's activation state, so the expiry comparison works as intended. If systemd-analyze cannot resolve the calendar, it falls back to the previous NextElapseUSecRealtime lookup.

It also guards the date parse: systemd-analyze prints Next elapse: never for a calendar with no future run (e.g. a bounded one-shot override), and any unparseable next_time must fall through to a direct renewal as well. Without that guard an empty target_time produces a large negative expiry threshold and the script would exit without renewing at all — worse than the original bug. With the guard, an unresolvable schedule degrades to the pre-existing "renew directly" behavior.

Which issue(s) this PR fixes:

Fixes #13072

Special notes for your reviewer:

  • This restores the intended behavior. The expiry-comparison path was effectively dead before this change (it was always skipped), so in practice renewals become roughly annual instead of monthly and the monthly control-plane pod restart stops.

  • systemd-analyze calendar has existed since systemd v235; the Next elapse: label is unchanged from v239 through current. On systemd too old to have the verb (e.g. Amazon Linux 2 / systemd 219), the command produces no output, next_time is empty, and the script falls back exactly to today's behavior — no regression.

  • Verified on a systemd 252 host. The single quoted calendar spec always yields exactly one Next elapse: line, sed extracts it, and date -d "${next_time} + ${days_buffer} days" parses it (including a trailing timezone abbreviation and numeric offsets such as +0330). Branch behavior with a mocked kubeadm:

    calendar spec cert state result
    default monthly far from expiry (300d) skip renew, exit 0
    default monthly near expiry (5d) proceed to renew
    past one-shot (Next elapse: never) far from expiry renew directly
    invalid spec (empty) far from expiry renew directly
  • One user-visible side effect worth calling out: because of the bug, systemctl start k8s-certs-renew.service used to be a guaranteed force-renew. It is now conditional on expiry. The explicit force path remains kubeadm certs renew all.

  • These templated shell scripts don't have a unit-test harness in the repo (shellcheck CI matches *.sh, not *.sh.j2), so this was verified manually as above; happy to add a test if reviewers would like one in a particular form.

  • Approach suggested by @zhenliangliang in the issue discussion.

  • Parts of this change were drafted with the assistance of generative AI; I have reviewed, understand, and tested all of it.

Does this PR introduce a user-facing change?:

Fixed the `k8s-certs-renew` service renewing all control-plane certificates on every monthly timer fire; certificates are now renewed only when they are close to expiry, as intended.

@kubernetes-prow kubernetes-prow Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. kind/bug Categorizes issue or PR as related to a bug. labels Jul 7, 2026
@linux-foundation-easycla

linux-foundation-easycla Bot commented Jul 7, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: polarAli / name: Ali Ghotbizadeh (479c0f8)

@kubernetes-prow

Copy link
Copy Markdown
Contributor

Welcome @polarAli!

It looks like this is your first PR to kubernetes-sigs/kubespray 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kubespray has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@kubernetes-prow kubernetes-prow Bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jul 7, 2026
@kubernetes-prow kubernetes-prow Bot requested review from ErikJiang and cyclinder July 7, 2026 13:45
@kubernetes-prow

Copy link
Copy Markdown
Contributor

Hi @polarAli. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 7, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: polarAli
Once this PR has been reviewed and has the lgtm label, please assign tico88612 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jul 7, 2026
@polarAli polarAli force-pushed the fix/k8s-certs-renew-monthly-13072 branch from 5720442 to 6586453 Compare July 7, 2026 13:47
@kubernetes-prow kubernetes-prow Bot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Jul 7, 2026
@polarAli polarAli force-pushed the fix/k8s-certs-renew-monthly-13072 branch from 6586453 to 8559838 Compare July 7, 2026 15:04
… renewal

The renew script read the next scheduled run from the timer's
NextElapseUSecRealtime property. systemd clears that property while the service
the timer triggered is still running, so it was always empty during the script's
own execution. The empty value made the script skip the certificate-expiry
comparison and run `kubeadm certs renew all` on every timer fire, effectively
renewing all control-plane certificates monthly instead of only near expiry.

Compute the next elapse from the calendar spec with `systemd-analyze calendar`
instead, which is independent of the timer's activation state. Fall back to the
previous NextElapseUSecRealtime lookup if systemd-analyze cannot resolve it, so
the existing skip-comparison safety net is preserved.

Also guard the date parse: systemd-analyze prints "Next elapse: never" for a
calendar with no future run (e.g. a bounded one-shot override), and any
unparseable next_time must fall through to a direct renewal too. Without the
guard an empty target_time yields a large negative expiry threshold and the
script exits without renewing at all, which is worse than the bug being fixed.

Approach suggested by zhenliangliang in the issue discussion.

Signed-off-by: Ali Ghotbizadeh <ali.ghotbizadeh@neor.pro>
@polarAli polarAli force-pushed the fix/k8s-certs-renew-monthly-13072 branch from 8559838 to 479c0f8 Compare July 7, 2026 15:12
@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 7, 2026
@yankay

yankay commented Jul 8, 2026

Copy link
Copy Markdown
Member

/ok-to-test

@kubernetes-prow kubernetes-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 8, 2026
@yankay yankay requested a review from Copilot July 8, 2026 04:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes k8s-certs-renew.sh renewing control-plane certificates on every timer fire by avoiding reliance on NextElapseUSecRealtime (which can be empty while the triggered service is running). It derives the next run time from the systemd calendar spec via systemd-analyze calendar, with a fallback to the previous timer-property approach, and guards date parsing so unparseable schedules degrade to the existing “renew directly” behavior.

Changes:

  • Compute next_time using systemd-analyze calendar "{{ auto_renew_certificates_systemd_calendar }}" and parse Next elapse: output.
  • Fall back to systemctl show ... NextElapseUSecRealtime when systemd-analyze can’t resolve the calendar.
  • Add guarded date parsing to prevent an unparseable next time from causing a “never renew” outcome.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

fi

if [ "${target_time}" == "" ]; then
echo "## Skip expiry comparison due to fail to parse next elapse from systemd calendar,do renewal directly ##"
# the timer property if systemd-analyze cannot resolve the calendar.
next_time=$(LC_ALL=C systemd-analyze calendar "{{ auto_renew_certificates_systemd_calendar }}" 2>/dev/null | sed -n 's/^[[:space:]]*Next elapse:[[:space:]]*//p')
if [ "${next_time}" == "" ]; then
next_time=$(systemctl show k8s-certs-renew.timer -p NextElapseUSecRealtime --value)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this fallback if NextElapseUSecRealtime is empty during runtime?

Also, please don't just dump an issue number into an AI tool and ask it to write a fix. It's really important to manually read the issue, investigate the related files, and properly test the changes yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

k8s-certs-renew renews certs every month

4 participants