PMM-14665 Skip nodes internal to a PMM deployment when adding a service - #5704
Open
ademidoff wants to merge 8 commits into
Open
PMM-14665 Skip nodes internal to a PMM deployment when adding a service#5704ademidoff wants to merge 8 commits into
ademidoff wants to merge 8 commits into
Conversation
In an HA deployment the PostgreSQL cluster backing PMM's persistence layer runs pmm-client sidecars, so its pods were offered in the Nodes dropdown as valid monitoring delegates. Those Nodes are dedicated and must not take on extra monitoring workloads. PMM Server now accepts PMM_INTERNAL_NODE_NAME_PREFIXES, reports the Nodes it matches as is_pmm_internal_node, and rejects requests which delegate monitoring of a remote address to an Agent running on one of them. Local addresses remain allowed, so those Nodes keep monitoring the services inside their own pod. AddAzureDatabaseRequest gains pmm_agent_id, which the UI has always sent but the server discarded, placing Azure exporters on the PMM Server unconditionally.
ademidoff
requested review from
4nte and
JiriCtvrtka
and removed request for
a team
July 27, 2026 22:25
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5704 +/- ##
==========================================
+ Coverage 43.59% 45.27% +1.67%
==========================================
Files 415 418 +3
Lines 43134 43397 +263
==========================================
+ Hits 18804 19647 +843
+ Misses 22454 21805 -649
- Partials 1876 1945 +69
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`pmm-agent setup` registered the Node on every run, and a forced registration makes PMM Server drop the Node together with every Service on it. That is harmless for the database operators, whose sidecars re-add their own services on each container start, but it would silently delete user-added services from a long-lived PMM Client pod, which is what the pmm-ha Helm chart is about to pre-provision as monitoring delegates. Registration is now skipped when the Agent already holds an ID and is being set up against the same PMM Server it is registered with. `--force` still registers the Node again, and pointing an Agent at a different PMM Server registers it there as before.
…gent Skipping registration for an Agent which already holds an ID strands it whenever PMM Server no longer knows that ID, e.g. after the server was reinstalled or restored from a backup taken before the Agent was registered. pmm-agent then loops on "No Agent with ID" while the Node monitors nothing. `pmm-agent setup` now asks the server whether it still knows the Agent and registers the Node again when it does not. Credentials the server rejects count as an answer too: registering reports that with an actionable message, whereas skipping it would fail silently later on. An unreachable server is not an answer: a failed check keeps the registration, so an Agent still starts while PMM Server has no leader elected yet.
ademidoff
marked this pull request as draft
July 28, 2026 07:56
Keeping this ticket to what it is about: reporting the Nodes of a PMM deployment's own infrastructure as internal and refusing to delegate monitoring to them. Making `pmm-agent setup` idempotent is what would let a pre-provisioned PMM Client pod register once and keep its identity across restarts, but the chart can do that with a one-time init container instead. The agent side is worth its own effort: it changes pmm-agent CLI behaviour, it needs a release note, and the surrounding questions - how an Agent recovers when PMM Server no longer knows it, and how an orphaned Node is reclaimed - deserve their own design and review.
ademidoff
marked this pull request as ready for review
July 30, 2026 07:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket number: PMM-14665
Feature build: Percona-Lab/pmm-submodules#4498
In an HA deployment the PostgreSQL cluster backing PMM's persistence layer runs pmm-client sidecars, so its pods were offered in the "Nodes" dropdown of Add Service as valid monitoring delegates. Those nodes are dedicated and must not take on any extra monitoring workloads.
PMM Server now accepts
PMM_INTERNAL_NODE_NAME_PREFIXES(set by the pmm-ha Helm chart), reports the nodes it matches asis_pmm_internal_nodeonListNodes/GetNode, and rejects requests which delegate monitoring of a remote address to an agent running on one of them.Local addresses (empty,
localhost,127.0.0.1,::1) stay allowed on those nodes: that is how the PostgreSQL operator's sidecar registers its own PostgreSQL and Patroni services on every pod start, so a blanket rejection would break the persistence layer's own monitoring. An airtight alternative is to key the check on the registering service account instead, which is deliberately deferred - see the ticket.AddAzureDatabaseRequestgainspmm_agent_id, which the UI has always sent but the server discarded, placing Azure exporters on the PMM Server unconditionally.PMM Server nodes stay eligible on purpose: until the Azure API accepts an agent, Azure monitoring cannot run anywhere else.
If this PR adds, removes or alters one or more API endpoints, please review and update the relevant API documentation as well:
If this PR is related to other PRs, contributions, or ongoing work in this or other repositories, please reference them here: