Skip to content

Add incident.io alert provider - #1353

Merged
stefanprodan merged 1 commit into
fluxcd:mainfrom
xchose:incidentio-provider
Jul 26, 2026
Merged

Add incident.io alert provider#1353
stefanprodan merged 1 commit into
fluxcd:mainfrom
xchose:incidentio-provider

Conversation

@xchose

@xchose xchose commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Fixes #1124.

Adds the incidentio Provider type posting alert events to an incident.io HTTP alert source (bearer-token auth, 202 Accepted response).

Design decisions

  1. Severity mapping: error events are sent with status firing, all other severities with status resolved — the same semantics as the PagerDuty provider, enabling automatic alert resolution in incident.io.
  2. Deduplication key: the involved object's UID — same approach as the PagerDuty provider. A name-based key (<kind>/<namespace>/<name>) would collide across clusters reporting to the same alert source (every cluster fires Kustomization/flux-system/flux-system), letting an info event from one cluster resolve another cluster's alert. Changed after review feedback.
  3. Progressing events are skipped, matching the PagerDuty provider behavior.
  4. Docs note that eventSeverity: error on an Alert disables auto-resolution (info events must flow for recovery to be delivered).

Testing

  • Unit tests covering payload mapping, bearer auth header, 202 handling, firing/resolved mapping, Progressing skip, and constructor validation; fuzz harness consistent with the other providers.
  • Tested end-to-end on a kind cluster against a real incident.io workspace: alert fires on a broken GitRepository branch, auto-resolves on recovery via the deduplication key, separate objects produce separate alerts, and an invalid token surfaces as a 401 error in the controller logs.
  • Retested after the dedup-key change: captured the raw payload with an in-cluster echo server — firing and recovery events carry the same object UID as deduplication_key, and a second object gets its own UID key (no sharing). Screenshots in the review thread show the alert detail with the UID key.

Alert fired for the broken GitRepository branch:

incident.io alert list showing the alert in Firing state

Automatically resolved after the recovery event (same deduplication key):

incident.io alert list showing both alerts Resolved

Alert detail with event message, deduplication key and firing→resolved timeline:

incident.io alert detail with deduplication key and resolved status

Step-by-step testing procedure on a kind cluster:
INCIDENTIO_E2E_REPORT.md

🤖 Generated with Claude Code

Comment thread internal/notifier/incidentio.go Outdated
Add the incidentio Provider type for incident.io. Events are posted to
an HTTP alert source endpoint as alert events, authenticated with a
bearer token. Error events are sent with status firing, all other
severities with status resolved, using the involved object UID as the
deduplication key (same as the PagerDuty provider) so that recovery
events automatically resolve the alert fired for the same object, and
objects with the same kind/namespace/name on different clusters never
share a key. Progressing events are skipped, matching the PagerDuty
provider behavior.

Assisted-by: Claude Code/claude-fable-5
Signed-off-by: Chose Carreras <xchose@gmail.com>
@xchose
xchose force-pushed the incidentio-provider branch from 7052fb9 to 5119753 Compare July 26, 2026 17:19
@stefanprodan stefanprodan added enhancement New feature or request area/alerting Alerting related issues and PRs labels Jul 26, 2026

@stefanprodan stefanprodan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks @xchose 🏅

@stefanprodan
stefanprodan merged commit 20f1441 into fluxcd:main Jul 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/alerting Alerting related issues and PRs enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incidentio notification provider

2 participants