Alerting on app status transitions

[pachev]

Rewritten 2026-05-23. Original draft (2026-05-18) predates Releases-as-first-class (ADR 0008), server_stats time-series (#?, May 23), and assumed Swoosh was already a dep. None of those held. This version matches the current code.

Goal

Turn the AlertsLive stub into a real alert pipeline: detect Server and Release state changes plus metric thresholds, persist them, surface them in the Alerts view and a dashboard tile, and notify by email.

What changed since the original draft (read before scoping)

• Releases are first-class (ADR 0008). A probe failure marks a Release unhealthy, which is distinct from a Server Check (SSH reachability). Alert subjects are Server and Release, not "app." Individual Application status (running|stopped|unreachable|unknown) is intentionally not an alert subject — a Release has many OTP apps and per-app alerts would be noise.
• Time-series exists. server_stats (bucket, recorded_at, jsonb stats, per Server) is collected every minute by Mast.Workers.StatsCollect and downsampled by StatsDownsample. Threshold alerts are now feasible and are in scope.
• Swoosh is NOT a dependency. Not in mix.exs, not in mix.lock, no mailer module, no config. The word "swoosh" appears only in an OTP-app allowlist string and a stale dev comment. Email therefore adds a dependency (see step 5) and must be flagged per CLAUDE.md.
• No transition-detection hook exists today. Apps.upsert_from_probe/2 does a bulk update_all(status: "unreachable") (apps.ex:91) with no old→new diff. ConnectionCheck broadcasts :server_updated but records no transition. Detection must be built, not merely subscribed to.

Scope

1. Schema: alerts table

id (binary_id), kind (string: server_down | release_unhealthy | metric_threshold), subject_type (server | release), subject_id (binary_id), severity (info | warning | critical), message, state (open | acknowledged | resolved), opened_at, resolved_at (nullable), jsonb context. Index (state) and (subject_type, subject_id). A partial unique index on (kind, subject_type, subject_id) WHERE state != 'resolved' so one subject can't have two open alerts of the same kind (the open/resolve toggle relies on this).

2. Transition detection (Server + Release)

• Server up↔down: hook ConnectionCheck where it already calls Fleet.record_metrics / Fleet.mark_unreachable (connection_check.ex:45,50). Compare the prior status to the new one; on up→down open a server_down alert, on down→up resolve it. The old status is available from the loaded server before the write.
• Release health: hook the Release probe path. A Release's health is the rollup over its Applications (per ADR 0008). Define the rollup explicitly (e.g. unhealthy if any required app is unreachable, or all apps unreachable — open question below). On transition, open/resolve a release_unhealthy alert.
• Reuse the open/resolve toggle via the partial unique index; opening is insert ... on_conflict: :nothing, resolving is an update of the matching open row.

3. Threshold alerts (metric_threshold)

• New Oban cron worker Mast.Workers.AlertEval, scheduled "* * * * *" (every minute, right after StatsCollect writes the 1m bucket). Add to the crontab in config/config.exs alongside the existing stats jobs.
• Reads the latest 1m server_stats per server via Fleet.list_stats/3 (or a new latest-per-server helper).
• Hysteresis required (the "for Y minutes" part): a threshold opens only after N consecutive breaching samples, resolves only after M consecutive clear samples. Otherwise a one-sample CPU spike flaps. Track the streak in the alert's context jsonb or recompute from the last N samples.
• Thresholds configured per-metric (CPU %, memory %, disk %) in Settings → Notifications. Start with fleet-wide defaults; per-server overrides are a follow-up.
• Use unique: [period: 55] on the worker to dedupe with the 1-minute cron, matching the existing stats workers.

4. AlertsLive + dashboard tile

• Replace the stub (alerts_live.ex, currently 29 lines pointing at ADR 0004). List open + resolved alerts, filter by state and severity, acknowledge / resolve buttons. Subscribe to a new "alerts" PubSub topic; apply deltas, do not refetch the whole list on each event (see the render-efficiency lessons in perf: SSH connection churn + render/query inefficiencies (audited, batch fix) #27).
• Dashboard stat row: add an "Open Alerts" ui_stat tile. Compute the count in assigns alongside the other fleet_stats values (dashboard_live.ex:308), not in render/1.

5. Email channel ⚠ ADDS A DEPENDENCY — confirm before building

• Add two deps: {:swoosh, "~> 1.x"} and {:gen_smtp, "~> 1.x"} (SMTP is not a built-in Swoosh adapter). Per CLAUDE.md these must be flagged and OK'd before the step proceeds — decision recorded above (Gmail SMTP).
• Mast.Mailer + Settings → Notifications config (recipient, on-open / on-resolve toggles, threshold values from step 3).
• Send on alert open and resolve. Failures to send must not block the detection path — fire-and-forget via Oban or a Task, log on failure.

Out of scope

• Per-app (individual Application) alerts — too noisy.
• Pager integrations (PagerDuty, Opsgenie), alert routing rules, escalation.
• Per-server threshold overrides (fleet-wide defaults first).
• Non-email channels (Slack/webhook) — separate issue once the channel abstraction exists.

Resolved decisions

1. Release health rollup: a Release is unhealthy when any of its required Applications is unreachable. (Introduces a notion of "required app" per Release — needs a CONTEXT.md term and a way to mark an app required; default-all-required is a sane start, refine in implementation.)
2. Threshold hysteresis: N=3 consecutive breaching 1m samples to open (~3 min), M=2 consecutive clear samples to resolve (~2 min). Per-metric thresholds: CPU %, memory %, disk %.
3. Email channel = SMTP via Gmail. Adapter is Swoosh.Adapters.SMTP + gen_smtp (SMTP is not in Swoosh's default adapters, so this is two new deps: swoosh + gen_smtp). Uses a personal Gmail account: requires 2-Step Verification on the account and a generated App Password (plain-password SMTP is disabled by Google since 2022). Config: smtp.gmail.com:587, TLS, username = the Gmail address, password = the 16-char app password, stored encrypted/in env (not committed). Note Gmail's ~500 recipients/day cap — acceptable at personal-fleet scale; a flapping subject could approach it, which is another reason hysteresis (above) matters.

Acceptance

• alerts table + partial unique index (one open alert per kind/subject)
• Server up→down opens, down→up resolves a server_down alert
• A Release can mark Applications as required; health = any required app unreachable
• Release health transition opens/resolves a release_unhealthy alert
• AlertEval cron evaluates thresholds with hysteresis; no flapping on single-sample spikes
• AlertsLive lists/filters/acks/resolves, applies PubSub deltas (no full refetch)
• Dashboard "Open Alerts" tile, count computed in assigns
• Email sent on open/resolve (after dep flagged + OK'd); send failure doesn't block detection
• Tests cover each transition + the hysteresis edges
• mix precommit green