You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We currently run three observability/analytics systems side by side:
Sentry — error tracking across six independent, DSN-gated surfaces:
ORB self-host server — src/selfhost/sentry.ts (opt-in SENTRY_DSN): the largest integration — key/payload/value redaction pipeline, structured-log forwarding (console wrap in src/server.ts), SENTRY_MIN_SEVERITY/SENTRY_REPO_MIN_SEVERITY thresholds, five cron monitors via captureCheckIn (SENTRY_MONITOR_NAMES: scheduled-loop, orb-export, orb-relay-drain, orb-relay-register, queue-dead-letter-revive), and an OpenTelemetry span bridge (buildSentryOpenTelemetryBridge).
ORB hosted Worker — @sentry/hono/cloudflare middleware in src/api/routes.ts (opt-in WORKER_SENTRY_DSN).
discovery-index Worker — packages/discovery-index/src/sentry.ts, currently reporting into the shared metagraphed Sentry project rather than anything loopover-owned, plus a full @sentry/cli release/sourcemap pipeline (src/upload-sourcemaps.ts, scripts/validate-sentry-release.mjs, scripts/validate-sourcemaps.mjs).
review-enrichment service — review-enrichment/src/sentry.ts + its own validate-sentry-release.mjs/validate-sourcemaps.mjs.
Umami (self-hosted) — UI pageview analytics through the first-party /stats proxy (apps/loopover-ui/src/lib/analytics-proxy.ts), local-beacon pattern with a deliberate no-third-party-script posture.
Three platforms means three dashboards, three mental models, and no correlation between "what breaks" (Sentry), "what users do" (PostHog), and "who visits" (Umami). PostHog now covers all three domains — plus session replay, feature flags, and AI/LLM observability — on one platform with one event store, and the free-tier headroom is materially better where it matters (error tracking: 100k exceptions/mo free vs Sentry's 5k/mo + 1-seat cap).
This is the same consolidation metagraphed is mid-flight on (JSONbored/metagraphed#7757). This epic deliberately mirrors its shape and reuses its shipped patterns, adapted to loopover's different topology: metagraphed is all-Workers, while loopover adds a Node self-host server, a published miner CLI with an opt-in posture, and two small standalone services.
Principles (apply to every sub-issue)
Parallel-run before cutover. Sentry and Umami keep running until the PostHog replacement has demonstrably captured the same class of signal. Decommission is its own gated sub-issue, never a side effect of any other PR.
No SDK in Worker bundles.posthog-node stays out of Workers — Worker-side capture uses PostHog's raw capture HTTP API (pattern: feat(observability): add PostHog error tracking via raw $exception capture metagraphed#7777). The Node server and miner CLI may use posthog-node (already a root dependency via MCP telemetry); the UI uses posthog-js, measured against the existing bundle-size CI gate.
Redaction is non-negotiable. Every new capture surface ports an existing, proven redaction pipeline (src/selfhost/sentry.ts's SECRET_KEY/PAYLOAD_KEY/SECRET_VALUE + PUBLIC_LOCAL_PATH_SCRUB_PATTERN scrubbing, or the typed-allowlist approach of src/mcp/telemetry.ts) to the PostHog sink — never re-derives one from scratch.
Billing limits first. Before each new capture surface goes live, set a per-product billing limit in PostHog. Web analytics + error events share the Product Analytics event pool with the existing MCP telemetry — watch combined volume.
First-party endpoints. Browser-side capture goes through a first-party proxy on our own origin, the same posture analytics-proxy.ts already proves out for Umami.
One taxonomy.$-prefixed PostHog-native events where PostHog defines a shape ($exception, $pageview, $ai_generation); snake_case custom events otherwise; existing MCP event names unchanged.
Phases
Sub-issues are tracked as native GitHub sub-issues of this epic, with blocked-by relationships encoding the real order:
Phase 2 — Analytics: Umami → PostHog web analytics behind the first-party proxy; insights/dashboards/alert routing.
Phase 3 — Depth: session replay; AI observability ($ai_generation over the ORB's review-pipeline AI calls); one combined evaluation spike (feature flags vs the config-as-code mandate, Logs, Data Warehouse).
One platform for errors, usage analytics, web analytics, replay, and AI observability, correlated per session/caller.
Error-tracking headroom ~20x the current Sentry free tier, without the single-seat cap.
discovery-index stops reporting into metagraphed's Sentry project and gets a loopover-owned home.
Two fewer systems to operate once the sunset gates pass.
Out of scope
Distributed tracing/APM until PostHog tracing is GA. Self-host OTLP export (src/selfhost/otel.ts) is vendor-independent and unaffected either way — only the Sentry span bridge is in scope, and only at decommission time (deferred sub-issue).
metagraphed's own migration (tracked in its repo).
Changing what data is collected: this epic swaps sinks and adds analytics surfaces; the MCP telemetry allowlist and the opt-in postures stay exactly as ratified.
Why
We currently run three observability/analytics systems side by side:
src/selfhost/sentry.ts(opt-inSENTRY_DSN): the largest integration — key/payload/value redaction pipeline, structured-log forwarding (console wrap insrc/server.ts),SENTRY_MIN_SEVERITY/SENTRY_REPO_MIN_SEVERITYthresholds, five cron monitors viacaptureCheckIn(SENTRY_MONITOR_NAMES: scheduled-loop, orb-export, orb-relay-drain, orb-relay-register, queue-dead-letter-revive), and an OpenTelemetry span bridge (buildSentryOpenTelemetryBridge).@sentry/hono/cloudflaremiddleware insrc/api/routes.ts(opt-inWORKER_SENTRY_DSN).packages/loopover-miner/lib/sentry.ts(opt-inLOOPOVER_MINER_SENTRY_DSN; an operator points at their own project — nothing phones home by default, docs(selfhost): add combined ORB + AMS quickstart #6011).packages/discovery-index/src/sentry.ts, currently reporting into the shared metagraphed Sentry project rather than anything loopover-owned, plus a full@sentry/clirelease/sourcemap pipeline (src/upload-sourcemaps.ts,scripts/validate-sentry-release.mjs,scripts/validate-sourcemaps.mjs).review-enrichment/src/sentry.ts+ its ownvalidate-sentry-release.mjs/validate-sourcemaps.mjs.apps/loopover-ui/src/lib/browser-sentry.ts(opt-inVITE_SENTRY_DSN,@sentry/react) +.github/workflows/ui-sentry-release.yml+SENTRY_BUILD_SOURCEMAPSinvite.config.ts.src/mcp/telemetry.ts(Spec: usage-telemetry instrumentation strategy for MCP (PostHog) #6228/feat(mcp): add a typed PostHog wrapper module for src/mcp/server.ts (remote) #6235 — hosted MCP tool-call events under a strict typed field allowlist,POSTHOG_API_KEY/POSTHOG_HOST) andpackages/loopover-mcp/lib/telemetry.ts(feat(mcp): instrument the local MCP tool-dispatch chokepoint with PostHog events (opt-in) #6238 — the opt-in stdio CLI equivalent), surfaced in the UI viamcp-tool-usage-card.tsx./statsproxy (apps/loopover-ui/src/lib/analytics-proxy.ts), local-beacon pattern with a deliberate no-third-party-script posture.Three platforms means three dashboards, three mental models, and no correlation between "what breaks" (Sentry), "what users do" (PostHog), and "who visits" (Umami). PostHog now covers all three domains — plus session replay, feature flags, and AI/LLM observability — on one platform with one event store, and the free-tier headroom is materially better where it matters (error tracking: 100k exceptions/mo free vs Sentry's 5k/mo + 1-seat cap).
This is the same consolidation metagraphed is mid-flight on (JSONbored/metagraphed#7757). This epic deliberately mirrors its shape and reuses its shipped patterns, adapted to loopover's different topology: metagraphed is all-Workers, while loopover adds a Node self-host server, a published miner CLI with an opt-in posture, and two small standalone services.
Principles (apply to every sub-issue)
SENTRY_DSN) and miner operators (LOOPOVER_MINER_SENTRY_DSN) opt in and point at their own project; swapping vendor must not change the no-phone-home default (docs(selfhost): add combined ORB + AMS quickstart #6011/feat(mcp): add a typed PostHog wrapper module for src/mcp/server.ts (remote) #6235's posture). The loopover-owned project (Provision a loopover-owned Posthog project for the hosted fleet #7875) serves the hosted fleet, our own Workers, and our own UI deploy — not self-hosters.posthog-nodestays out of Workers — Worker-side capture uses PostHog's raw capture HTTP API (pattern: feat(observability): add PostHog error tracking via raw $exception capture metagraphed#7777). The Node server and miner CLI may useposthog-node(already a root dependency via MCP telemetry); the UI usesposthog-js, measured against the existing bundle-size CI gate.src/selfhost/sentry.ts's SECRET_KEY/PAYLOAD_KEY/SECRET_VALUE +PUBLIC_LOCAL_PATH_SCRUB_PATTERNscrubbing, or the typed-allowlist approach ofsrc/mcp/telemetry.ts) to the PostHog sink — never re-derives one from scratch.analytics-proxy.tsalready proves out for Umami.$-prefixed PostHog-native events where PostHog defines a shape ($exception, $pageview, $ai_generation); snake_case custom events otherwise; existing MCP event names unchanged.Phases
Sub-issues are tracked as native GitHub sub-issues of this epic, with blocked-by relationships encoding the real order:
$ai_generationover the ORB's review-pipeline AI calls); one combined evaluation spike (feature flags vs the config-as-code mandate, Logs, Data Warehouse).Expected outcomes
Out of scope
src/selfhost/otel.ts) is vendor-independent and unaffected either way — only the Sentry span bridge is in scope, and only at decommission time (deferred sub-issue).Resources