Is it Observable

📺 Watch the episode: https://www.youtube.com/@Isitobservable — Kagent: Build and Observe Kubernetes-Native AI Agents

---

Episode : Kagent — Build and Observe Kubernetes-Native AI Agents

Everyone is shipping AI agents. Almost no one can see what they do. This episode builds a fully observable agentic platform on Kubernetes with kagent — the CNCF Sandbox framework where every agent, model and tool is a Kubernetes CRD — and then makes every reasoning step, tool call and delegation hop visible in Dynatrace through OpenTelemetry.

You will stand up a cluster, deploy the full observability + service-mesh stack, run an SRE agent that troubleshoots a broken gRPC route on live cluster state, and compose a BMAD crew of six specialist agents that ships a real Todo app on GitHub — issues, milestone, code, PR and review — all watched live in two Dynatrace dashboards. Because kagent emits span-derived GenAI telemetry (gen_ai.*, no custom OTLP metrics), you see token usage, tool calls and A2A delegation without adding a single line to the agents.

Components used in this episode:

• kagent v0.9.11 — Kubernetes-native agentic framework (agents/models/tools as CRDs)
• OpenTelemetry Operator — manages the two collectors that carry all telemetry
• Two OpenTelemetry Collectors — a DaemonSet node-ingest tier + a StatefulSet gateway (the sole Dynatrace egress)
• Dynatrace Operator v1.9.1 (kubernetes-monitoring mode) — k8s entity enrichment + log/metric ingest
• Istio (ambient profile) + Gateway API / GAMMA — the mesh that carries the gRPC-route demo
• OpenTelemetry Demo — the OTel-native microservices workload (plays the "app" + workload generator)
• Ollama (qwen3.6:latest, self-hosted) — the tool-calling model behind every agent
• cert-manager — webhook CA prerequisite for the OTel Operator
• kagent GitHub MCP server + mcp-k8s-networking — the tools the agents act through

Everything is destination Dynatrace, and the whole repo ships no environment-specific values in git — every IP, tenant and token comes from deploy/.env, so you can run all of it in your cluster.

---

Prerequisite

Install the CLIs and have a Dynatrace tenant ready before you start:

• kubectl — talk to the cluster
• helm 3.x — install kagent, Istio, the operators
• git — clone this repo
• jq — parse JSON from the agent A2A endpoints
• curl — hit the kagent controller / A2A APIs
• clusterctl — only for the Cluster API provisioning path
• gcloud — only for the GKE provisioning path
• the kagent CLI — optional, for driving agents from your terminal

You also need:

• A Kubernetes cluster (this episode provisions one with Cluster API on Proxmox; GKE works too).
• A Dynatrace SaaS tenant + two API tokens (see Getting started below).
• An Ollama endpoint hosting a tool-calling model (qwen3.6:latest in this episode). kagent runs on tool calls — a model that can't call tools will fail the SRE and BMAD steps.
• A GitHub account and a fine-grained PAT scoped to a single demo repo (for the BMAD crew).

---

Getting started

All environment-specific values live in one git-ignored file. Copy the template, fill it in, and hydrate it into your shell before applying anything:

git clone https://github.com/isItObservable/Kagent.git
cd Kagent
cp deploy/.env.example deploy/.env
$EDITOR deploy/.env            # OLLAMA_HOST, K8S_CLUSTER_NAME, DT_*, GITHUB_*, ...
set -a && . deploy/.env && set +a

deploy/.env is git-ignored — never commit real endpoints, tenants or tokens. A CI grep-gate fails the build on a stray IP, dt0c01. token or github_pat_.

1. Dynatrace Tenant — start a trial

If you don't already have one, create a free trial tenant at https://www.dynatrace.com/trial. Save your tenant API base URL (including the /api suffix) into deploy/.env:

# e.g. https://abc12345.live.dynatrace.com/api
DT_ENDPOINT=https://YOUR_TENANT.live.dynatrace.com/api
DT_TENANT=YOUR_TENANT

2. Create the Dynatrace API Tokens

You need two tokens. Create them under Access Tokens in your Dynatrace tenant.

Operator token (DT_DATA_INGEST_TOKEN) — used by the Dynatrace Operator / DynaKube for Kubernetes monitoring. Scopes:

• Create ActiveGate tokens
• Read entities
• Read settings
• Write settings
• Access problem and event feed, metrics, and topology
• Read configuration
• Write configuration
• Paas integration - Installer download

Ingest data token (DT_API_TOKEN) — used by the gateway collector to export OTLP to Dynatrace. Scopes:

• Ingest metrics (metrics.ingest)
• Ingest logs (logs.ingest)
• Ingest OpenTelemetry traces (openTelemetryTrace.ingest)
• Ingest events (events.ingest)

Put both into deploy/.env:

DT_API_TOKEN=dt0c01.XXXX          # ingest (OTLP/metrics/logs) — used by the collector
DT_DATA_INGEST_TOKEN=dt0c01.YYYY  # DynaKube data-ingest — used by the operator

3. Spin up a k8s cluster

Two documented on-ramps — pick one. Everything downstream is identical once you have a kubeconfig. The model always stays external on your Ollama endpoint; the cluster never hosts the LLM. Full details live in deploy/provisioning/ — the exact commands are inlined below so you can run the episode without opening another file.

Option A — Cluster API + Proxmox (CAPMOX, primary)

The homelab reference cluster (observable-kagent). You need a management cluster with the CAPI + CAPMOX + in-cluster IPAM providers, a Proxmox API token with VM create/clone rights, and a CAPMOX-ready golden template (Ubuntu 24.04 + kubeadm — see the proxmox-template skill). Full walkthrough: deploy/provisioning/capi-proxmox.md.

# 3A.1 — provider + cluster inputs (RFC5737 example ranges — use your own LAN).
#         VIP / node pool / MetalLB pool / DHCP reserve MUST NOT overlap.
export PROXMOX_URL="https://your-proxmox:8006/api2/json"
export PROXMOX_TOKEN="user@pam!tokenid=xxxxxxxx-...-xxxxxxxxxxxx"
export CLUSTER_NAME="${K8S_CLUSTER_NAME}"          # e.g. observable-kagent
export KUBERNETES_VERSION="v1.31.4"
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=3
export CONTROL_PLANE_ENDPOINT_IP="192.0.2.72"      # your VIP
export NODE_IP_RANGES="192.0.2.73-192.0.2.90"      # node pool
export GATEWAY="192.0.2.1"; export IP_PREFIX=24; export DNS_SERVERS="192.0.2.1"
export CP_DISK_STORAGE="local-lvm" # etcd fsync (board policy)
export WORKER_DISK_STORAGE="zetlab-vmdisks"        # shared NFS for live-migration
export TEMPLATE_ID=9000; export PROXMOX_SOURCENODE="pve1"

# 3A.2 — initialize CAPI providers on the MANAGEMENT cluster (once).
clusterctl init --infrastructure proxmox --ipam in-cluster

# 3A.3 — generate + apply the workload-cluster manifests.
clusterctl generate cluster "${CLUSTER_NAME}" \
  --infrastructure proxmox \
  --kubernetes-version "${KUBERNETES_VERSION}" \
  --control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
  --worker-machine-count "${WORKER_MACHINE_COUNT}" \
  > "cluster-${CLUSTER_NAME}.yaml"
kubectl apply -f "cluster-${CLUSTER_NAME}.yaml"

# 3A.4 — wait, then pull the new cluster's kubeconfig.
clusterctl describe cluster "${CLUSTER_NAME}"
clusterctl get kubeconfig "${CLUSTER_NAME}" > "${CLUSTER_NAME}.kubeconfig"
export KUBECONFIG="$PWD/${CLUSTER_NAME}.kubeconfig"

# 3A.5 — install a CNI (Cilium) + a MetalLB pool from your reserved range.
#         (GKE users skip this — see Option B.) Then you are ready to deploy.

Serialized worker builds: zetlab-vmdisks (shared NFS) re-introduces cfs-lock serialization on parallel CAPI worker provisioning — expect workers to build one at a time. Dev clusters that prefer fast parallel rebuilds may set WORKER_DISK_STORAGE=local-lvm (opt-out).

Option B — GKE (managed, alternative)

No Proxmox? Bring a managed cluster; GKE ships a CNI + LoadBalancer already, so you skip Cilium/MetalLB. Full walkthrough: deploy/provisioning/gke.md.

# 3B.1 — cluster inputs.
export GCP_PROJECT="your-gcp-project"
export GKE_REGION="europe-west1"
export CLUSTER_NAME="${K8S_CLUSTER_NAME}"
export GKE_NODE_COUNT=3
export GKE_MACHINE_TYPE="e2-standard-4"            # 4 vCPU / 16 GB — headroom for otel-demo + kagent

# 3B.2 — create the cluster (Workload Identity on for clean metadata).
gcloud config set project "${GCP_PROJECT}"
gcloud container clusters create "${CLUSTER_NAME}" \
  --region "${GKE_REGION}" \
  --num-nodes "${GKE_NODE_COUNT}" \
  --machine-type "${GKE_MACHINE_TYPE}" \
  --cluster-version "1.31" \
  --workload-pool "${GCP_PROJECT}.svc.id.goog" \
  --release-channel stable

# 3B.3 — pull credentials into your kubeconfig.
gcloud container clusters get-credentials "${CLUSTER_NAME}" --region "${GKE_REGION}"

Either way, confirm kubectl is pointed at the new cluster before deploying:

kubectl cluster-info && kubectl config current-context

Deploy most of the components

One ordered, idempotent installer runs the entire platform stack (steps 00→09). Every step is kubectl apply + an explicit kubectl wait, so it is safe to re-run:

export KUBECONFIG=/path/to/${K8S_CLUSTER_NAME}.kubeconfig
./deploy/deploy-all.sh          # runs steps 00 → 09 in order
./deploy/validate.sh            # pass/fail acceptance checks

#	Script	What it installs
00	00-preflight.sh	CLI + cluster reachability + required-var checks
01	01-cert-manager.sh	cert-manager (OTel Operator webhook prerequisite)
02	02-otel-operator.sh	OpenTelemetry Operator
03	03-dynatrace-operator.sh	Dynatrace Operator v1.9.1, kubernetes-monitoring mode
04	04-istio-ambient.sh	Istio ambient mesh (ztunnel + istio-cni + istiod)
05	05-gateway-api.sh	Gateway API CRDs + GAMMA (mesh routing)
06	06-otel-demo.sh	OpenTelemetry Demo — standard chart, no Dynatrace auto-injection
07	07-kagent.sh	kagent (OTLP → gateway) + Ollama ModelConfig
08	08-collectors.sh	DaemonSet agent + StatefulSet gateway + Istio telemetry wiring
09	09-kagent-github-mcp.sh	kagent GitHub MCP ToolServer + github-mcp-token Secret

Resume or isolate a step: START_AT=06 ./deploy/deploy-all.sh or ONLY=08 ./deploy/deploy-all.sh.

Deploy the Workload Generator

The OpenTelemetry Demo ships its own load generator — it drives continuous, OTel-native traffic through the microservices so the traces, the gRPC-route demo and the dashboards all have live data. Step 06 deploys it as part of the demo; confirm it is running:

kubectl -n otel-demo get deploy load-generator frontend-proxy

---

Architecture: Two Collectors, One Egress

The architectural crux: two collectors live in the default namespace, and every workload points at them. The DaemonSet handles per-node ingest; the StatefulSet gateway aggregates and is the single door to Dynatrace — one credential surface.

                                  kagent OTLP + Istio ambient + otel-demo SDKs
        node logs/traces │                         │ OTLP
┌─────────────────────────┐   forward   ┌──────────▼───────────────────┐
│ otel-agent (DaemonSet)  │────OTLP────▶ │ otel-gateway (StatefulSet)   │──OTLP──▶ Dynatrace
│ per-node: hostmetrics,  │             │ cluster metrics, tail-sampling│  (Api-Token)
│ filelog, k8sattributes  │             │ SINGLE Dynatrace egress       │
└─────────────────────────┘             └──────────────────────────────┘
DynaKube ActiveGate (kubernetes-monitoring) ─────────────────────────────▶ Dynatrace (k8s metrics)

The OTel Operator names each collector Service <CR-name>-collector, so the CRs are named otel-agent and otel-gateway to yield exactly:

• DaemonSet (node logs/traces, host/kubelet metrics): otel-agent-collector.default.svc.cluster.local:4317
• StatefulSet gateway (metrics aggregation, tail sampling, sole Dynatrace egress): otel-gateway-collector.default.svc.cluster.local:4317

Wiring — every producer exports to the gateway:

• kagent OTLP → otel-gateway-collector.default:4317 (deploy/values/kagent-values.yaml).
• Istio ambient telemetry → the same gateway (deploy/values/istio-telemetry.yaml + meshConfig extension provider, patched in step 08).
• otel-demo app SDKs → the gateway (deploy/values/otel-demo-values.yaml).
• DaemonSet agent → forwards node logs/traces to the gateway (it never talks to Dynatrace).
• Only the gateway exports to Dynatrace, and it keeps all GenAI spans via tail-based sampling.

Why otel-demo stays OTel-native. The story is "OTel-native app → our collector → Dynatrace backend." We do not layer Dynatrace OneAgent on the demo workloads; DynaKube stays in kubernetes-monitoring mode so it enriches k8s entities without double-instrumenting the apps. This is the honest, portable demo.

---

Step-by-Step: The Kagent Walkthrough

Full hands-on companion in TUTORIAL.md. The three steps build on each other: a brain, then a brain with tools, then a crew of them shipping software.

Step 1 — The brain: a ModelConfig for Ollama

Every agent needs a model, declared as a ModelConfig. Prove the brain works before anything else. → manifests/01-modelconfig-mac-studio.yaml

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: mac-studio-ollama
  namespace: kagent
spec:
  model: qwen3.6:latest          # MUST support function/tool calling
  provider: Ollama
  ollama:
    host: http://${OLLAMA_HOST}  # hydrated from deploy/.env — no IP in git

envsubst < manifests/01-modelconfig-mac-studio.yaml | kubectl apply -f -
kubectl get modelconfig mac-studio-ollama -n kagent

An agent references the model by name — the field is modelConfig (a plain string), not modelConfigRef.

Step 2 — An SRE agent with MCP + a gRPC-route demo

An agent with a model but no tools can only talk. Tools come from MCP servers. We deploy an SRE networking agent whose tools include henrikrexed/mcp-k8s-networking (Gateway API / Istio route inspection) plus kagent's built-in read-only k8s tools.

# 2.1 — deploy + register the MCP server (RemoteMCPServer)
envsubst < manifests/sre/10-mcp-k8s-networking-server.yaml | kubectl apply -f -
kubectl -n kagent get remotemcpserver mcp-k8s-networking     # ACCEPTED True, status.tools non-empty

# 2.2 — deploy the SRE agent (toolNames explicitly enumerated — small-model discipline)
kubectl apply -f manifests/sre/11-sre-networking-agent.yaml
kubectl get agent sre-networking-agent -n kagent             # READY True, ACCEPTED True

The deterministic demo. Break a gRPC route in otel-demo, then let the agent diagnose it. The checkout GRPCRoute backendRef.port is set to 8080 (frontend HTTP) instead of 5050 (checkout gRPC) — gRPC/h2c to an HTTP/1 listener fails silently:

kubectl apply -f manifests/demos/grpc-misroute/broken.yaml

Ask the SRE agent (kagent UI or A2A): "Traffic to checkout is failing — what is broken in the network path?" It reads the GRPCRoute through mcp-k8s-networking, resolves backendRef to checkout:8080, compares it to the checkout Service (gRPC 5050), and reports backendRef.port 8080 should be 5050. Apply the fix and watch the otel-demo traces recover in Dynatrace:

kubectl apply -f manifests/demos/grpc-misroute/fix.yaml

Step 3 — A BMAD crew that ships a Todo app on GitHub

The payoff: orchestration. In kagent, an agent can be a tool of another agent — when one lists another as type: Agent, the orchestrator's LLM delegates over the A2A protocol. There is no Team/Ensemble CRD; the crew is the graph of references. Six BMAD specialists, each acting on the world through kagent's built-in GitHub MCP server, run the full software-delivery loop and build a Todo app.

Agent	BMAD role	GitHub tools (least-privilege)
bmad-analyst	Mary — requirements → issues	issues RW, repo RO
bmad-pm	John — milestone + plan	issues RW, repo RO
bmad-architect	Winston — design → dev-task issues	issues RW, repo RO
bmad-developer	Amelia — code → branch → PR	repo RW, PRs RW, issues RO
bmad-qa	Quinn — review the PR	PRs RW, issues RW
bmad-orchestrator	BMad Master — conduct the chain	type: Agent → the five above

No agent gets a merge_* tool — merge stays a human/CI step, so the crew can never merge its own PR. One token, one repo = the entire blast radius.

# 3.1 — install the GitHub MCP server (fine-grained PAT: Contents/Issues/Pull requests RW)
kubectl -n kagent create secret generic github-mcp-token --from-literal=token=$GITHUB_TOKEN
helm install github-mcp oci://ghcr.io/kagent-dev/kagent/helm/github-mcp-server \
  -n kagent -f manifests/bmad-crew/values-github-mcp.yaml
kubectl -n kagent get toolserver                              # each Accepted, status.tools non-empty

# 3.2 — deploy the crew
kubectl apply -k manifests/bmad-crew
kubectl get agents -n kagent -l app.kubernetes.io/part-of=bmad-crew   # all six READY True

# 3.3 — one kickoff to the orchestrator runs the whole chain (see TUTORIAL.md §6.3)

Expected artifacts on ${GITHUB_DEMO_REPO}: requirement issues → a "Todo app v1" milestone → dev-task issues → a feat/todo-app branch + PR → a QA review.

kagent vs. the Sympozium Ensemble. Same crew, same Todo-app workflow, opposite construction. Sympozium uses one Ensemble CRD that generates N agents; kagent is a reference-based graph of independently-authored agents that delegate over A2A. Ensemble = declarative roster; kagent = compositional graph.

---

Testing / Validating the Deployment

1. Acceptance checks

deploy/validate.sh runs a non-destructive pass/fail sweep mirroring the acceptance criteria — component health, both collector Services, the gateway/agent rollouts, that kagent's OTLP endpoint points at otel-gateway-collector:4317, and that otel-demo is serving traffic:

./deploy/validate.sh

2. Confirm the collector wiring by hand

kubectl -n default get svc otel-agent-collector otel-gateway-collector
kubectl -n default rollout status statefulset/otel-gateway-collector
kubectl -n default rollout status daemonset/otel-agent-collector
kubectl -n kagent get deploy kagent-controller -o yaml | grep otel-gateway-collector

3. Agents are healthy and reachable

kubectl get agents -n kagent                                  # READY True, ACCEPTED True
kubectl -n kagent get remotemcpserver,toolserver             # Accepted, status.tools non-empty

kagent exposes each agent as an A2A JSON-RPC endpoint on the controller (:8083), or use the kagent UI at kagent-ui:8080. A quick single-agent smoke test ("Inspect the cluster with your tools and name three namespaces you observed.") should return real cluster state — proof the model + MCP loop works.

---

Observability Reference: gen_ai.* Signals & Dashboards

kagent emits GenAI spans (gen_ai.*, ADK gcp.vertex.agent) and logs — there are no kagent OTLP metrics, so every efficiency signal is span-derived. The span tree is invoke_agent → call_llm → generate_content (+ execute_tool); crew work adds A2A delegation hops between agents.

Two Dynatrace dashboards (dashboards/) tell the story on live telemetry:

Dashboard	ID	What it shows
kagent Health	4b46b649	engine up, runs succeeding vs. erroring, pod/workload resource health (DynaKube k8s metrics)
kagent Agentic Efficiency	dfe6dfc4	token usage (gen_ai.usage.*), tool calls, A2A delegation hops, latency per turn — all span-derived

Both are templated on ${DT_TENANT} / ${K8S_CLUSTER_NAME} — no tenant baked in. Deploy per dashboards/README.md.

Signal notes (dashboards/EFFICIENCY-SIGNALS.md):

• Tokens: gen_ai.usage.* lands on both call_llm and generate_content spans — sum on one to avoid a 2× double-count. Deduped, a typical crew run is ~10,659 tokens/run.
• Model attribution: filter on gen_ai.request.model (the real qwen3.6), not gen_ai.system (ADK labels the provider gemini).
• Latency: run p50 ≈ 62s / p90 ≈ 176s; LLM call p50 ≈ 35s / p90 ≈ 102s (local model, multi-hop).
• Delegation and memory panels stay empty on single-agent captures — they populate once the BMAD crew runs.

---

Resources

• 📺 Is It Observable — the episode: https://www.youtube.com/@Isitobservable · channel
• kagent — docs & concepts: https://kagent.dev/docs/kagent/concepts · repo: https://github.com/kagent-dev/kagent
• kagent GitHub MCP server: contrib/tools/github-mcp-server in the kagent repo
• mcp-k8s-networking (SRE agent tools): https://github.com/henrikrexed/mcp-k8s-networking
• OpenTelemetry Operator: https://opentelemetry.io/docs/kubernetes/operator/
• OpenTelemetry Demo: https://opentelemetry.io/docs/demo/
• OTel GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/
• Dynatrace — trial: https://www.dynatrace.com/trial · OTLP ingest: https://docs.dynatrace.com/docs/ingest-from/opentelemetry
• Cluster API (provisioning): https://cluster-api.sigs.k8s.io
• Istio ambient mesh: https://istio.io/latest/docs/ambient/

---

This repo ships no environment-specific values — every IP, tenant and token lives in deploy/.env (git-ignored). Copy deploy/.env.example, fill it in, and run it in your own cluster.