perf: hot-path round — event amplification, fan-out, and per-request costs

[CMGS]

Six optimizations from a static performance analysis of the claim path and its neighbors. Each fix is paired with a benchmark committed immediately before it, so every claim below is an A/B on the same harness; commits alternate bench → fix.

Methodology. All numbers from a Linux container (golang:1.26) on a bare-metal bench host (AMD Ryzen 7 9700X, 16 threads), -count=10 (lookup: 6), compared with benchstat, p < 0.01 unless noted. Micro-benchmarks use the fake client / StaticInventorySource, which bounds what they can show (noted per item). Cluster-level claim-latency validation (poolbench p50/p95 at concurrency 20) is the follow-up acceptance run.

1. Pool events no longer re-enqueue bound claims (1463bbd)

mapWarmPoolToClaims re-enqueued every claim referencing a pool on every pool write; replenishment churn amplified into O(claims) no-op reconciles. Bound claims are now skipped — they are driven by their own Owns(Sandbox) watch.

BenchmarkMapWarmPoolToClaims (2500 claims, 50 unbound)	A	B
requests enqueued per pool event	2500	50 (−98%)

2. Warm-pool driver ignores its own status writes (e6eda85)

The global driver loop's writeStatus re-triggered its own For(SandboxWarmPool) watch; under claim churn the 5 s cadence degraded into a continuous back-to-back fleet PUT fan-out. GenerationChangedPredicate filters status-only events; spec edits, create/delete, NodeInventory events, and the 5 s requeue keep full coverage.

BenchmarkReconcileOnce (26 nodes, 4 pools): 0.10 ms + 81 KB per wake-up — the cost each filtered spurious wake no longer pays, plus the per-node PUT fan-out it no longer sends.

3. Warm pool: member-event filtering + copy-free member List (eb9ac04)

Owns(Sandbox) had no predicate: every member update — pod IP changes, unrelated conditions, annotations — triggered a full pool reconcile (one O(N) cache List that deep-copied all members, plus three O(N) passes). Now:

• updates filtered to what the reconcile reads: labels (LabelChangedPredicate), ownership, deletion, Ready flips, and generation bumps (keeps orphan blueprint re-vetting on spec edits);
• the member List is UnsafeDisableDeepCopy with deep-copy-on-write at the two mutation sites (launch-type backfill, orphan adoption) — same pattern the metrics collector already uses;
• SandboxTemplateRefHash hoisted out of the per-member loop;
• the stuck-member 5-minute grace gets its own RequeueAfter timer instead of riding on pool churn (also closes a pre-existing gap where a quiet pool never re-checked the deadline).

BenchmarkPoolMemberDeepCopy (2500 minimal members): 0.69 ms + 2.3 MB + 15k allocs per event — the informer-cache copy each pool event no longer pays (production members carry full pod specs, so the real saving is larger). The steady-state fake-harness pass is flat (−1.5%): the fake client copies regardless of the option, so the copy elimination doesn't show there by construction.

4. Store Get / warm-candidate sweeps fan out per node (8c272d4)

Get and warmCandidates (the node pick on every aggregated claim) walked the fleet sequentially with one full inventory decode per node; List already fanned out 16-way. Both now share List's bounded fan-out (extracted as fanOutNodes), and Get cancels on first hit.

ms/op	A	B
StoreGet 200 nodes × 2000	5.63	2.06 (−63%)
StoreWarmCandidates 200 × 2000	3.10	1.91 (−38%)
StoreGet 26 × 100	0.052	0.054 (+4%)
StoreWarmCandidates 26 × 100	0.037	0.053 (+42%)

Honest caveat: at the deployed 26-node scale the goroutine overhead costs ~15 µs per call — noise next to the ms-scale claim path, and the test double (typed DeepCopy) understates the win on the production path (reflection FromUnstructured per node).

5. e2b single-id verbs stop paying a fleet List (5d640f7, c3daa08)

Every get/delete/pause/connect/fork/snapshot/metrics request materialized and sorted the entire fleet to match one id. The store now resolves a claim id per node with first-hit cancel and materializes only the match (GetByClaimID, optional interface with the List fallback for other stores), and matchesID compares the DNS-safe rendering in place instead of allocating a rendered string per scanned entry.

BenchmarkLookupByID (200 × 2000 fleet, worst-case id)	A	B
latency	622 ms	2.09 ms (−99.7%)
allocated	1.76 GB	31 MB
allocations	6.0 M	837

6. v1alpha1 round-trip annotation slimmed (688ffd8)

The conversion webhook embedded a full serialized copy of the object in an annotation — near-doubling stored and wire size for every v1alpha1-touched Sandbox — to restore two replica fields. The annotation now carries exactly those fields; legacy full-object annotations still decode (pinned by test).

BenchmarkConvertRoundTrip (per object)	A	B
latency	11.9 µs	1.35 µs (−89%)
annotation size	810 B	47 B

Gates

• make lint (darwin + linux): 0 issues; asl ./... both GOOS: clean
• go test -race -count=1 ./...: pass
• Simplify review round applied (fan-out dedup, in-place id compare, predicate composition, shared fixtures)

Follow-up: cluster acceptance on the testbed (poolbench warm-claim p50/p95 at concurrency 1/5/20 against a 2500 pool) to attribute the event-amplification fixes (#1, #3) on the measured 316 ms @ c=20 curve.