test(recall): TDD spec — usage-metering OSS seam, COMPLETE (premium#743)#888
Open
laynepenney wants to merge 2 commits into
Open
test(recall): TDD spec — usage-metering OSS seam, COMPLETE (premium#743)#888laynepenney wants to merge 2 commits into
laynepenney wants to merge 2 commits into
Conversation
…m#743) Failing tests for the foundational third of the OSS seam half of the metering substrate (config/design/premium-metering-substrate-2026-07-15.md): - UsageEvent schema (identity-free by construction: no agent_id/org_id/team_id field can exist on it; counts never money: no cost/price field either), register_usage_sink/emit_usage_event registration seam, no-op default, optional local JSONL debug sink -- tests/recall/test_usage_events.py - ModelClient.chat() additive usage_out extension (base + vLLM + MLX): real tokens_in/tokens_out are captured from data the underlying engine already computes internally and currently discards (vLLM RequestOutput.token_ids, MLX stream_generate's response.prompt_tokens/.generation_tokens) -- tests/test_model_client_usage_out.py. vLLM tests use importorskip (Modal- cloud-only dependency, not present in every dev venv). - Tap point 1 (_make_recall_infer): one infer UsageEvent per call, stage- tagged in detail (B2/B4-compose/etc, defaulting to B1 for synapt-extract's own request shape), real tokens_in/out via usage_out. cached_tokens is STRUCTURALLY None here and a test pins that explicitly -- self-hosted MLX/ vLLM compute has no cache-read/write billing tier; that's a separate, future, not-yet-tapped surface (the agents' own hosted-API sessions), not this one -- tests/recall/test_consolidate_usage_tap.py Remaining for part 2: memory-op taps (recall_save/search/quick), consolidation- stage taps (one consolidate_stage event per B1-B4 per cluster), and this repo's half of the calibration mechanism (tap-1 calibrates against the backend's own reported counts, not the 84-day corpus -- that corpus belongs to the future hosted-API tap). All 3 files RED for the right reason (ModuleNotFoundError on synapt.recall.usage for the seam+tap files; explicit signature/behavior assertions for the ModelClient file). 2472 pre-existing tests collect and are unaffected. Premium boundary: recall is OSS (neutral usage-count emission seam, no identity resolution, no aggregation, no cost math -- all premium's job).
laynepenney
force-pushed
the
spec/usage-metering-oss-seam
branch
from
July 16, 2026 09:52
87ca7ba to
ca0c351
Compare
…m#743) Completes the OSS-seam half of the metering substrate: - Tap point 2 (memory ops): recall_save -> mem_write, recall_search -> mem_search, recall_quick -> mem_read. Each emits exactly one UsageEvent per call through the same seam tap-1 uses -- no new registration mechanism, no identity resolution at this layer. session_ref resolves via a shared helper reading SYNAPT_AGENT_ID when set (same opaque-token precedent already established at core.py:157's TranscriptChunk.agent_id), "unattributed" otherwise -- tests/recall/test_memory_op_usage_taps.py - Tap point 3 (consolidation stages): one consolidate_stage event per B1-B4 stage per cluster inside _run_extract_path, giving the per-run cost decomposition dogfood analyses currently reconstruct by hand. tokens_in/ out/cached_tokens are null (this op is not itself a model call -- tap-1 already meters the individual infer calls inside each stage); duration_ms is real wall-clock per stage. A broken sink must never disrupt the pipeline, pinned directly against the real _run_extract_path result -- tests/recall/test_consolidate_stage_usage_taps.py Remaining: the premium collector (storage/identity-attribution/rate-cards/ active-agent/rollup -- synapt-private, separate PR per the boundary pair) and the rescoped tap-1 calibration mechanism (against the backend's own reported counts, not the 84-day corpus -- that corpus belongs to the future hosted-API tap). This PR is now the complete OSS-seam spec. Both new files RED for the right reason (ModuleNotFoundError on synapt.recall.usage). 2472 pre-existing tests unaffected. Premium boundary: recall is OSS (neutral usage-count emission seam, no identity resolution, no aggregation, no cost math).
laynepenney
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sprint-41 pull item (premium#743, Goal 1 — billable engagement overage). TDD
spec off the locked design (
config/design/premium-metering-substrate-2026-07-15.md).Now the complete OSS-seam half — started as part 1 (schema/seam/tap-1),
extended with part 2 (memory-op taps, consolidation-stage taps). The premium
collector (storage, identity attribution, rate cards, active-agent
determination, team rollup) is a separate PR in
synapt-privateper theboundary declaration pair.
UsageEventschema + registration seam (tests/recall/test_usage_events.py):identity-free by construction (structural test — no
agent_id/org_id/team_idfield can exist on the dataclass), counts never money,register_usage_sink/emit_usage_event, no-op default, optional localJSONL debug sink.
ModelClient.chat()additiveusage_outextension(
tests/test_model_client_usage_out.py, base + vLLM + MLX): both backends'underlying engine calls already compute real token counts internally and
discard them at the point
chat()extracts only completion text — theextension captures what's already there. Additive, non-breaking.
_make_recall_infer) (tests/recall/test_consolidate_usage_tap.py):one
inferevent per call, stage-tagged indetail, realtokens_in/tokens_out.cached_tokensstructurallyNonefor this self-hosted tap— the 84-day corpus's cache-read finding belongs to a different, future,
not-yet-tapped surface (the agents' own hosted-API sessions).
tests/recall/test_memory_op_usage_taps.py):recall_save→mem_write,recall_search→mem_search,recall_quick→
mem_read.session_refresolves via a shared helper readingSYNAPT_AGENT_IDwhen set (same opaque-token precedent alreadyestablished at
core.py:157'sTranscriptChunk.agent_id).tests/recall/test_consolidate_stage_usage_taps.py):one
consolidate_stageevent per B1–B4 stage per cluster inside_run_extract_path— the per-run cost decomposition dogfood analysescurrently reconstruct by hand.
tokens_in/out/cached_tokensnull (not amodel call itself — tap-1 already meters the individual infer calls inside
each stage);
duration_msreal. A broken sink must never disrupt thepipeline — pinned directly against the real
_run_extract_pathresult.Test plan
ModuleNotFoundErroronsynapt.recall.usage, which doesn't exist yet)pytest tests/ --cocollects 2472 pre-existing tests cleanly acrossboth commits — only the 4 new import-dependent files error, as expected
Closes #889
Premium boundary: recall is OSS — a neutral usage-count emission seam. No
identity resolution, no aggregation, no cost math. All of that is premium's
job (the collector PR, separate,
synapt-private).Reviewers: Opus + Sentinel (interstitial, standard two-reviewer TDD contract-
handoff — post a contract-read before implementing against this).
🤖 Generated with Claude Code