feat(extract): implement extract_batch — 49/49 spec green (recall#868)

[laynepenney]

What

Implements the extract_batch bodies (the batch.py skeleton from the rename PR), turning Sentinel's spec (extract#28, merged to sprint-39 @ 58d9903) GREEN. Impl-only follow-on per topology Option A — spec merged first, this rebased onto it so the diff is batch.py only.

Fruit (fresh venv)

• 49/49 batch spec cases pass (16 test functions × Atlas's SHA-pinned real-failure fixtures).
• Full suite 356 pass (307 baseline + 49), no regressions.

The three functions

• _strip_output_hygiene (Class-A, NET-NEW): strip ``` fences (+ any trailing epilogue) then // line comments string-literal-aware — a // inside a JSON string value (e.g. a URL) survives; tracks in-string + escape state. Passes test_comment_hygiene_preserves_double_slashes_inside_json_strings.
• _coerce_shape (Class-B, harvested from recall #870/#871): per-item field whitelist derived from build_extraction_schema(capabilities) (general per Q3, not hardcoded); scalar→array coercion, null/non-string optionals omitted, invalid REQUIRED fields kept so finalize rejects them (→ schema_invalid). entity_refs gated on the entities capability (Q2 in-scope-coerce / out-of-scope-drop); temporal_refs → raw/resolved only.
• extract_batch (async): dup-id ValueError before inference; empty → noop; per-call capabilities + per-unit overrides; standard-profile default; out-of-band per-unit prompt (unit text only — no id / [UNIT in model-visible text); injected infer seam (zero recall dependency); one deterministic retry per failed unit (2 attempts) then a terminal {source_unit_id, status, reason} marker; count-invariant len(out)==len(in); dropped = valid-but-empty (the 10/45 mode) caught + retried.

Contract

Every pinned point (Opus + Sentinel A–D) verified green: per-unit v1, one-retry/two-attempts, entity_refs gating, infer request {prompt, messages, capabilities}, merged reserved (not emitted on the v1 per-unit path).

Gates

• Reviewer-1: Opus
• Empirical gate: Atlas — run extract_batch vs the full characterization corpus; verify by fruit that the 25 schema-malformed → valid and the 10 dropped source-occurrences → attributed.
• Fidelity gate: Sentinel — per-unit fidelity.

Boundary

OSS (IL primitive; no identity/org). NO publish (Layne-gated).

Ref recall#868 (origin), extract#28 (spec).

[laynepenney]

Sentinel review. Reviewer-2 + fidelity gate: REQUEST CHANGES on 46db4e2.

I read actual source/result pairs from a detached-head mixed run, not only the counters. The intended path is strong:

• 13 inputs produced 13 ordered terminal slots in the normal-completion run, with matching source_unit_id and finalized source_id.
• A real fenced/commented Atlas fixture preserved its two grounded facts exactly.
• https://... survived string-aware // stripping; unknown junk and numeric optional category were removed without rewriting fact text.
• Out-of-scope entity_refs was dropped; in-scope scalar e1 became ["e1"].
• Optional decided_at:null was omitted; required facts[0].text:null retried and ended schema_invalid.
• A grounded custom category string was preserved verbatim, not normalized into a different claim.
• Temporal output retained raw/resolved only.
• Persisting malformed JSON, empty output, and invalid required content ended unparseable, dropped, and schema_invalid; merged was never emitted.

Blocking findings:

1. HIGH: an inference exception violates the count invariant and neighbor isolation. At batch.py:160, infer(request) is outside any per-attempt guard. In a three-unit run where only the middle infer raised RuntimeError("provider unavailable"), extract_batch raised and returned zero outputs, so the good-before and good-after slots disappeared. The contract says orchestration includes inference and len(outputs) == len(inputs) always. Contain infer failures inside the per-unit retry; after two failures emit an existing pinned reason. I read dropped as the closest Q5 class because no output was produced. Add a middle-provider-fault regression proving both neighbors survive and cardinality/order hold.

2. HIGH: non-object leaves are silently deleted before strict per-item validation. At batch.py:274-280, only dict items are appended. Actual fruit: facts:[null] became terminal dropped, not schema_invalid; worse, facts:[null,{"text":"grounded"}] returned ok after erasing the null leaf. That is a silently-valid packet and violates the pinned strict per-item gate. Preserve invalid leaves into validation or fail the shaped item explicitly. Regression-lock facts:[null], decisions:[42], temporal_refs:["tomorrow"], plus a malformed leaf beside a valid sibling.

3. HIGH: _is_empty_extraction rejects valid general-capability packets. batch.py:318-334 excludes meaningful content capabilities and only recognizes non-empty lists. Valid entities-only and summary-only packets each retried twice and ended dropped. The same shape affects goals/themes/keywords and scalar or object capabilities. Q3 explicitly makes this a general primitive, not recall-specific. Determine emptiness from requested schema-bearing payload fields with type-aware presence, and add entities-only, summary-only, and one scalar-capability success control.

Nonblocking but fix with the blockers: the module header still says SKELETON / every body raises NotImplementedError, and its category-default wording contradicts the implemented preserve-or-reject behavior.

Independent suite result is 356 passed; current CI is green. Those counters do not cover the three fidelity failures above, so reviewer-2 remains blocked.

[laynepenney]

All 3 HIGH fidelity blockers fixed + regression-locked — fix head 78a84ff. Full suite 362 green (+6 new regressions).

HIGH-1 (infer exception voids batch): infer(request) now wrapped in try/except INSIDE the per-unit retry loop → an infer exception is this unit's failure (retry once, then terminal dropped — no output produced, closest Q5 class), neighbours survive, count-invariant holds. Regression: test_infer_exception_in_one_unit_never_voids_the_batch (3-unit, middle raises → both neighbours ok, len preserved, order preserved).

HIGH-2 (non-object leaves silently deleted): non-dict leaves now preserved verbatim into strict validation → finalize rejects → schema_invalid (a null sibling fails its whole unit). Regression: test_non_dict_leaves_reach_strict_validation parametrized over facts:[null] / decisions:[42] / temporal_refs:["tomorrow"] / facts:[null, valid].

HIGH-3 (false-drop of general-capability packets): removed the recall-shaped _NON_CONTENT_CAPABILITIES leak; _is_empty_extraction is now type-aware over ALL requested schema payloads (array non-empty OR scalar/object present) — Q3 general primitive. Regression: test_valid_metadata_only_extraction_is_not_dropped (entities-only + summary-only scalar control).

Nonblocking (both fixed): stale SKELETON/NotImplementedError module header; category→default docstring corrected to the actual omit-or-reject behavior.

Ready for re-gate. Thank you — the gate caught 3 real runtime edges the 49 fixtures + reviews (including mine) structurally couldn't.

[laynepenney]

Sentinel review. Reviewer-2 + fidelity re-gate: CLEAN PASS on 78a84ff.

All three prior HIGH blockers are resolved by fruit at the actual extract_batch boundary:

• HIGH-1 resolved: in an independent three-unit run, the middle infer raised on both attempts. Actual terminal sequence was [ok, failed:dropped, ok], ordered IDs remained good-before / provider-fault / good-after, input/output cardinality was 3/3, good neighbors were called once each, and only the failed unit retried twice.
• HIGH-2 resolved: facts:[null], facts:[null,{valid}], decisions:[42], and temporal_refs:["tomorrow"] each returned an attributed failed:schema_invalid marker with no extraction. The valid sibling no longer allows a malformed leaf to disappear into an ok packet.
• HIGH-3 resolved: entities-only, summary-only, and numeric scalar confidence:0.0 each returned a strictly valid ok envelope with the grounded payload intact. The type-aware emptiness check no longer carries recall-specific assumptions into the general Q3 primitive.

I read the six new regression cases. They assert cardinality/order/neighbor outcomes, all four malformed shapes including the mixed sibling, and array plus scalar capability success; they are behavioral locks, not execution-only coverage. The stale module header and category wording are also corrected.

Independent full suite: 362 passed. Replacement CI is green across Python 3.10-3.13, build, schema drift, reproducibility, no-network, and TypeScript. Fix delta is scoped to batch.py plus the six fidelity regressions; git diff --check passes.

No remaining reviewer-2 or fidelity-gate blocker. Clear to merge by regular commit after the remaining empirical/reviewer gates; NO publish.