Skip to content

A2: freeze time-bounded adaptive read experiment#308

Draft
witbrock wants to merge 3 commits into
mainfrom
codex/JVNAUTOSCI-2598-a2-thin-read
Draft

A2: freeze time-bounded adaptive read experiment#308
witbrock wants to merge 3 commits into
mainfrom
codex/JVNAUTOSCI-2598-a2-thin-read

Conversation

@witbrock

Copy link
Copy Markdown
Member

Experimental freeze — do not merge. This draft exists to bind the A2 condition before unseen cases. It adds no production route and authorises no production deletion or cutover.

Why

A1 showed that its fixed two-tool-batch ceiling directly caused all three thin-candidate failures, while current Von produced one substantive answer from three cases in 52 minutes. A2 tests the more general hypothesis: ordinary read-only administrative and scientific work should leave route selection and recovery to the model, bounded by trusted identity, read-only capability and elapsed time rather than counts of model calls, tool batches, retries or routes.

A1 commit 207178f0 and draft PR #307 remain immutable evidence. This branch descends from that frozen condition but defines a new candidate and a fresh experiment.

What changed

  • Revised only the disposable scripts/try_thin_readonly_turn.py candidate and its case-independent tests; no production runtime file or route is changed.
  • Retained the frozen model, actor context and 52-tool read-only palette, with the read category rechecked at invocation.
  • Replaced count-based termination with a 180-second submitted-turn deadline and a 30-second final-synthesis window.
  • Allowed research model errors, incomplete responses, empty responses and read failures to recover within the remaining elapsed-time window, without a retry-count policy.
  • Gave all calls in one provider batch equal opportunity in parallel while propagating actor context.
  • Excluded read results that miss the research deadline from that turn, correlating every call with a typed result so final synthesis can proceed.
  • Retained and marked late completed model answers instead of silently hiding or discarding useful work.
  • Used stateless Responses continuation with the same tool definitions and tool_choice="none" for final synthesis.

The tests cover capability/protocol boundaries and observable outcomes. They do not prescribe a tool order, route, workflow identity, intent class or answer wording.

Frozen comparison protocol

After this commit/PR freeze, Michael supplies three fresh unseen low-risk read cases. Each exact case is submitted once to this candidate and once to current Von in a fresh chat: exactly six runs. Within-turn adaptation is allowed; after-result retry, rephrasing, tuning, replacement cases and a seventh run are not. The A1 cases and entities are forbidden from A2 prompts, fixtures and acceptance logic. Every outcome is retained.

The comparison baseline is origin/main at 95c6a4caad1f50b0845f2e654b97522cc586426b.

Validation

  • pytest -q tests/test_thin_readonly_turn.py — 12 passed
  • Ruff check and format check — passed
  • Python compile check — passed
  • Candidate preflight — 52/52 frozen tools present and currently registered read-only
  • Content-neutral real Responses wire preflight — function call completed; stateless correlated continuation with the same tools and tool_choice="none" completed with text
  • Independent strategic audit — no remaining unsupported constraint or production coupling found

Stopping point

Do not merge, route production traffic, delete production controller code, tune against evaluation cases or begin A3. A2 stops after the six frozen runs with evidence ready for Michael's H2 decision.

Copilot AI review requested due to automatic review settings July 25, 2026 21:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a disposable, non-production “A2” thin read-only turn candidate plus a focused test suite to freeze the experimental condition for a time-bounded adaptive read experiment (elapsed-time budget, tool-parallelism within a batch, and tool-free final synthesis).

Changes:

  • Adds scripts/try_thin_readonly_turn.py: a standalone A2 candidate runner using the OpenAI Responses API with a frozen read-only tool palette and elapsed-time termination (research deadline + final synthesis reserve).
  • Adds tests/test_thin_readonly_turn.py: tests for protocol boundaries (read-only enforcement, provider call-id correlation), elapsed-time budgeting, parallel within-batch tool execution, and late-answer marking.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
scripts/try_thin_readonly_turn.py Implements the time-bounded adaptive read-only A2 candidate, parallel tool execution, transcript emission, and read-only boundary enforcement.
tests/test_thin_readonly_turn.py Adds behavioural tests covering the candidate’s time-budget semantics, tool execution behaviour, and protocol/guardrail handling.

Comment on lines +409 to +413
if phase == "research":
if text:
transcript["last_partial_text"] = text
continue
transcript.update(
Comment on lines +496 to +499
executor = ThreadPoolExecutor(
max_workers=len(prepared),
thread_name_prefix="a2-read",
)
Comment on lines +515 to +519
remaining_research_seconds = max(0.0, research_deadline - clock())
completed_futures, _ = wait(
[future for _, _, _, future in pending],
timeout=remaining_research_seconds,
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants