fix(ci): anchor arXiv digest window to announcement bands

[Jammy2211]

Overview

The #papers digest silently drops strong-lensing papers. Two were missed on 2026-07-15 (2607.12129, a lensed-arc candidate in MACS J0308.9+2645; 2607.12209, the NGC 6505 Einstein ring OSN recovery test). The keyword query is not at fault — re-running arxiv_fetch.py with LOOKBACK_HOURS=72 returns both. The window is anchored to submission time, but the arXiv search API only indexes a paper once it is announced, 1–3 days later. Every run therefore has a permanently-dropped tail, worst on Mon/Tue.

Plan

• Replace the rolling "last N hours" window with arXiv's announcement bands, so each run selects exactly the batch that just became searchable.
• Compute the band from the run time: find the most recent 20:00 ET announcement (Sun–Thu), take the latest 14:00 ET submission deadline at or before it as the band end, and the previous deadline as the band start.
• Derive the ET↔UTC offset with zoneinfo rather than hard-coding it, so the band stays correct across DST.
• Drop the Mon-72h / weekday-24h branch in the workflow — the band logic subsumes it, including the 3-day Fri→Mon weekend band.
• Keep lookback_hours as a manual-dispatch escape hatch for test-fires and for a one-off backfill of already-dropped papers.
• Add a --selftest mode covering the weekday and DST cases, and correct the docstring's "gapless" rationale to name the announcement anchor.

Detailed implementation plan

Affected Repositories

• PyAutoMind (primary)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoMind	main	clean

Suggested branch: feature/arxiv-digest-announcement-window

Worktree: ~/Code/PyAutoLabs-wt/arxiv-digest-announcement-window/. A worktree is used deliberately so PyAutoMind's main checkout stays on main for registry operations — prompt_sync_push runs git add -A on whatever branch is checked out, so editing Mind's own CI on a branch in the main checkout would commit registry state to the feature branch.

The bug

.github/scripts/arxiv_fetch.py:77 keeps papers whose <published> (v1 submission timestamp) is within LOOKBACK_HOURS of now. Both missed papers were submitted Mon 2026-07-13 at 20:24 and 23:18 UTC — after the Mon 14:00 ET cut-off — so they entered the Tue 20:00 ET announcement batch and became searchable at 00:00 UTC Wed. Wednesday's 02:00 UTC run looked back 24 h to Tue 02:00 UTC; their Monday timestamps fall ~3 h the wrong side of that boundary. Tuesday's run could not have seen them either (not yet announced).

The docstring's claim that consecutive runs "cover disjoint day-bands (gapless)" only holds if papers are searchable at submission. They are not:

Run (02:00 UTC)	Window queried (by submission)	Batch actually announced by then	Gap
Tue (24 h)	Mon 02:00 → Tue 02:00	Fri 18:00 → Mon 18:00 UTC	misses Fri 18:00 → Mon 02:00
Wed–Fri (24 h)	prev 02:00 → 02:00	prev-day 18:00 → 18:00 UTC	misses the 18:00 → 02:00 tail
Mon (72 h)	Fri 02:00 → Mon 02:00	Thu 18:00 → Fri 18:00 UTC	sweeps three days not yet announced

Monday's 72 h "weekend sweep" is the clearest case: weekend submissions are not announced until Mon 20:00 ET — after Monday's run — and Tuesday's 24 h window has already moved past them.

arXiv's actual schedule

Announcements fire 20:00 ET, Sun–Thu. Submission deadlines are 14:00 ET, Mon–Fri. Each announcement covers submissions between the previous deadline and its own:

Submitted between (ET)	Announced (ET)
Mon 14:00 → Tue 14:00	Tue 20:00
Tue 14:00 → Wed 14:00	Wed 20:00
Wed 14:00 → Thu 14:00	Thu 20:00
Thu 14:00 → Fri 14:00	Sun 20:00
Fri 14:00 → Mon 14:00	Mon 20:00

The 02:00 UTC cron lands at 21:00–22:00 ET the previous day — just after that day's 20:00 ET announcement. So the Mon–Fri runs pick up the bands Thu→Fri, Fri→Mon, Mon→Tue, Tue→Wed, Wed→Thu. Their union is exactly one week with no overlap: disjoint and gapless, for real this time.

Implementation Steps

1. .github/scripts/arxiv_fetch.py — add a pure announcement_band(now: datetime) -> tuple[datetime, datetime]:
   • convert now to ZoneInfo("America/New_York");
   • walk back to the most recent 20:00 ET on a Sun–Thu at or before it;
   • band_end = latest Mon–Fri 14:00 ET at or before that announcement (Sunday announcements resolve to Fri 14:00);
   • band_start = the previous Mon–Fri 14:00 ET before band_end (Monday resolves back to Friday, giving the 3-day weekend band);
   • return both as UTC.
2. Replace the since/ts < since filter with band_start < ts <= band_end. Keep the <published> (v1) basis so v2 revisions still do not resurface.
3. Keep LOOKBACK_HOURS as an explicit override: when set and non-empty, use the legacy rolling window (manual test-fires and the backfill). When unset, use the band. Record which mode ran in the JSON.
4. Widen max_results 100 → 200 — the 3-day weekend band is the largest, and the cap is applied before band selection.
5. Extend the output JSON with mode, band_start, band_end alongside the existing keys; keep since/until populated so the downstream Claude and Slack steps are untouched.
6. .github/workflows/arxiv_papers.yml — delete the dow/72h/24h branch; pass LOOKBACK_HOURS only when the dispatch override is non-empty. Update the timing comment block to describe the announcement anchor.
7. Add --selftest to arxiv_fetch.py: table-driven asserts over each weekday run, both DST regimes (a July run at UTC-4 and a January run at UTC-5), the weekend band, and the two missed papers as a regression case (Mon 20:24 UTC must fall inside the band a Wed 02:00 UTC run computes). Run it as a workflow step before the fetch.
8. Rewrite the module docstring: the window is announcement-anchored; state why submission-anchored windows drop papers.

Backfill

After merge, one manual workflow_dispatch with lookback_hours=168 to sweep the week and recover what the current windows dropped. This runs the normal Claude + Slack path, so it posts a catch-up digest to #papers.

Testing

• python3 .github/scripts/arxiv_fetch.py --selftest — band maths, no network.
• A live run against the real API for the current band, checking both missed papers appear when the clock is faked to Wed 2026-07-15 02:00 UTC.
• workflow_dispatch test-fire with an override to confirm the override path is intact.

Key Files

• .github/scripts/arxiv_fetch.py — window logic, band function, selftest
• .github/workflows/arxiv_papers.yml — lookback branch removal, selftest step, timing comment

Trade-offs

• Band vs. state. A cross-run "seen IDs" store would also fix this, but it trades the current stateless design for a committed state file and its drift modes. The band keeps statelessness and gets exact disjointness rather than dedup-after-overlap.
• Residual. Papers announced out of band (submission on hold, a cross-list added later) are still missed. Rare; a seen-ID store is the only real fix and is not worth it yet.
• Jitter. GitHub cron only ever fires late. The band is stable anywhere from 20:00 ET to the next 20:00 ET, so it absorbs ~22 h of slip — far more tolerance than the ±0 the current rolling window really had.

Original Prompt

Click to expand starting prompt

arXiv #papers digest silently drops papers — window anchored to submission, not announcement

Type: bug
Target: pyautomind
Repos:

• PyAutoMind
  Difficulty: medium
  Autonomy: safe
  Priority: normal
  Status: formalised

The #papers digest missed two strong-lensing papers announced on 2026-07-15
(arXiv:2607.12129, a lensed-arc candidate in MACS J0308.9+2645; arXiv:2607.12209,
the NGC 6505 Einstein ring OSN recovery test). Both are matched by the keyword
query — re-running .github/scripts/arxiv_fetch.py with LOOKBACK_HOURS=72
returns both — so the recall-first QUERY is not at fault.

Root cause: arxiv_fetch.py:77 keeps papers whose <published> (v1 submission
timestamp) falls within LOOKBACK_HOURS of now, but the arXiv search API only
indexes a paper once it is announced, which is 1–3 days after submission.
Both papers were submitted Mon 2026-07-13 at 20:24 and 23:18 UTC — after the
Mon 14:00 ET cut-off — so they went into the Tue 20:00 ET announcement batch and
became searchable at 00:00 UTC Wed 2026-07-15. Wednesday's 02:00 UTC run looked
back 24 h to Tue 02:00 UTC; their Monday timestamps fall ~3 h the wrong side of
that boundary. Tuesday's run could not have seen them either (not yet announced).
Permanently dropped, with no error and no gap in the heartbeat.

The module docstring's claim that consecutive runs "cover disjoint day-bands
(gapless)" only holds if papers are searchable at submission time. They are not,
so every run has a dropped tail, and Monday/Tuesday are worst:

Run (02:00 UTC)	Window queried (by submission)	Batch actually announced by then	Gap
Tue (24 h)	Mon 02:00 → Tue 02:00	Fri 18:00 → Mon 18:00 UTC	misses Fri 18:00 → Mon 02:00
Wed–Fri (24 h)	prev 02:00 → 02:00	prev-day 18:00 → 18:00 UTC	misses the 18:00 → 02:00 tail
Mon (72 h)	Fri 02:00 → Mon 02:00	Thu 18:00 → Fri 18:00 UTC	sweeps three days not yet announced

Monday's 72 h "weekend sweep" is the clearest illustration: weekend submissions
are not announced until Mon 20:00 ET — after Monday's run — and Tuesday's 24 h
window has already moved past them.

Fix: anchor the window to arXiv's announcement bands rather than a rolling
"last N hours", preserving the existing stateless disjoint-band design and its
jitter tolerance; only the band edges change. At a 02:00 UTC run the freshly
announced batch is submissions from roughly 32 h to 8 h ago; the Monday run needs
the three-day Fri 14:00 ET → Mon 14:00 ET band. The ET→UTC offset shifts by an
hour across DST (14:00 ET = 18:00 UTC in EDT, 19:00 UTC in EST) and must be
computed, not hard-coded. Update the docstring's gapless-band rationale to state
the announcement anchor. Include a one-off backfill run to recover papers the
current windows have already dropped.

[Jammy2211]

Parked at ship sign-off (supervised cap) — needs your call

Implemented and verified on feature/arxiv-digest-announcement-window (409f746, pushed, no PR opened). Effective autonomy is supervised (prompt header safe, bug work-type cap supervised, min() → supervised), so this parks here rather than running to PR-open.

Scripts Changed

• .github/scripts/arxiv_fetch.py — announcement_band() derives the window from the run time against arXiv's schedule (20:00 ET announcements Sun–Thu, 14:00 ET deadlines Mon–Fri) via zoneinfo. Filter becomes band_start < published <= band_end. LOOKBACK_HOURS retained as the rolling-window escape hatch. max_results 100 → 200 (the 3-day weekend band is the largest, and the cap applies before band selection). New mode key in the JSON; since/until keep their names so the Claude and Slack steps are untouched.
• .github/workflows/arxiv_papers.yml — the dow/72h/24h branch is gone; a --selftest step runs before the fetch; timing comment rewritten to the announcement anchor, with a note not to reintroduce a rolling window.

No API changes; nothing outside this repo's CI is touched.

Verification

• --selftest passes: both DST regimes (EDT 14:00 ET = 18:00 UTC, EST = 19:00 UTC), all five run weekdays, the Fri→Mon weekend seam, 3 h cron slip stability, and both dropped papers as an explicit regression case.
• Live API run at a simulated Wed 2026-07-15 02:00 UTC computes the band 2026-07-13T18:00Z .. 2026-07-14T18:00Z and recovers both 2607.12129 and 2607.12209. Monday's earlier papers are correctly excluded — they belong to the previous band, which Tuesday's run owns.
• Override path re-checked: LOOKBACK_HOURS=72 still yields the legacy rolling window (mode=lookback, 6 papers).

Questions

1. Open the PR? That is the remaining ship step and the supervised cap holds it for you.
2. Backfill. The plan proposes one workflow_dispatch with lookback_hours=168 after merge. This posts a real catch-up digest to #papers — outward-facing, so I will not fire it without an explicit go-ahead. It would also re-post anything already sent this week, since the rolling window does not know what the digest has covered. Options: (a) fire it as planned and accept some duplicates, (b) pick a narrower window covering only the known gap, (c) skip the backfill and let the two papers here stand as the record.
3. Design check. I kept the stateless design and fixed the band edges, rather than adding a cross-run "seen IDs" store. The band gives exact disjointness instead of dedup-after-overlap, at the cost of still missing genuinely out-of-band announcements (submission held, cross-list added late). I judged a state file's drift modes not worth it yet — flag if you disagree.

Note

The bug cap is supervised because the calibration log holds too few bug rows to justify raising it. This run is a candidate row.

[Jammy2211]

Backfill DONE — both papers delivered to #papers

Human chose the narrow-window option. Fired run 29398900801 from main with lookback_hours=40: success, Slack POST ok.

The gap turned out smaller and cleaner than the plan assumed. Reconstructing what each broken run effectively saw (its window ∩ the index frontier at that moment) showed the weekend region was empty, so the only real gap was Mon 18:00 → Tue 02:00 UTC, containing exactly three papers:

Paper
2607.12081 (MOA-2020-BLG-108Lb)	gap — microlensing keyword false-positive
2607.12129 (gravitational arc, MACS J0308)	gap — reported miss
2607.12209 (NGC 6505 Einstein ring)	gap — reported miss

Because the last already-posted paper was Mon 13:40 and the first gap paper was Mon 18:58, a rolling window could be sized to land in that 5.3 h dead zone: W=40 → window opens Mon 15:54 UTC. Result: count=3, zero duplicates. Claude highlighted both lensing papers and correctly dropped the MOA microlensing one. So the full 168 h sweep was not needed — nothing was re-posted.

Two findings worth recording

1. The backfill did not need this PR. It runs the legacy rolling window, which main already has, so it was fired from main and is independent of the merge.
2. claude-code-action cannot run from a branch that edits its own workflow file. My first attempt dispatched from feature/arxiv-digest-announcement-window to backfill and smoke-test the new code in one go. It failed at the Claude step: "Workflow validation failed. The workflow file must exist and have identical content to the version on the repository's default branch." That is the action's own guard, not a bug in this PR. Nothing reached Slack (POST failed with no payload). Consequence: the band path cannot be exercised in real CI until after merge — the first scheduled run on main is its first live outing. The pre-merge evidence is the selftest (which did pass on the branch runner) plus the local live-API run.

Evidence from the failed branch run (useful anyway)

The steps before Claude all worked on the branch:

• --selftest step: PASS in CI, all 15 assertions.
• Fetch: mode=lookback band=2026-07-13T15:52..2026-07-15T07:52 count=3 — the new script's override path behaves identically to the old one.

Remaining

Only the merge, which stays human. After merge, the first scheduled run (Thu 02:00 UTC) takes band Tue 18:00 .. Wed 18:00 UTC — worth a glance to confirm the band path works live.

[Jammy2211]

Shipped — PR #80 merged (7d4e92d)

Human merged. The announcement-band window is live on main; selftest passes there. Remote branch deleted.

Both reported papers were delivered to #papers via the narrow backfill (run 29398900801, lookback_hours=40, count=3, zero duplicates) — 2607.12129 and 2607.12209 highlighted, the MOA microlensing false-positive correctly dropped.

Deliberately not test-fired post-merge: at this moment the band computes to Mon 18:00 → Tue 18:00 UTC — the exact three papers just backfilled — so a dispatch would re-post them. The first live outing of the band path is the Thu 02:00 UTC scheduled run (band Tue 18:00 .. Wed 18:00). Worth a glance, since per the workflow-validation trap above the band path could not be exercised in CI before merge.

Closing.