Skip to content

fix(ci): anchor arXiv digest window to announcement bands#80

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/arxiv-digest-announcement-window
Jul 15, 2026
Merged

fix(ci): anchor arXiv digest window to announcement bands#80
Jammy2211 merged 1 commit into
mainfrom
feature/arxiv-digest-announcement-window

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Closes #79.

The bug

The #papers digest silently and permanently dropped papers. arxiv_fetch.py filtered on <published> (v1 submission) within a rolling 24 h window (72 h on Mondays), but the arXiv API only indexes a paper once it is announced — one to three days after submission. Any paper whose announcement lag pushed its v1 timestamp outside the window could never be seen: not yet indexed on the run that covered its submission, already too old by the run that could first see it.

The reported case: 2607.12129 and 2607.12209, both strong-lensing, announced 2026-07-15. Submitted Mon 20:24 and 23:18 UTC — after the Mon 14:00 ET cut-off — so announced Tue 20:00 ET = Wed 00:00 UTC. Wednesday's 02:00 UTC run looked back 24 h to Tue 02:00 UTC and missed them by ~3 h. The keyword query was never at fault; a 72 h re-run returns both.

Monday's 72 h "weekend sweep" was the same bug at its worst: weekend submissions are not announced until Mon 20:00 ET — after Monday's run — and Tuesday's 24 h window had already moved past them.

The fix

Anchor the window to arXiv's announcement bands, derived from the run time (20:00 ET announcements Sun–Thu, 14:00 ET deadlines Mon–Fri) via zoneinfo so the ET→UTC offset stays correct across DST.

Submitted between (ET) Announced (ET)
Mon 14:00 → Tue 14:00 Tue 20:00
Tue 14:00 → Wed 14:00 Wed 20:00
Wed 14:00 → Thu 14:00 Thu 20:00
Thu 14:00 → Fri 14:00 Sun 20:00
Fri 14:00 → Mon 14:00 Mon 20:00

The 02:00 UTC cron lands at 21:00–22:00 ET the previous day, just after that day's announcement, so the Mon–Fri runs take the bands Thu→Fri, Fri→Mon, Mon→Tue, Tue→Wed, Wed→Thu. Their union is exactly one week with no overlap — disjoint and gapless, still with no cross-run state. The docstring's "gapless" claim is now actually true; it only ever held if papers were searchable at submission.

Scripts Changed

  • .github/scripts/arxiv_fetch.py — new pure announcement_band(); filter becomes band_start < published <= band_end; LOOKBACK_HOURS retained as the rolling-window escape hatch (test-fires, backfills); max_results 100 → 200 (the 3-day weekend band is the largest, and the cap applies before band selection); new mode key in the JSON. since/until keep their names so the Claude and Slack steps are untouched. Filter stays on <published> (v1), so v2 revisions still do not resurface.
  • .github/workflows/arxiv_papers.ymldow/72h/24h branch removed; --selftest step runs before the fetch so a DST or schedule regression fails loudly rather than posting an empty digest; timing comment rewritten, with a note not to reintroduce a rolling window.

No API changes; nothing outside this repo's CI is touched.

Verification

  • python3 .github/scripts/arxiv_fetch.py --selftest passes: both DST regimes (EDT 14:00 ET = 18:00 UTC, EST = 19:00 UTC), all five run weekdays, the Fri→Mon weekend seam, 3 h cron-slip stability, and both dropped papers pinned as an explicit regression case.
  • Live API run at a simulated Wed 2026-07-15 02:00 UTC computes band 2026-07-13T18:00Z .. 2026-07-14T18:00Z and recovers both papers. Monday's earlier papers are correctly excluded — they belong to the previous band, which Tuesday's run owns.
  • Override path intact: LOOKBACK_HOURS=72 still yields the legacy rolling window (mode=lookback).

Trade-offs

  • Band vs. state. A cross-run "seen IDs" store would also fix this, but trades statelessness for a committed state file and its drift modes. The band gives exact disjointness rather than dedup-after-overlap.
  • Residual. Papers announced out of band (submission held, cross-list added late) are still missed. Rare; a seen-ID store is the only real fix and is not worth it yet.
  • Jitter. GitHub cron only ever fires late. The band is stable from one 20:00 ET announcement to the next (~22 h of tolerance), far more than the rolling window really had.

Follow-up (not in this PR)

A backfill of already-dropped papers — one workflow_dispatch with lookback_hours=168. It posts a real catch-up digest to #papers and would duplicate anything already sent this week, so it is held for an explicit human go-ahead. Options are on #79.

🤖 Generated with Claude Code

The #papers digest silently and permanently dropped papers. arxiv_fetch.py
filtered on <published> (v1 submission) within a rolling 24h window (72h on
Mondays), but the arXiv API only indexes a paper once it is *announced*, one to
three days after submission. Any paper whose announcement lag pushed its v1
timestamp outside the window could never be seen: not yet indexed on the run
that covered its submission, too old by the run that could first see it.

2607.12129 and 2607.12209 (both strong-lensing, announced 2026-07-15) are the
reported case: submitted Mon 20:24 and 23:18 UTC, after the Mon 14:00 ET
cut-off, so announced Tue 20:00 ET = Wed 00:00 UTC. Wednesday's 02:00 UTC run
looked back 24h to Tue 02:00 UTC and missed them by ~3h. The keyword query was
never at fault; a 72h re-run returns both.

Monday's 72h "weekend sweep" was the same bug at its worst: weekend submissions
are not announced until Mon 20:00 ET, after Monday's run, and Tuesday's 24h
window had already moved past them.

Replace the rolling window with arXiv's announcement bands, derived from the run
time (20:00 ET announcements Sun-Thu, 14:00 ET deadlines Mon-Fri) via zoneinfo
so the ET->UTC offset stays right across DST. The Mon-Fri runs now take the
bands Thu->Fri, Fri->Mon, Mon->Tue, Tue->Wed, Wed->Thu: union of exactly one
week, no overlap, no cross-run state. The docstring's "gapless" claim is now
actually true — it only ever held if papers were searchable at submission.

LOOKBACK_HOURS still selects the old rolling window, for manual test-fires and
for backfilling what the old window dropped.

Verified: --selftest passes (both DST regimes, every run weekday, the weekend
seam, 3h cron slip, and both dropped papers as a regression case); a live API
run at a simulated Wed 02:00 UTC recovers both papers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Jammy2211 Jammy2211 merged commit 7d4e92d into main Jul 15, 2026
1 check failed
@Jammy2211 Jammy2211 deleted the feature/arxiv-digest-announcement-window branch July 15, 2026 08:27
Jammy2211 pushed a commit that referenced this pull request Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ci): anchor arXiv digest window to announcement bands

1 participant