python-math #12: feat(delphi): Python math poller service (phase 1) — Clojure math-container replacement - #2625
Draft
jucor wants to merge 1 commit into
Draft
python-math #12: feat(delphi): Python math poller service (phase 1) — Clojure math-container replacement#2625jucor wants to merge 1 commit into
jucor wants to merge 1 commit into
Conversation
This was referenced Jul 18, 2026
Draft
jucor
marked this pull request as draft
July 18, 2026 02:43
This was referenced Jul 18, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces a new Python-based “math poller” service under polismath.poller to replace the production responsibilities of the legacy Clojure math container (phase 1), including Postgres polling (votes + moderation), per-zid serialized processing, and writing math outputs back to Postgres under a configurable math_env (shadow-mode by default).
Changes:
- Add
MathPollerService(dual watermark poll loops + per-zid compute pipeline + error dump/retry/park) and a per-zid serializedConversationWorkerPoolwith batch coalescing. - Add
MathWriterorchestration formath_main,math_bidtopid, andmath_ptptstatsusing a sharedmath_tickper cycle and Clojure-parity SQL inPostgresClient. - Add CLI entrypoint, docker-compose profile wiring, and a comprehensive unit + opt-in integration test suite.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| example.env | Documents configuration knobs for running the new poller, including shadow-mode defaults. |
| docker-compose.yml | Adds a profile-gated delphi-math-poller service that runs the poller CLI with shadow MATH_ENV. |
| delphi/pyproject.toml | Registers a new integration pytest marker for Postgres-backed tests. |
| delphi/scripts/math_poller.py | CLI wrapper to run the poller once or continuously with signal-based shutdown. |
| delphi/polismath/poller/init.py | Package-level architecture + contract documentation and public exports. |
| delphi/polismath/poller/service.py | Core poller service: watermark loops, dispatch, load-or-init, recompute/write, and error handling. |
| delphi/polismath/poller/worker_pool.py | Per-zid serialized worker pool with batch coalescing and bounded cross-zid concurrency. |
| delphi/polismath/poller/math_writer.py | Derives bidToPid/ptptstats envelopes and writes all tables with one shared math_tick. |
| delphi/tests/poller/init.py | Declares the new poller test package. |
| delphi/tests/poller/test_watermark.py | Unit tests for watermark advancement and allow/block filtering invariants. |
| delphi/tests/poller/test_service.py | Tests service dispatch behavior, watermark advancement, env engine-mode passthrough, and LRU eviction. |
| delphi/tests/poller/test_serialization.py | Thread-safety tests for per-zid serialization, cross-zid concurrency, and parking behavior. |
| delphi/tests/poller/test_coalescing.py | Tests batch coalescing rules (votes before moderation, arrival-order flattening). |
| delphi/tests/poller/test_math_writer.py | Tests bidToPid derivation, shared math_tick behavior, and SQL shape via a recorder client. |
| delphi/tests/poller/test_load_or_init.py | Tests from_dict restoration limits and the required full-history rebuild behavior. |
| delphi/tests/poller/test_error_path.py | Tests dump→retry→park circuit breaker behavior and “self-heal on new batch”. |
| delphi/tests/poller/test_integration_postgres.py | Opt-in end-to-end integration test that provisions Postgres, seeds data, runs poll→compute→write, and verifies restart-resume semantics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jucor
force-pushed
the
spr/edge/6324b652
branch
2 times, most recently
from
July 21, 2026 19:49
1791ca3 to
d05e53d
Compare
This was referenced Jul 25, 2026
Draft
Draft
Draft
python-math #43: docs(delphi): s7 goal state + journal — mode collapse executed, battery 20/20
#2672
Draft
Draft
Draft
This was referenced Jul 28, 2026
Draft
jucor
changed the title
3 tasks
… Clojure math-container replacement ## What The poller half of `MATH_POLLER_DESIGN.md` phase 1: a Python service that replaces the Clojure math container's production duties — vote/moderation watermark polling (a watermark is the timestamp up to which rows have already been processed), per-zid serialized compute (a zid is a conversation ID), and the four-table Postgres writes. ## Components - `polismath/poller/service.py`: `MathPollerService` — vote + moderation watermark loops mirroring `poller.clj:12-37` (strict `>`, max-of-batch advancement, `POLL_FROM_DAYS_AGO=10` boot window, allowlist/blocklist); per-zid engine chain `update_votes -> update_moderation -> recompute()` with load-or-init from `math_main` + full-history rebuild (`conv_man.clj:188-207` analog); error path dump -> one retry -> park (analog of Clojure's errorconv). - `polismath/poller/worker_pool.py`: per-zid FIFO + single-owner flag (strict serialization), `take-all!`/`split-batches` coalescing (votes before moderation), bounded cross-zid `ThreadPoolExecutor`. - `polismath/poller/math_writer.py`: one `math_tick` per cycle, shared across `math_main` / `math_bidtopid` / `math_ptptstats` (`conv_man.clj:158-169`); bidToPid positionally aligned with base-cluster ids ascending (`prep-bidToPid`, `conv_man.clj:35-40`; server `participants.ts:33-51`). - `scripts/math_poller.py` CLI (`--once` / run-forever, SIGTERM/SIGINT) + docker-compose service `delphi-math-poller` (profile `delphi-math`, shadow `MATH_ENV` by default) + `example.env` documentation. ## Testing `tests/poller/`: 44 unit tests (watermark, coalescing, serialization thread-safety, writer SQL incl. the `caching_tick` MAX+1 subquery + shared tick, bidToPid shape, allow/block, engine-mode passthrough, error path, load-or-init restoration boundary) + 1 opt-in integration test (throwaway `postgres:17` on port 5435): end-to-end poll -> compute -> write, shadow `math_env` isolation, shared `math_tick`, `caching_tick=1` on first write, restart-resumes. ## Cutover phasing Per `MATH_POLLER_DESIGN.md`, this ships SHADOW mode only: writes land under a `math_env` value the server does not read (`math_env` is the column that keys each math result row; the server only reads rows matching its own setting). The flip to serving traffic is gated on the blob-shape alignment work — the deltas between Python's `to_dict` and Clojure's `prep-main` catalogued by the H-B cross-language comparison (H-B = the replay harness's Clojure-side driver and comparer, commit 8 in this series). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> commit-id:18c7048f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The poller half of
MATH_POLLER_DESIGN.mdphase 1: a Python service that replaces the Clojure math container's production duties — vote/moderation watermark polling (a watermark is the timestamp up to which rows have already been processed), per-zid serialized compute (a zid is a conversation ID), and the four-table Postgres writes.Components
polismath/poller/service.py:MathPollerService— vote + moderation watermark loops mirroringpoller.clj:12-37(strict>, max-of-batch advancement,POLL_FROM_DAYS_AGO=10boot window, allowlist/blocklist); per-zid engine chainupdate_votes -> update_moderation -> recompute()with load-or-init frommath_main+ full-history rebuild (conv_man.clj:188-207analog); error path dump -> one retry -> park (analog of Clojure's errorconv).polismath/poller/worker_pool.py: per-zid FIFO + single-owner flag (strict serialization),take-all!/split-batchescoalescing (votes before moderation), bounded cross-zidThreadPoolExecutor.polismath/poller/math_writer.py: onemath_tickper cycle, shared acrossmath_main/math_bidtopid/math_ptptstats(conv_man.clj:158-169); bidToPid positionally aligned with base-cluster ids ascending (prep-bidToPid,conv_man.clj:35-40; serverparticipants.ts:33-51).scripts/math_poller.pyCLI (--once/ run-forever, SIGTERM/SIGINT) + docker-compose servicedelphi-math-poller(profiledelphi-math, shadowMATH_ENVby default) +example.envdocumentation.Testing
tests/poller/: 44 unit tests (watermark, coalescing, serialization thread-safety, writer SQL incl. thecaching_tickMAX+1 subquery + shared tick, bidToPid shape, allow/block, engine-mode passthrough, error path, load-or-init restoration boundary) + 1 opt-in integration test (throwawaypostgres:17on port 5435): end-to-end poll -> compute -> write, shadowmath_envisolation, sharedmath_tick,caching_tick=1on first write, restart-resumes.Cutover phasing
Per
MATH_POLLER_DESIGN.md, this ships SHADOW mode only: writes land under amath_envvalue the server does not read (math_envis the column that keys each math result row; the server only reads rows matching its own setting). The flip to serving traffic is gated on the blob-shape alignment work — the deltas between Python'sto_dictand Clojure'sprep-maincatalogued by the H-B cross-language comparison (H-B = the replay harness's Clojure-side driver and comparer, commit 8 in this series).Co-Authored-By: Claude Fable 5 noreply@anthropic.com
commit-id:18c7048f
Stack: