Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -4470,3 +4470,16 @@ and bounded-distinct semantics (NaN rows, -0.0==0.0).
- Bench FULL prod shape (33422x783x2.0M, scratch/vectorized_full.json):
cold 28.15s, warm 26.66s — vs the ~31 min warm tick measured s6/s7
(~70x). The largest prod conversation now ticks in under 30s.

### Rename (Julien ruling, s7): the python poller is engine, not delphi

Compose service delphi-math-poller → **math-python**, profile delphi-math
→ **math-python**, env var DELPHI_MATH_ENV → **MATH_PYTHON_ENV** (default
math_env value 'delphi' → 'python'; free rename — no rows exist yet
anywhere). Living docs updated (runbook step 1, design §4, example.env);
journal history left as written. Deploy reality recorded in the runbook:
prod starts services BY NAME via scripts/after_install.sh role dispatch
(SERVICE_FROM_FILE: server|math|delphi; profiles gate dev only), prod
tracks branch `stable` — so the shadow wiring is one edit to the math
role's compose-up line, deliberately NOT made yet (Julien weighing
shadow vs clean replace).
15 changes: 10 additions & 5 deletions delphi/docs/CUTOVER_RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,17 +82,22 @@ equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19).

## Step 1 — shadow in prod (same day)

Infrastructure already in compose (#2625): service `delphi-math-poller`,
profile `delphi-math`, `MATH_ENV=${DELPHI_MATH_ENV:-delphi}` — distinct
Infrastructure already in compose (#2625; renamed s7 per Julien — the
math poller is engine, not delphi/UMAP): service `math-python`,
profile `math-python`, `MATH_ENV=${MATH_PYTHON_ENV:-python}` — distinct
from Clojure's env, rows invisible to the server (UNIQUE(zid, math_env)).

```
docker compose --profile delphi-math up -d delphi-math-poller
# env: DELPHI_MATH_ENV=delphi (engine has one path since the mode collapse)
docker compose --profile math-python up -d math-python
# env: MATH_PYTHON_ENV=python (engine has one path since the mode collapse)
# PROD NOTE (deploy-script reality, s7): prod instances start services BY
# NAME from scripts/after_install.sh per-role dispatch (profiles are a
# dev-only gate) — shadow on the math role = add `math-python` to its
# `docker-compose up -d math` line; prod tracks branch `stable`.
```

Verify within minutes:
- math_main rows appearing under math_env='delphi' with advancing
- math_main rows appearing under math_env='python' with advancing
caching_tick;
- no errorconv dumps / parked zids in the poller log;
- spot-compare a few active zids' blobs vs the clojure rows (the certify
Expand Down
5 changes: 3 additions & 2 deletions delphi/docs/MATH_POLLER_DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,9 @@ math_env string written), `VOTE_POLLING_INTERVAL` (ms, default 1000),

## 4. Cutover phases

1. **Shadow (this PR):** new compose service `delphi-math-poller` (profile-gated,
`--profile delphi-math`) running next to the Clojure `math` service, writing under a
1. **Shadow (this PR):** new compose service `math-python` (profile-gated,
`--profile math-python`; renamed s7 from delphi-math-poller) running
next to the Clojure `math` service, writing under a
DIFFERENT `math_env` (e.g. `MATH_ENV=delphi` while Clojure writes `prod`/`dev`).
`UNIQUE(zid, math_env)` makes the rows invisible to the prod server. No consumer
change, zero production risk.
Expand Down
8 changes: 4 additions & 4 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,12 +147,12 @@ services:

# Python math poller — the eventual replacement for the Clojure `math`
# container. Profile-gated so it only runs when explicitly requested
# (`--profile delphi-math`). Reuses the delphi build target and overrides the
# (`--profile math-python`). Reuses the delphi build target and overrides the
# command to run the poller CLI. Defaults to SHADOW mode: writes under a
# DISTINCT math_env (`delphi`) so its rows are invisible to the prod server
# (UNIQUE(zid, math_env)) — zero production risk while parity is validated.
# See delphi/docs/MATH_POLLER_DESIGN.md §4.
delphi-math-poller:
math-python:
image: 050917022930.dkr.ecr.us-east-1.amazonaws.com/polis/delphi:latest
build:
context: ./delphi
Expand All @@ -166,7 +166,7 @@ services:
- POSTGRES_CONNECT_TIMEOUT=${POSTGRES_CONNECT_TIMEOUT:-30}
- LOG_LEVEL=${DELPHI_LOG_LEVEL:-INFO}
# Shadow-mode math_env (distinct from the Clojure math service's MATH_ENV).
- MATH_ENV=${DELPHI_MATH_ENV:-delphi}
- MATH_ENV=${MATH_PYTHON_ENV:-python}
# Poll cadences (ms) and boot window (days).
- POLL_VOTE_INTERVAL_MS=${POLL_VOTE_INTERVAL_MS:-1000}
- POLL_MOD_INTERVAL_MS=${POLL_MOD_INTERVAL_MS:-1000}
Expand All @@ -193,7 +193,7 @@ services:
memory: ${DELPHI_POLLER_CONTAINER_MEMORY:-16g}
restart: unless-stopped
profiles:
- delphi-math
- math-python

postgres:
restart: always
Expand Down
6 changes: 3 additions & 3 deletions example.env
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,15 @@ LOCAL_SERVICES_DOCKER=true
# Leave empty for autodetection on AWS deployment. See delphi/DELPHI_AUTOSCALING_SETUP.md for configuring instance size in production.
INSTANCE_SIZE=dev

###### PYTHON MATH POLLER (delphi-math-poller, --profile delphi-math) ######
###### PYTHON MATH POLLER (math-python, --profile math-python) ######
# The Python replacement for the Clojure `math` container. Runs in SHADOW mode by
# default: writes math_main/math_bidtopid/math_ptptstats under a DISTINCT math_env
# so its rows stay invisible to the prod server (UNIQUE(zid, math_env)) while
# parity is validated. See delphi/docs/MATH_POLLER_DESIGN.md.
#
# math_env the poller writes under. Keep distinct from the Clojure MATH_ENV while
# shadowing; set equal to the server's MATH_ENV to cut over. Default: delphi
# DELPHI_MATH_ENV=delphi
# shadowing; set equal to the server's MATH_ENV to cut over. Default: python
# MATH_PYTHON_ENV=python
# Watermark boot window: start polling from N days ago. Default 10
# POLL_FROM_DAYS_AGO=10
# Poll cadences in ms. Defaults 1000/1000
Expand Down
Loading