python-math #9: feat(math): Clojure-exact base-cluster lineage + warm start (PR-C) - #2622
Draft
jucor wants to merge 1 commit into
Draft
python-math #9: feat(math): Clojure-exact base-cluster lineage + warm start (PR-C)#2622jucor wants to merge 1 commit into
jucor wants to merge 1 commit into
Conversation
This was referenced Jul 18, 2026
jucor
marked this pull request as draft
July 18, 2026 02:29
This was referenced Jul 18, 2026
jucor
force-pushed
the
spr/edge/ca2af3c7
branch
2 times, most recently
from
July 18, 2026 13:35
6a3085f to
eb3dcb5
Compare
This was referenced Jul 18, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Clojure-faithful, lineage-preserving k-means implementation for the clojure-legacy engine mode and wires it into the Delphi Conversation clustering pipeline, enabling warm-start clustering across ticks while keeping the default improved path unchanged.
Changes:
- Introduces
legacy_kmeans.py, a Python port of Clojure’sclusters.cljk-means semantics (lineage IDs, clean-start behavior, weighted recentering). - Wires legacy warm-start clustering into
Conversation._compute_clusters(base + group levels) and threads previous-tick base/group clustering state viarecompute(). - Adds/updates unit + integration tests to validate lineage behavior and cold-start invariance (with float-tolerant center comparisons).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| delphi/polismath/pca_kmeans_rep/legacy_kmeans.py | New Clojure-faithful k-means + lineage implementation used in legacy engine mode. |
| delphi/polismath/conversation/conversation.py | Integrates legacy k-means warm-start at base/group levels; threads previous tick state through recompute. |
| delphi/tests/test_legacy_kmeans.py | Unit tests covering the ported Clojure k-means rules and lineage semantics. |
| delphi/tests/test_base_cluster_lineage.py | Integration tests validating warm-start threading and base-cluster id lineage across ticks. |
| delphi/tests/test_engine_mode.py | Updates cold-start invariance checks to allow float noise in cluster centers; adds structural equality check. |
| delphi/tests/test_group_k_smoother.py | Adds tests for legacy smoother behavior on degenerate ticks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This was referenced Jul 25, 2026
Draft
Draft
Draft
python-math #43: docs(delphi): s7 goal state + journal — mode collapse executed, battery 20/20
#2672
Draft
Draft
Draft
This was referenced Jul 28, 2026
Draft
jucor
changed the title
… start (PR-C)
## What
Ports the warm-start k-means that the Clojure engine threads across conversation-update ticks (`:last-clusters`, `clusters.clj:301-312` / `conversation.clj:403-445`) into a new module `polismath/pca_kmeans_rep/legacy_kmeans.py`, wired into the `clojure-legacy` engine mode (the setting that reproduces the old Clojure engine's behavior exactly) of `Conversation._compute_clusters`. Improved mode (the default) is byte-for-byte unchanged.
## Lineage semantics (all cited to `math/src/polismath/math/clusters.clj`)
- Cold init = first-k-distinct rows, ids `0..k-1` (`init-clusters`, `:55-65`), reusing the production `_get_first_k_distinct_centers` so cold init is identical to `kmeans_sklearn`'s first-k init.
- `clean-start-clusters` (`:230-277`): safe-recenter drop-vanished + fallback (`:171-191`), uniqify identical centers (`:220-227`), most-distal split loop with new ids = `(inc max id)` (`:202-217, :267`).
- Merge keeps the LARGER cluster's id, tie -> later argument (`merge-clusters`, `:194-199`).
- `cluster-step` drops empty clusters, ties -> later cluster (`:142-158`).
- `same-clustering` sorts centers and zip-TRUNCATES to the shorter list (`utils/zip`), reproduced deliberately (`:68-76`).
## Wiring (`conversation.py`)
- Base level: `legacy_kmeans` warm-started from `prev.base_clusters` (base-iters = 100). Base-cluster ids are stable across ticks; new participants get strictly larger ids.
- Group level: per-k `legacy_kmeans` over base-cluster CENTERS, weighted by base-cluster member counts (`:weights base-clusters-weights`, `conversation.clj:444`), warm-started from the previous per-k clusterings. Clojure passes the MISNAMED `:cluster-iters` key that kmeans ignores, so the group level runs kmeans' DEFAULT max-iters (20), not group-iters (100) — matched here.
- `self.group_clusterings` now holds id-carrying cluster dicts `{id, members, center}` in legacy mode (it was the `(labels, centers, member_lists, silhouette)` tuple), so the next tick can warm-start from it. Improved mode never writes it (stays `{}`).
- `recompute()` threads `prev_base_clusters` alongside the existing `prev_*` state.
## Cold-start invariance (HARD GATE)
MEASURED on the small `vw` test conversation dataset (67 in-conversation participants -> 67 singleton base clusters; groups k=2..5): legacy cold clustering is STRUCTURALLY bit-identical to improved at BOTH levels — same base/group memberships, ids, counts, and all downstream repness (representativeness stats), priorities, and group-votes. Cluster CENTER coordinates differ only at floating point (~1e-13: the ported weighted mean via `np.average` vs sklearn's centroid on identical memberships). The existing `test_engine_mode` cold-identity test is adjusted to compare numbers with a 1e-6 tolerance and everything else exactly, plus a new belt-and-braces test that drops center coordinates and asserts EXACT structural equality.
On near-duplicate projections (a degenerate synthetic set), legacy and improved DIVERGE at the base level too: legacy keeps each near-duplicate as its own exact-init singleton (Clojure-faithful) while sklearn's Lloyd collapses a pair and leaves an empty cluster. Base cold identity therefore holds only when projections are distinct (as on `vw`), and legacy is the more Clojure-faithful side.
## Testing
RED evidence (before this commit): `tests/test_legacy_kmeans.py` — ModuleNotFoundError (module absent); `tests/test_base_cluster_lineage.py` warm-start-threading tests — `legacy_kmeans` never called (0 calls), `group_clusterings` stored a tuple rather than id-carrying dicts, and a new participant received id 0 (no lineage) instead of a larger id.
GREEN: 22 unit tests (hand-derived from the Clojure rules) + 8 integration tests (incl. `vw` cross-mode cold identity) pass; full suite 511 passed / 17 skipped / 47 xfailed (was 480/17/47), zero regressions.
## Not ported
The agg-bucket-votes unknown-pid edge case (`conv_edge_cases_test.clj:110-127`) is not applicable — Python base-cluster members are always the current in-conversation participant ids (a subset of `rating_mat.index`) and `_compute_group_votes` already skips unknown pids defensively (`conversation.py:1481-1486`).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
commit-id:ca2af3c7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the warm-start k-means that the Clojure engine threads across conversation-update ticks (
:last-clusters,clusters.clj:301-312/conversation.clj:403-445) into a new modulepolismath/pca_kmeans_rep/legacy_kmeans.py, wired into theclojure-legacyengine mode (the setting that reproduces the old Clojure engine's behavior exactly) ofConversation._compute_clusters. Improved mode (the default) is byte-for-byte unchanged.Lineage semantics (all cited to
math/src/polismath/math/clusters.clj)0..k-1(init-clusters,:55-65), reusing the production_get_first_k_distinct_centersso cold init is identical tokmeans_sklearn's first-k init.clean-start-clusters(:230-277): safe-recenter drop-vanished + fallback (:171-191), uniqify identical centers (:220-227), most-distal split loop with new ids =(inc max id)(:202-217, :267).merge-clusters,:194-199).cluster-stepdrops empty clusters, ties -> later cluster (:142-158).same-clusteringsorts centers and zip-TRUNCATES to the shorter list (utils/zip), reproduced deliberately (:68-76).Wiring (
conversation.py)legacy_kmeanswarm-started fromprev.base_clusters(base-iters = 100). Base-cluster ids are stable across ticks; new participants get strictly larger ids.legacy_kmeansover base-cluster CENTERS, weighted by base-cluster member counts (:weights base-clusters-weights,conversation.clj:444), warm-started from the previous per-k clusterings. Clojure passes the MISNAMED:cluster-iterskey that kmeans ignores, so the group level runs kmeans' DEFAULT max-iters (20), not group-iters (100) — matched here.self.group_clusteringsnow holds id-carrying cluster dicts{id, members, center}in legacy mode (it was the(labels, centers, member_lists, silhouette)tuple), so the next tick can warm-start from it. Improved mode never writes it (stays{}).recompute()threadsprev_base_clustersalongside the existingprev_*state.Cold-start invariance (HARD GATE)
MEASURED on the small
vwtest conversation dataset (67 in-conversation participants -> 67 singleton base clusters; groups k=2..5): legacy cold clustering is STRUCTURALLY bit-identical to improved at BOTH levels — same base/group memberships, ids, counts, and all downstream repness (representativeness stats), priorities, and group-votes. Cluster CENTER coordinates differ only at floating point (~1e-13: the ported weighted mean vianp.averagevs sklearn's centroid on identical memberships). The existingtest_engine_modecold-identity test is adjusted to compare numbers with a 1e-6 tolerance and everything else exactly, plus a new belt-and-braces test that drops center coordinates and asserts EXACT structural equality.On near-duplicate projections (a degenerate synthetic set), legacy and improved DIVERGE at the base level too: legacy keeps each near-duplicate as its own exact-init singleton (Clojure-faithful) while sklearn's Lloyd collapses a pair and leaves an empty cluster. Base cold identity therefore holds only when projections are distinct (as on
vw), and legacy is the more Clojure-faithful side.Testing
RED evidence (before this commit):
tests/test_legacy_kmeans.py— ModuleNotFoundError (module absent);tests/test_base_cluster_lineage.pywarm-start-threading tests —legacy_kmeansnever called (0 calls),group_clusteringsstored a tuple rather than id-carrying dicts, and a new participant received id 0 (no lineage) instead of a larger id.GREEN: 22 unit tests (hand-derived from the Clojure rules) + 8 integration tests (incl.
vwcross-mode cold identity) pass; full suite 511 passed / 17 skipped / 47 xfailed (was 480/17/47), zero regressions.Not ported
The agg-bucket-votes unknown-pid edge case (
conv_edge_cases_test.clj:110-127) is not applicable — Python base-cluster members are always the current in-conversation participant ids (a subset ofrating_mat.index) and_compute_group_votesalready skips unknown pids defensively (conversation.py:1481-1486).Co-Authored-By: Claude Fable 5 noreply@anthropic.com
commit-id:ca2af3c7
Stack: