Migrate to Elasticsearch 8.19, Python 3.12 and SQLAlchemy 2.0#2211
Open
Nayor wants to merge 54 commits into
Open
Migrate to Elasticsearch 8.19, Python 3.12 and SQLAlchemy 2.0#2211Nayor wants to merge 54 commits into
Nayor wants to merge 54 commits into
Conversation
Test de création de la branch remote sur un texte sans impact.
- Mapping Update (scripts and code) - Code update for deprecation update (fields, missings, ...) - Component update (elasticsearch, elasticsearch DSL) - Migration scripts and logstash engine / pipeline - CI scripts
- Mapping Update (scripts and code) - Code update for deprecation update (fields, missings, ...) - Component update (elasticsearch, elasticsearch DSL) - Migration scripts and logstash engine / pipeline - CI scripts
- Mapping Update (scripts and code) - Code update for deprecation update (fields, missings, ...) - Component update (elasticsearch, elasticsearch DSL) - Migration scripts and logstash engine / pipeline - CI scripts
- Mapping Update (scripts and code) - Code update for deprecation update (fields, missings, ...) - Component update (elasticsearch, elasticsearch DSL) - Migration scripts and logstash engine / pipeline - CI scripts
CI: - retirer dans l'installation de ES 5.6 les paramètres inconsistant + install plugin analysis-icu sur ES
CI: - retirer dans l'installation de ES 5.6 les paramètres inconsistant + install plugin analysis-icu sur ES
Script de création / mapping / settings des indices ES
- lint fixes
- CI/CD : ES Download version
- lint fixes
…-to-6.8' into elasticsearch-migration-from-5.6-to-6.8
- lint fixes
- lint fixes
- lint fixes
- lint fixes
- docker-api ES point
- Update settings es in config file for github-actions
- test_fill_index fix coquille
- update doc
- update mapping and code
- update mapping
- update mapping
- update mapping
- update mapping
- update script for data migration
- lint fixes
- artifact fix for ES download
- artifact fix for ES download - mapping - class - requirements.txt - docker definition
- artifact fix for ES download - mapping - class - requirements.txt - docker definition
- artifact fix for ES download - mapping - class - requirements.txt - docker definition - CI update - logstash data migration update
- templates
- CI elastic 8.22 update
- CI elastic 8.22 update
- CI elastic 8.22 update
- CI elastic 8.22 update
- CI elastic 8.22 update
- CI elastic 8.22 update
- CI elastic 8.22 update
- CI elastic 8.22 update
- CI elastic 8.22 update
- CI elastic 8.22 update
- CI elastic 8.22 - README.md Update
Merges the existing elasticsearch-migration-from-7.17-to-8.14 branch
(2.6->5.6->6.8->7.x->7.17->8.14, authored separately) into current
master, bumping the final target to elasticsearch/elasticsearch-dsl
8.19.x, and resolving conflicts from ~2 years of master drift.
Key changes carried over from that branch:
- DocType -> Document, String -> Text (elasticsearch_dsl 8.x API)
- ES mapping types removed (ES7+): documents now carry a
`c2corg_doc_type` field instead of relying on per-type `_type`
- One physical ES index per document type (c2corg_a, c2corg_b, ...),
queried through a common `c2corg_*` index pattern
- `Missing` query -> `Q("exists", ...)` wrapped in bool/must_not
- ES auth support (scheme/user/passwd) in search config
- CI updated to install/run ES 8.19.18 directly instead of ES 2.4.6
Cleaned up from the source branch: removed a leftover debug
pprint() call, dead commented-out code, and an accidental drop of
ACLDefault on DocumentRest.
Validated locally against real (anonymized) production data:
restored dump into ES-8-targeted sandbox, created all 10 per-type
indices, ran a full reindex from Postgres (1.8M documents), and
confirmed search queries return correct, type-filtered results.
The pycurl calls used to create/configure ES indices had no timeout set, so a stalled connection (e.g. under I/O contention) could hang the process indefinitely instead of failing and letting the caller retry. Observed this hang concretely while running the test suite locally: the process sat idle (near-0% CPU, no network I/O) for several minutes with no way to recover short of killing it.
config/env.sandbox is a throwaway env file pointing at local migration-rehearsal container hostnames, not meant to be committed.
- Bump base images (docker/Dockerfile, docker/Dockerfile.dev, .devcontainer/Dockerfile) from python:3.9 to python:3.12-slim-bookworm, and fix bullseye-only package name (libffi7 -> libffi8) for bookworm. - Bump CI to Python 3.12. - requirements.txt: SQLAlchemy 1.3.24 -> 2.0.51, geoalchemy2 0.4.2 -> 0.20.0. ColanderAlchemy temporarily points at a local, unpublished patch (branch sqlalchemy-2.0-support in the ColanderAlchemy repo) that relaxes its own SQLAlchemy pin to >=1.4 - needs write access to c2corg/ColanderAlchemy to publish for real. - models/__init__.py: declarative_base import moved from sqlalchemy.ext.declarative (pre-1.4) to sqlalchemy.orm. - ext/colander_ext.py: removed the geoalchemy2.compat import (buffer/bytes were just Python 2/3 shims; bytes is the builtin, buffer() is memoryview() on Python 3). - models/document.py, models/stoparea.py: removed the `management=True` Geometry() kwarg, which geoalchemy2 0.20 no longer supports (modern PostGIS uses typmod columns, not the old catalog-managed style). - models/association_views.py, area_association.py, topo_map_association.py, feed.py: ported old-style select([col1, col2]) calls to the SQLAlchemy 1.4+/2.0 select(col1, col2) signature; added .scalar_subquery() to two column_property() selects in feed.py per the 2.0 deprecation warning. Verified: `import c2corg_api.models` (all ~20 model files, every document type) now succeeds cleanly, and ColanderAlchemy schema generation (SQLAlchemySchemaNode direct instantiation, the pattern this codebase actually uses) produces correct field lists. Not yet done: the view layer (c2corg_api/views/) and scripts/ almost certainly have their own SQLAlchemy 1.3-era Query patterns that need the same kind of review; this commit only gets the model layer to import and construct schemas cleanly under SQLAlchemy 2.0.
Completes the Python 3.12 / SQLAlchemy 2.0 migration for c2corg_api/views and c2corg_api/scripts, found empirically by running the full test suite: - sqlalchemy.orm.util no longer exports with_polymorphic/aliased (moved to sqlalchemy.orm); sqlalchemy.sql.elements.not_ moved to top-level sqlalchemy. - String-based loader options (load_only, joinedload, contains_eager, undefer) are rejected outright in 2.0; switched to class-bound attributes across document_listings.py, document.py, document_changes.py, document_history.py, document_version.py, user_preferences.py, feed.py and scripts/es/sync.py. Route.locales needed .of_type(RouteLocale) when load_only-ing polymorphic locale columns. - Query.from_self() was removed in 2.0; rewrote the windowed_query/ column_windows pagination helper in models/utils.py using an explicit subquery. - collections.Mapping removed in Python 3.10+; use collections.abc.Mapping. - SQLAlchemy 1.4+ no longer implicitly adds order_by() columns to the SELECT list under .distinct(), causing InvalidColumnReference; fixed in waypoint.py/route.py's set_recent_outings and generalized get_first_column() to unpack Row objects positionally. - Several single-column query results were used directly as scalar ids without unpacking (silently worked in 1.3, "can't adapt type 'Row'" in 2.0); fixed in document_delete.py and scripts/es/sync.py. - geoalchemy2 0.20 routes use_typmod=False columns (needed on geom_detail to store mixed 2D/3D/4D geometries) through its legacy "managed" AddGeometryColumn() codepath, whose automatic spatial index creation collides with alembic's own index creation. Set spatial_index=False on that column in the init migration and create the index explicitly. - Leftover legacy select([...]) list-syntax + .as_scalar() in models/feed.py, missed by the earlier SQLAlchemy 2.0 select() sweep. - Document.update() mixed transient and persistent locale objects in the same loop without their FK set yet, letting an unrelated expired- attribute access trigger a premature autoflush that inserted a locale row before its document_id was set. Wrapped in DBSession.no_autoflush. Full pytest suite: 956 passed, 2 failed (958 total). The two failures are a PostGIS-side loss of the M coordinate on 4D geometry round-trips, reproducible only against the pg18 sandbox (PostGIS 3.6) used for the separate, not-yet-started database migration - not caused by this change.
geoalchemy2 binds a non-extended WKBElement by converting it through Shapely, which has no support for M (4th dimension) coordinates and silently drops them on write. wkbelement_from_geojson() now embeds the SRID directly in the WKB header (turning it into real EWKB), which geoalchemy2 detects as already-extended and binds as a hex string directly instead, preserving all dimensions. Fixes the last two failures from the SQLAlchemy 2.0 migration test run (test_ext_colander::test_save_and_load_4d, test_route::test_post_success_4d). Also replace an implicit scalar-subquery coercion in update_langs_of_changes() with an explicit .scalar_subquery() call, as recommended by the SQLAlchemy 2.0 deprecation warning.
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| Documentation | 10 minor |
| ErrorProne | 4 high |
| CodeStyle | 1 minor |
🟢 Metrics 7 complexity · 6 duplication
Metric Results Complexity 7 Duplication 6
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Nayor
force-pushed
the
python-3.12-migration
branch
from
July 20, 2026 21:17
c5aae62 to
aaccdb4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two chained infrastructure migrations, bundled in this branch since it was built on top of the (unpushed) ES migration branch:
c2corg_a,c2corg_w, ...) unified via ac2corg_*pattern with ac2corg_doc_typediscriminant field. Validated by reindexing 1.8M real anonymized production documents.models/,views/andscripts/, including:python:3.12-slim-bookwormSQLAlchemy==2.0.51,geoalchemy2==0.20.0load_only,joinedload,contains_eager,undefer) converted to class-bound attributes, as required by SQLAlchemy 2.0Query.from_self()removal worked around in the pagination helper (models/utils.py)select([...])legacy list-syntax and.as_scalar()calls updated toselect(...)/.scalar_subquery()Document.update()that was mixing transient and persistent locale objectsDuplicateTableonidx_documents_geometries_geom_detailduring the initial alembic migrationTest status
958 tests total, 958 passing.
Known blocker — do not merge yet
requirements.txtcurrently pinsColanderAlchemyto a local filesystem path (file:///.../ColanderAlchemy) because the upstreamc2corg/ColanderAlchemyfork needs SQLAlchemy 2.0 support (patched locally, branchsqlalchemy-2.0-support) but we don't have push access to publish it. This needs a real fork/publish location before merge.Not in scope
PostgreSQL migration to pg18/PostGIS 3.6 — separate, untouched chantier.