feat(python/sedonadb-geopandas): GeoPandas-compatible API package (experimental) by jiayuasu · Pull Request #1052 · apache/sedona-db

jiayuasu · 2026-07-15T08:01:52Z

An experimental GeoPandas-compatible API on top of SedonaDB, as a new pure-Python package sedonadb-geopandas (structured like sedonadb-expr). It provides GeoDataFrame / GeoSeries wrappers whose methods mirror GeoPandas but delegate to lazy SedonaDB expressions via the .geo accessor.

import geopandas
import sedonadb_geopandas as sgpd

gdf = sgpd.from_geopandas(geopandas.read_file("cities.geojson"))
big = gdf[gdf["pop"] > 1_000_000]      # boolean-mask filter
buf = gdf.geometry.buffer(0.5).area    # element-wise .geo operations
web = gdf.to_crs("EPSG:3857")          # reproject (CRS tracked through)
out = web.to_geopandas()               # escape hatch -> real GeoDataFrame

Why this exists

Two motivations, neither of which is "GeoPandas compatibility as a headline feature":

1. Consistency across Apache Sedona engines. Sedona already ships a GeoPandas API for Spark (sedona.spark.geopandas, with 39 GeoSeries and 10 GeoDataFrame functions). Someone moving between SedonaSpark and SedonaDB should meet a recognizably similar API — the same motivation as aligning Python, R, and SQL elsewhere in the project. This package deliberately mirrors the Spark side's conventions:

the sgpd import alias (parallel to GeoPandas' gpd),
GeoDataFrame / GeoSeries naming,
geometry operations implemented on GeoSeries and delegated from the active geometry column,
to_geopandas() for conversion back,

and later slices will match its signatures (for example sjoin(other, predicate=..., distance=...), plus bounds, envelope, boundary, geom_type, is_valid, to_wkt() / to_wkb()).

It is deliberately similar, not identical: the Spark implementation extends pyspark.pandas.DataFrame and so inherits a pandas index and a much wider surface, whereas this wraps a lazy relational SedonaDB frame with no row index. The Spark contributor guide's lazy-evaluation guidance applies here too — eagerly resolving the CRS on every GeoSeries construction would make otherwise-lazy calls eager, so .crs is read from the schema and repr() does not execute.

2. Exercising the core DataFrame API. Building a real API surface on top of SedonaDB's DataFrame and expression layer puts genuine load on it and surfaces gaps. It already has:

DataFrame.unnest was missing, needed for explode (feat(python/sedonadb): add DataFrame.unnest #1050, merged),
ST_Union_Agg returns NULL for point inputs, which dissolve depends on (ST_Union_Agg returns NULL for point inputs #1093),
a predicate join over two frames that both name their geometry column geometry requires an explicit .alias() per side.

Packaging

A standalone, opt-in package kept separate from the core sedonadb API (consistent with the direction in #791), co-located in this monorepo alongside sedonadb and sedonadb-expr so it shares CI and releases and can track the core API closely. It is not a dependency of sedonadb.

Dependencies

Hard dependencies are sedonadb, sedonadb-expr (for the .geo accessor), and pyarrow (for literals). geopandas is an optional extra, needed only for the from_geopandas() / to_geopandas() interop helpers, so installing this package does not pull in the full GeoPandas dependency footprint.

Scope (v1 "core slice")

from_geopandas() + a lazy default context (no explicit connect()), and to_geopandas().
GeoDataFrame: __getitem__ (boolean-mask filter, column access, column subset), .geometry, .crs, .columns, .to_crs, len.
Series / GeoSeries: comparison / & | ~ mask building; GeoSeries.buffer(), .centroid, .area, .length.

Deferred to later slices: sjoin / dissolve, __setitem__ (rebind), reductions, and broader .geo method coverage — converging on the sedona.spark.geopandas signatures listed above.

Intentional differences from GeoPandas (documented in the README)

Lazy, not eager — operations build a query; data materializes on to_geopandas(). repr() is deliberately cheap and does not execute (IDEs call it frequently); Jupyter gets a small _repr_html_ preview.
No row index / alignment — filters and (eventual) joins are relational, not index-aligned. Comparing against a pandas/numpy array-like raises rather than silently misaligning.
Immutable under the hood — "in-place" style operations return a new frame.
Plotting / arbitrary apply — use the to_geopandas() escape hatch.

A companion "Migrating from GeoPandas" cookbook documents the relational model behind each mapping.

Tests

pip install -e + 14 tests comparing behavior against GeoPandas: filter and boolean composition, buffer/area, centroid, to_crs, roundtrip, column types and subsets, lazy repr, array-like rejection, geometry-column validation, and frames with no geometry. Includes an assert_geopandas_expr_equal(gdf, op) harness that applies the same operation to a GeoPandas frame and to the wrapper and compares results, so a broader corpus of GeoPandas expressions can be added cheaply. ruff format and ruff check clean.

paleolimbot

No major concerns from my end...I don't generally think we gain a lot as a project by spending a lot of effort emulating the GeoPandas API or publicizing the fact that we did (many of the performance gains we're able to get are from the rethinking of pandas pipelines as general sequences of dataframe transformations, and I am not sure the much loved GeoPandas project sees attempts to duplicate its API kindly, although I haven't asked).

That said, this is not conceptually complicated and doesn't add complexity to the core of SedonaDB. If nothing else, developing it will help expose bugs and friction points by actual real live use of our DataFrame API. It needs some polishing but it's definitely worth merging as experimental.

paleolimbot · 2026-07-27T00:45:52Z

+    def __repr__(self):
+        return repr(self.to_geopandas())


You probably want to make this a bit cheaper since it gets called by IDEs and such (even possibly omitting any collect whatsoever). The jupyter rich-text version of this could probably do a limit + collect to GeoPandas.

Fixed — __repr__ no longer collects at all; it reports the columns and the active geometry column. Added a _repr_html_ for Jupyter that collects a bounded preview (up to 10 rows) via limit(), falling back to __repr__ if that fails. A test asserts the repr does not execute.

paleolimbot · 2026-07-27T00:48:09Z

+    def __repr__(self):
+        return repr(self.to_pandas())


You probably want to leave this as spitting out the expression (perhaps with a suggestion to collect and then extract the series)

Fixed — Series.__repr__ now shows the underlying expression rather than executing, with a hint to collect: <GeoSeries Expr(st_centroid(geometry)) (lazy; call .to_pandas())>.

paleolimbot · 2026-07-27T00:50:39Z

+        from sedonadb.expr import lit
+
+        transformed = self._df[self._geometry_name].geo.transform(lit(str(crs)))


Suggested change

from sedonadb.expr import lit

transformed = self._df[self._geometry_name].geo.transform(lit(str(crs)))

from sedonadb.expr import lit

transformed = self._df[self._geometry_name].geo.transform(lit(crs))

(in theory lit already knows about CRS objects)

Applied, thanks. Confirmed lit handles a plain string, a pyproj.CRS object, and an EPSG int, so the str() was both unnecessary and lossy for CRS objects.

paleolimbot · 2026-07-27T00:51:59Z

+    @property
+    def crs(self):
+        """The CRS of the active geometry column (via a zero-row materialization)."""
+        return self._df.limit(0).to_pandas().crs


I think you are better off with self._df.schema.field(self._geometry_name).type.crs here

Applied — .crs is now self._df.schema.field(self._geometry_name).type.crs, so it no longer executes anything. Returns None when the frame has no active geometry column.

paleolimbot · 2026-07-27T01:18:16Z

+@pytest.fixture
+def cities():
+    return gpd.GeoDataFrame(
+        {"name": ["A", "B", "C"], "pop": [100, 200, 300]},
+        geometry=gpd.GeoSeries.from_wkt(["POINT (0 0)", "POINT (1 1)", "POINT (5 5)"]),
+        crs="EPSG:4326",
+    )
+
+


These tests are great for now, but you will probably get some mileage out of a testing framework like

def assert_geopandas_expr_equal(gdf, op): sgdf = sgpd.from_geopandas(gdf) gdf_result = op(gdf) sgdf_result = op(sgdf) geopandas.assert_geodataframe_equal(sgdf_result, gdf_result)

...and throw a corpus of derived GeoPandas code at it. That will probably work better once you have a few more components in place.

Added, close to your sketch:

def assert_geopandas_expr_equal(gdf, op, *, sort_by): expected = op(gdf).sort_values(sort_by).reset_index(drop=True) got = op(sgpd.from_geopandas(gdf)).to_geopandas().sort_values(sort_by).reset_index(drop=True) assert_geodataframe_equal(got, expected, check_like=True, check_crs=False)

Sorting and resetting the index is needed because neither row order nor the index is preserved, and check_crs=False tolerates the EPSG:4326 to CRS84 normalization. Using it for the filter test now; agreed it earns its keep once there are more components to throw a corpus at.

You'll want to ensure you have check_crs=True (you definitely want to make sure this is one of the things that propagates). To avoid that normalization issue just avoid EPSG 4326 in your test data.

Fixed — the harness now uses check_crs=True; agreed that CRS propagation is exactly one of the things worth asserting rather than waving through.

One finding while verifying: check_crs=True passes with EPSG:4326 data too, including propagation through to_crs. The normalization I had worked around only shows up via SQL (ST_SetSRID(..., 4326) comes back as OGC:CRS84), not through the from_geopandas path, so the fixture did not actually need to change. To keep that honest either way I added a parametrized test over both a geographic (EPSG:4326) and a projected (EPSG:32633) source, plus a harness-driven to_crs("EPSG:3857") case that asserts propagation directly. Happy to move the fixtures off 4326 as well if relying on that seems fragile.

I don't see this used in this PR but I may have missed it (no need to to include it here if it's not used yet)

It is used — three call sites, though they are further down the file so easy to miss: test_filter_matches_geopandas, test_to_crs_matches_geopandas (which is what exercises CRS propagation now that check_crs=True), and a parametrized test_crs_propagates_from_projected_and_geographic over geographic and projected sources.

paleolimbot · 2026-07-27T01:21:39Z

+def _operand(other):
+    """Unwrap a Series to its expression; pass scalars through unchanged."""
+    return other._expr if isinstance(other, Series) else other


A few gotchas could happen here worth erroring for or catching somehow

An actual Pandas Series or numpy array. These will probably fail with no change because SedonaDB usually rejects literals that are len != 1 I think, but the error would be clearer if you check for these in advance. It's natural somebody might want to interact with a regular GeoPandas dataframe and this dataframe in the same code.

A SedonaDB expression you might want to handle as is here

Both handled. _operand now passes a SedonaDB Expr through unchanged, and rejects anything array-like (checked via __array__) up front with a clear message: there is no row alignment, so it suggests either staying within the frame or collecting with to_pandas() first. A test covers comparing against a real pandas Series.

I forgot about Literal which you may want to explicitly accept (allows lit() escape hatch for one-row-one-col GeoDataFrames, which is actually a quite useful way to specify a literal with a CRS)

Good catch — added. Literal is not an Expr subclass, so it was only surviving by falling through the final catch-all, which is exactly the kind of accident worth making explicit. _operand now accepts (Expr, Literal) up front, with a test using lit() as an operand, and the docstring notes the lit() escape hatch for carrying a CRS.

paleolimbot · 2026-07-27T01:22:41Z

+    """Load a ``geopandas.GeoDataFrame`` into a SedonaDB-backed ``GeoDataFrame``.
+
+    Args:
+        data: A ``geopandas.GeoDataFrame`` (or any object accepted by


The double quotes are rEST here but we're using markdown (also for other docs)

Fixed — converted the docstrings across the package from reST (double backticks, :class: roles) to markdown.

jiayuasu · 2026-07-27T05:59:23Z

Thanks for the thorough review, and for the green light to land this as experimental. Each comment has an inline reply with the specific change; the conclusions and decisions are here.

Framing. The PR description is rewritten around two motivations rather than GeoPandas compatibility as an end in itself.

The one I had underweighted is cross-engine consistency: Sedona already ships sedona.spark.geopandas, so someone moving between SedonaSpark and SedonaDB meeting a recognizably similar API seems worth something, in the same spirit as aligning Python, R, and SQL. The conventions converge more than expected — that implementation also uses the sgpd alias, puts geometry operations on GeoSeries and delegates from the active geometry column, and its contributor guide makes exactly the lazy-evaluation point about not eagerly resolving the CRS, which is what motivated the .crs and repr changes here. Later slices will match its signatures (sjoin(other, predicate=..., distance=...), bounds, envelope, geom_type, to_wkt(), and so on) rather than inventing new ones.

The second motivation is the one named in the review, and it is holding up: this exercise has already produced #1050 (unnest was genuinely missing, needed for explode) and #1093 (ST_Union_Agg returns NULL for point inputs, which dissolve depends on), plus the smaller friction that a predicate join between two frames that both name their geometry column geometry needs an explicit .alias() per side.

Deferred: generating method docstrings from the .geo accessor documentation — reasoning in that thread; worth a follow-up.

Open question, probably separate from this PR: for exact symmetry with sedona.spark.geopandas, the natural namespace would be sedona.db.geopandas. That lives in the apache-sedona package rather than here, so the import stays sedonadb_geopandas for now.

Tests went 9 → 14 with the new harness; ruff format and ruff check are clean.

paleolimbot

A few other things worth considering (or tracking) but this is looking good!

I think there a are some other more productive/maintenance reducing migration pathways into Spark but we can pursue those in parallel / guague interest in both as we go.

paleolimbot · 2026-07-28T16:38:33Z

+        # Column subset -> GeoDataFrame; re-derive the geometry column since the
+        # subset may have dropped it.
+        if isinstance(key, list):
+            return GeoDataFrame(self._df.select(*key))


It's worth checking what GeoPandas does for the case where the subset drops the geometry and make sure you match it here. It's either that the geometry column is preserved (sticky even if not included in the column subset), that the active geometry column is persisted if it still exists (or a regular pandas DataFrame is returned if it's dropped). I would be surprised if the geometry column is rederived (but you can test and find out).

Tested it, and you were right to be suspicious — re-deriving was wrong. GeoPandas behaves as follows:

subset that keeps the geometry column: GeoDataFrame, with the active column persisted (including a custom name like geom),

subset that drops it: a plain pandas DataFrame, whose .geometry raises AttributeError.

Now matched. The active column is persisted when it survives the subset and set to None when it does not, rather than re-derived — which mattered concretely: with an active geom alongside a geometry column, re-deriving silently switched the active column to geometry. There is no separate non-geo frame type here, so the dropped case returns a GeoDataFrame with no active geometry, where .geometry raises AttributeError just as it does in GeoPandas. Tests cover both, including the custom-name case.

paleolimbot · 2026-07-28T16:39:00Z

+        # Single column -> (Geo)Series.
+        if isinstance(key, str):
+            expr = self._df[key]
+            if key == self._geometry_name:
+                return GeoSeries(self._df, expr, key)
+            return Series(self._df, expr, key)


Do integers or slices work here?

Neither did anything useful — both fell through to a generic TypeError. Now handled explicitly, following GeoPandas where it is well defined:

an integer key raises KeyError, since in GeoPandas/pandas an integer is a column label rather than a row position (g[0] raises KeyError: 0),

a slice raises TypeError explaining that positional row slicing is not supported: there is no row index and no guaranteed row order, so g[0:2] cannot be answered faithfully.

For the bounded-rows case I added head(n), which is well defined as a limit and matches the method that already exists on the Spark side, with a docstring noting that which rows come back is not guaranteed without an ordering. Happy to make slices work as a best-effort limit instead if that trade seems better, but silently returning arbitrary rows for [0:2] felt like the wrong default.

paleolimbot · 2026-07-28T16:39:35Z

+        """Execute and return a `geopandas.GeoDataFrame` (or plain DataFrame)."""
+        return self._df.to_pandas()


Persist the custom geometry column name here?

Good catch, this was a real bug. With an active geom column alongside a geometry column, to_geopandas() returned a frame whose active geometry was geometry, because the conversion applies its own primary-geometry heuristic. It now carries the active column over explicitly (via set_geometry when it differs), with a test using exactly that two-geometry-column setup.

paleolimbot · 2026-07-28T16:40:43Z

+    # -- materialization ---------------------------------------------------
+    # -- materialization ---------------------------------------------------


Suggested change

# -- materialization ---------------------------------------------------

# -- materialization ---------------------------------------------------

# -- materialization ---------------------------------------------------

Fixed — that was a duplicated line I introduced when adding the section comment. Thanks for catching it.

paleolimbot · 2026-07-28T16:42:27Z

+    if isinstance(other, Series):
+        return other._expr


Sorry, I keep thinking of other things that could happen here. Do we want to allow Series that derive from different DataFrames to be mixed with each other? Is it reasonable to error for that case or will that be annoying?

No apology needed, this one was worth catching: it was silently wrong rather than annoying. Comparing Series from two different frames built a plan that returned zero rows instead of failing, so it now raises ValueError explaining that there is no row alignment and suggesting either referencing a single frame or joining first.

I do not think it will be annoying in practice, since a mask is almost always derived from the frame being filtered, and same-frame column-vs-column comparisons (gdf["pop"] > gdf["other"]) still work. Test added.

paleolimbot · 2026-07-28T16:44:43Z

+@pytest.fixture
+def cities():
+    return gpd.GeoDataFrame(
+        {"name": ["A", "B", "C"], "pop": [100, 200, 300]},
+        geometry=gpd.GeoSeries.from_wkt(["POINT (0 0)", "POINT (1 1)", "POINT (5 5)"]),
+        crs="EPSG:4326",
+    )
+
+


I don't see this used in this PR but I may have missed it (no need to to include it here if it's not used yet)

github-actions Bot requested a review from paleolimbot July 15, 2026 08:02

paleolimbot reviewed Jul 27, 2026

View reviewed changes

jiayuasu force-pushed the feature/geopandas-compat-package branch from eb4841d to bf18a1d Compare July 27, 2026 05:58

jiayuasu changed the title ~~feat(python/sedonadb-geopandas): GeoPandas-compatible API package (prototype)~~ feat(python/sedonadb-geopandas): GeoPandas-compatible API package (experimental) Jul 27, 2026

jiayuasu force-pushed the feature/geopandas-compat-package branch from bf18a1d to 6d29332 Compare July 28, 2026 06:23

paleolimbot approved these changes Jul 28, 2026

View reviewed changes

feat(python/sedonadb-geopandas): add GeoPandas-compatible API package

dd026bd

jiayuasu force-pushed the feature/geopandas-compat-package branch from 6d29332 to dd026bd Compare July 28, 2026 18:47

jiayuasu marked this pull request as ready for review July 29, 2026 05:18

jiayuasu merged commit daf65f2 into apache:main Jul 29, 2026
5 checks passed

jiayuasu mentioned this pull request Jul 29, 2026

ci(python/sedonadb-geopandas): run tests, build wheels, and verify in releases #1095

Draft

		from sedonadb.expr import lit

		transformed = self._df[self._geometry_name].geo.transform(lit(str(crs)))

		"""Execute and return a `geopandas.GeoDataFrame` (or plain DataFrame)."""
		return self._df.to_pandas()

		# -- materialization ---------------------------------------------------
		# -- materialization ---------------------------------------------------

Uh oh!

Conversation

jiayuasu commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this exists

Packaging

Dependencies

Scope (v1 "core slice")

Intentional differences from GeoPandas (documented in the README)

Tests

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jiayuasu commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

jiayuasu commented Jul 15, 2026 •

edited

Loading

jiayuasu commented Jul 27, 2026 •

edited

Loading