Support for geoparquet

[Libbum]

Description

The geoparquet 1.0 schema has been released recently. Since this a parquet based schema, there shouldn't be much work to formally support it.

Current situation

If I take the example file, I can load it with geopandas:

import geopandas
df = geopandas.read_parquet("example.parquet")
df
       pop_est      continent  ... gdp_md_est                                           geometry
0     889953.0        Oceania  ...       5496  MULTIPOLYGON (((180.00000 -16.06713, 180.00000...
1   58005463.0         Africa  ...      63177  POLYGON ((33.90371 -0.95000, 34.07262 -1.05982...
2     603253.0         Africa  ...        907  POLYGON ((-8.66559 27.65643, -8.66512 27.58948...
3   37589262.0  North America  ...    1736425  MULTIPOLYGON (((-122.84000 49.00000, -122.9742...
4  328239523.0  North America  ...   21433226  MULTIPOLYGON (((-122.84000 49.00000, -120.0000...

But attempting to write the table with delta-rs fails via pyarrow since pyarrow has no comprehension of the properties of the binary geometry column.

from deltalake import DeltaTable, write_deltalake
write_deltalake("./data/geoparquet_delta", df)
...
pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column geometry with type geometry')

An attempt at next steps

Using the geoarrow spec, which is implemented in both python and rust already, we can overcome the pyarrow error above.

• to delta: Just assume geometry is a binary blob.
• from delta: Convert the binary blob with geoarrow's to_geopandas method.

import geopandas
import geoarrow.pyarrow as ga
import pyarrow.parquet as pq
from deltalake import DeltaTable, write_deltalake

df = geopandas.read_parquet("example.parquet")
write_deltalake("./data/geoparquet_delta", df) # Fail converting `geometry`

arrow_df = pq.read_table("example.parquet") # Read geometry as a binary blob
write_deltalake("./data/geoparquet_delta", arrow_df) # writes geometry as blob

pandas_df = DeltaTable("./data/geoparquet_delta").to_pandas()

df = geopandas.GeoDataFrame(pandas_df, geometry=ga.to_geopandas(pandas_df.geometry)) # From delta

Ultimately a solution may just be a case of adding a to_geopandas() method: wrapping the last two lines above.
However I don't know how the inner workings of delta would want to interact with this blob. Would be interested in getting some opinions from experts in that sense.

Use Case

Storing geospatial datasets in a delta lake. For those interested in a deeper reasoning surrounding geoparquet: this blog post (now a little old but still relevant) is a good motivational basis.

Related Issue(s)

Was not able to find any specific ones.

[MrPowers]

What's the datatype of the geometry column?

Something like this should work:

import pyarrow.parquet as pq

pq.read_table('example.parquet').schema

[wjones127]

Delta lake isn't just Parquet files, but has it's own spec and schema definition. So the most official way to go about this would be to add geometry columns as a type in the schema spec. I've had previous discussions with the core Delta Lake maintainers and they seem open to more specialized data types; just no one has proposed them yet.

there shouldn't be much work to formally support it.

I have one question on the GeoParquet spec. Initially I thought we could have geo-spatial as a column type. But in the docs it says:

Geometry columns MUST be at the root of the schema. A geometry MUST NOT be a group field or nested in a group.

I think if we were to keep with that, we wouldn't make it a data type. Though at the same time, maybe it's fine if we ignore that part?

[astrojuanlu]

Potentially relevant https://cloudnativegeo.org/blog/2025/02/geoparquet-2.0-going-native/

[Libbum]

datatype of the geometry column

pop_est: double
continent: string
name: string
iso_a3: string
gdp_md_est: int64
geometry: binary
-- schema metadata --
{b'geo': b'{"version": "1.1.0-dev", "primary_column": "geometry", "columns": {"geometry": {"encoding": "WKB", "geometry_types": ["Polygon", "MultiPolygon"], "crs": {"$schema": "https://proj.org/schemas/v0.6/projjson.schema.json", "type": "GeographicCRS", "name": "WGS 84 (CRS84)", "datum_ensemble": {"name": "World Geodetic System 1984 ensemble", "members": [{"name": "World Geodetic System 1984 (Transit)", "id": {"authority": "EPSG", "code": 1166}}, {"name": "World Geodetic System 1984 (G730)", "id": {"authority": "EPSG", "code": 1152}}, {"name": "World Geodetic System 1984 (G873)", "id": {"authority": "EPSG", "code": 1153}}, {"name": "World Geodetic System 1984 (G1150)", "id": {"authority": "EPSG", "code": 1154}}, {"name": "World Geodetic System 1984 (G1674)", "id": {"authority": "EPSG", "code": 1155}}, {"name": "World Geodetic System 1984 (G1762)", "id": {"authority": "EPSG", "code": 1156}}, {"name": "World Geodetic System 1984 (G2139)", "id": {"authority": "EPSG", "code": 1309}}], "ellipsoid": {"name": "WGS 84", "semi_major_axis": 6378137, "inverse_flattening": 298.257223563}, "accuracy": "2.0", "id": {"authority": "EPSG", "code": 6326}}, "coordinate_system": {"subtype": "ellipsoidal", "axis": [{"name": "Geodetic longitude", "abbreviation": "Lon", "direction": "east", "unit": "degree"}, {"name": "Geodetic latitude", "abbreviation": "Lat", "direction": "north", "unit": "degree"}]}, "scope": "Not known.", "area": "World.", "bbox": {"south_latitude": -90, "west_longitude": -180, "north_latitude": 90, "east_longitude": 180}, "id": {"authority": "OGC", "code": "CRS84"}}, "edges": "planar", "bbox": [-180.0, -90.0, 180.0, 83.6451]}}}'}

Which can also be found here. The formal spec is here.

has it's own spec and schema definition

Thanks for the clarification. I'll read up on this as well as attempt an answer to your question - although I'm hoping we could pull in some of the experts on the geoparquet schema to tell us those details with confidence. I'll reach out to them.