Engines
Multi-engine DataFrame abstraction: pandas, DuckDB, Spark+Sedona, PostGIS. Same analysis at different scales.
Engine abstractions — engine-agnostic DataFrame operations.
Holds the DataFrameEngine ABC and its four concrete
implementations (Pandas, DuckDB, Spark, PostGIS).
Submodules load on first attribute access via PEP 562 __getattr__.
- class siege_utilities.engines.Engine[source]
Bases:
EnumSupported DataFrame engine back-ends.
- PANDAS = 'pandas'
- DUCKDB = 'duckdb'
- SPARK = 'spark'
- POSTGIS = 'postgis'
- class siege_utilities.engines.DataFrameEngine[source]
Bases:
ABCAbstract base class for engine-agnostic DataFrame operations.
- abstractmethod read_csv(path, **kwargs)[source]
Read a CSV file and return a DataFrame-like object.
- abstractmethod read_parquet(path, **kwargs)[source]
Read a Parquet file and return a DataFrame-like object.
- abstractmethod query(sql, **kwargs)[source]
Execute sql and return a DataFrame-like object.
Back-end specific keyword arguments (e.g.
tablefor DuckDB,connectionfor PostGIS) are passed through via kwargs.
- abstractmethod groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- abstractmethod filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- abstractmethod join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- abstractmethod read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file (GeoJSON, Shapefile, GeoParquet, GPKG).
Returns an engine-native spatial DataFrame. crs defaults to
EPSG:4326(viageo.crs.get_default_crs).
- abstractmethod spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- abstractmethod buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.
- abstractmethod distance(df, other, geometry_col='geometry', other_geom='geometry')[source]
Compute distances between geometries.
other may be a DataFrame (row-aligned pairwise) or a single geometry (shapely object / WKT string).
- abstractmethod to_geodataframe(df, geometry_col='geometry', *, crs=None)[source]
Convert df to a GeoPandas
GeoDataFrame.Always returns a
geopandas.GeoDataFrameregardless of engine.
- from_geodataframe(gdf, geometry_col='geometry')[source]
Convert a GeoPandas
GeoDataFrameto this engine’s native format.The inverse of
to_geodataframe(). Geometry is serialized as WKB-hex (DuckDB) or WKT (Spark/PostGIS) for engine-native spatial operations.Subclasses should override this for engine-specific conversion. The default implementation returns the GeoDataFrame unchanged (suitable for PandasEngine).
- index_points(df, lat_col, lon_col, *, grid=None, resolution=None, level=None, as_token=True)[source]
Compute a grid (H3 or S2) cell ID for each row.
Default: round-trip through pandas. Engines that benefit from a native path (Spark via pandas UDF, DuckDB via SQL when the spatial extension is loaded) should override.
See
siege_utilities.geo.grids.index_points()for the kwarg inference rules – passresolution=for H3,level=for S2, or setgrid=explicitly.Returns the same engine-native frame type as
dfwould have been augmented with – for the default, that’s a column appended to a pandas-converted copy. Engines that override return their native shape.
- index_polygon(geometry, *, grid=None, resolution=None, level=None, max_cells=None, min_level=None, max_level=None, refine=True)[source]
Cover a polygon with grid cells.
Engine-agnostic by construction – the input is a single geometry, not a frame, so no engine-specific dispatch is required at this level. Provided here for API symmetry with
index_points()and so consumer code can acceptengineas a config knob and callengine.index_polygon(...)uniformly.
- point_in_polygon(points_df, polygons_df, *, points_geom='geometry', polygons_geom='geometry')[source]
Find which polygon each point falls within.
Delegates to
spatial_join()withpredicate='within'.
- dissolve(df, by, geometry_col='geometry', **agg_kwargs)[source]
Dissolve/merge geometries grouped by by columns.
Default converts to GeoDataFrame and uses
gdf.dissolve(). Engines may override for native performance.
- load_polygons(path, *, geometry_col='geometry', format='auto', crs=None)[source]
Load polygon geometries from any supported spatial format.
Handles GeoJSON, Shapefile, GeoParquet, GPKG, WKT CSV. Returns engine-native DataFrame with a geometry column.
- Parameters:
path (str) – Path to the spatial file.
geometry_col (str) – Name for the geometry column in the returned DataFrame. If the source data uses a different column name it will be renamed.
format (str) – Spatial format hint.
"auto"(default) infers from the file extension. Explicit values:"geojson","shapefile","gpkg","geopackage","parquet","geoparquet".crs (str or None) – Target CRS. Defaults to
EPSG:4326viaget_default_crs().
- Return type:
- load_points(df, lat_col='lat', lon_col='lon', geometry_col='geometry')[source]
Create point geometries from latitude/longitude columns.
Adds a
geometrycolumn with WKT POINT strings (or native geometry objects, depending on engine). Engines may override for native point construction.
- load_lines(path, *, geometry_col='geometry', format='auto', crs=None)[source]
Load line/multiline geometries from any supported spatial format.
Same interface as
load_polygons()– works for roads, rivers, transit routes, or any linear feature.- Parameters:
path (str) – Path to the spatial file.
geometry_col (str) – Name for the geometry column in the returned DataFrame. If the source data uses a different column name it will be renamed.
format (str) – Spatial format hint.
"auto"(default) infers from the file extension. Explicit values:"geojson","shapefile","gpkg","geopackage","parquet","geoparquet".crs (str or None) – Target CRS. Defaults to
EPSG:4326viaget_default_crs().
- Return type:
- assign_boundaries(points, polygons, *, point_geom='geometry', polygon_geom='geometry', how='left')[source]
Assign each point to its containing polygon(s).
Core DSTK replacement: given addresses (points) and boundaries (polygons), determine which boundary contains each address.
Default delegates to
spatial_join()with predicate=’contains’ (reversed: polygon contains point). Engines may override for broadcast optimization or native spatial indexing.
- intersect_boundaries(poly_a, poly_b, *, a_geom='geometry', b_geom='geometry')[source]
Find overlapping regions between two polygon boundary sets.
Returns all pairs of polygons that intersect, with both sets of attributes. Useful for county-to-district apportionment or boundary change analysis.
Default delegates to
spatial_join()with predicate=’intersects’. Engines may override to compute intersection geometry and area.
- apportion(source_polys, target_polys, weight_col, *, source_geom='geometry', target_geom='geometry')[source]
Apportion a value from source polygons to target polygons by overlap.
Distributes
weight_colfrom source to target proportional to the fraction of source area that overlaps each target. Example: apportion tract population to congressional districts.Default: intersect, compute area ratios via GeoPandas, aggregate. Engines may override for native area computation (Sedona ST_Area, PostGIS ST_Area, etc.).
- nearest(points, targets, *, k=1, max_distance=None, point_geom='geometry', target_geom='geometry')[source]
Find the k nearest targets for each point.
Default: brute-force via GeoPandas sjoin_nearest (supports k=1 only). Engines should override for native spatial indexing (Sedona KNN, PostGIS ST_DWithin + ORDER BY distance, DuckDB spatial) and k>1.
- multi_assign(points, boundary_layers, *, point_geom='geometry')[source]
Assign points to multiple boundary layers simultaneously.
Calls
assign_boundaries()for each layer and renames every polygon-side column to be{layer_name}_{column}so layers do not collide on shared schemas (TIGER layers sharegeoid/namecolumns – without prefixing, the second join silently overwrites the first layer’s columns).- Parameters:
- Returns:
Points enriched with
{layer_name}_*columns for every polygon-side column from each boundary layer. Bare polygon column names never appear in the output.- Return type:
DataFrame
- class siege_utilities.engines.PandasEngine[source]
Bases:
DataFrameEngineEngine backed by pandas (and optionally GeoPandas).
- read_csv(path, **kwargs)[source]
Read a CSV file.
**kwargs: Forwarded to
pandas.read_csv().
- read_parquet(path, **kwargs)[source]
Read a Parquet file.
**kwargs: Forwarded to
pandas.read_parquet().
- query(sql, **kwargs)[source]
Execute SQL via
pandas.read_sql.Requires a
connectionkeyword argument (a SQLAlchemy engine or DBAPI2 connection).
- groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file.
**kwargs: Forwarded to
geopandas.read_file().
- spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.
- class siege_utilities.engines.DuckDBEngine[source]
Bases:
DataFrameEngineEngine backed by DuckDB.
- read_csv(path, **kwargs)[source]
Read a CSV via DuckDB’s
read_csv_auto.Engine-specific kwargs (DuckDB’s
read_csv_autooptions likeheader,delim,columns) are not yet forwarded; pass the equivalent options inside a custom SQL viaquery()if you need to override auto-detection.
- read_parquet(path, **kwargs)[source]
Read a Parquet file via DuckDB’s
read_parquet.Engine-specific kwargs are not yet forwarded; use
query()for DuckDB-specific options.
- query(sql, **kwargs)[source]
Execute sql against the DuckDB connection.
If a
tablekeyword is supplied together with a pandas DataFrame value, that DataFrame is first registered as a virtual table so it can be referenced in sql.
- groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file via DuckDB’s
ST_Read.**kwargs: Forwarded to
geopandas.read_file()(unused in the current DuckDB path but accepted for interface compatibility).
- spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.
- distance(df, other, geometry_col='geometry', other_geom='geometry')[source]
Compute distances between geometries.
other may be a DataFrame (row-aligned pairwise) or a single geometry (shapely object / WKT string).
- class siege_utilities.engines.SparkEngine[source]
Bases:
DataFrameEngineEngine backed by PySpark.
- Parameters:
spark (optional) – An existing
SparkSession. When None the engine creates or retrieves the active session on first use.
- read_csv(path, **kwargs)[source]
Read a CSV file via Spark.
**kwargs: Forwarded to Spark’s
DataFrameReader.csv().
- read_parquet(path, **kwargs)[source]
Read a Parquet file via Spark.
**kwargs: Forwarded to Spark’s
DataFrameReader.parquet().
- query(sql, **kwargs)[source]
Execute sql via Spark SQL.
SparkEngine.query accepts no engine-specific kwargs;
self._session.sql()takes no options beyond the SQL text. The**kwargsparameter is present for interface symmetry with DuckDBEngine (which acceptstable/df) and PostGISEngine (which acceptsgeom_col). A non-empty kwargs is a caller error and is raised to surface the mistake rather than silently ignored.
- groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file via GeoPandas and convert to a Spark DataFrame.
**kwargs: Forwarded to
geopandas.read_file().
- spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.
- distance(df, other, geometry_col='geometry', other_geom='geometry')[source]
Compute distances between geometries.
other may be a DataFrame (row-aligned pairwise) or a single geometry (shapely object / WKT string).
- to_geodataframe(df, geometry_col='geometry', *, crs=None)[source]
Convert df to a GeoPandas
GeoDataFrame.Always returns a
geopandas.GeoDataFrameregardless of engine.
- load_polygons(path, *, geometry_col='geometry', format='auto', crs=None)[source]
Load polygons – Spark-native parquet/json reader.
- load_points(df, lat_col='lat', lon_col='lon', geometry_col='geometry')[source]
Create point geometries – Spark-native WKT string construction.
- class siege_utilities.engines.PostGISEngine[source]
Bases:
DataFrameEngineEngine backed by PostGIS via SQLAlchemy + GeoPandas.
- Parameters:
connection_string (str) – A SQLAlchemy connection string, e.g.
"postgresql://user:pass@host:5432/dbname".
- index_points(df, lat_col, lon_col, *, grid=None, resolution=None, level=None, as_token=True)[source]
PostGIS does not natively compute H3 / S2 cell IDs.
For S2-keyed warehouse joins, the recommended pattern is:
Compute cell IDs ONCE during ingest (e.g. via the PandasEngine or in your ETL job using
s2_index_points).Store as a
BIGINTcolumn on the row.For region queries, generate
(range_min, range_max)pairs vias2_cells_to_ranges()and filter withWHERE s2_cell BETWEEN range_min AND range_max.
Computing cell IDs at query time over PostGIS would defeat the whole point – the database can’t index a value it has to compute per-row. Hence this engine raises rather than silently degrade.
- read_csv(path, **kwargs)[source]
Read a CSV file.
**kwargs: Forwarded to
pandas.read_csv().
- read_parquet(path, **kwargs)[source]
Read a parquet file. Returns a
GeoDataFramewhen the file carries GeoParquet metadata, a plainDataFrameotherwise.The decision is by file inspection (pyarrow schema metadata), not by exception-fallback. An earlier implementation tried
gpd.read_parquetfirst and silently fell back topd.read_parqueton any exception – that pattern hid real gpd errors (a corrupt geoparquet file, an out-of-date geopandas) as “this must not be a geoparquet file.” The current path checks the parquet metadata directly so the decision is explicit.
- query(sql, **kwargs)[source]
Execute sql against the PostGIS database.
Pass
geom_col="<column>"to indicate a geometry column in the result; the query then returns aGeoDataFramewith that column parsed as geometry. Omitgeom_col(or pass an empty string) for queries that return no geometry; the result is a plainDataFrame.The Geo-vs-plain decision is made by the caller (via the presence of
geom_col), not by exception-fallback. An earlier implementation triedgpd.read_postgisfirst and silently fell back topd.read_sqlon any exception – that pattern hid real PostGIS errors (a malformed SQL, a missing column) as “this must not have a geometry column.”
- groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file or execute a spatial SQL query.
**kwargs: Forwarded to
geopandas.read_file().
- spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.
- siege_utilities.engines.get_engine(name, **kwargs)[source]
Return an engine instance for the requested back-end.
- Parameters:
- Return type:
Submodules
Engine-agnostic DataFrame operations.
Provides a thin abstraction so the same analytical operations can run on DuckDB, Spark, or PostGIS instead of only pandas/GeoPandas.
Why one abstraction
The point is not that the back-ends are interchangeable in performance –
they are not. The point is that consumer code should not branch on
back-end. A reporting module asked to summarise voter rolls should not
care whether the frame came from a 10-million-row Spark job or a 5-row
pandas test fixture; it calls engine.groupby_agg(...) and is done.
This is load-bearing. The anti-pattern to avoid: if isinstance(df,
pd.DataFrame): ... else: ... scattered through reporting/. If you
find yourself reaching for it, the right fix is a new method on the
DataFrameEngine interface – not a special case in the consumer.
Usage:
from siege_utilities.engines.dataframe_engine import get_engine, Engine
engine = get_engine(Engine.PANDAS)
df = engine.read_csv("data.csv")
result = engine.groupby_agg(df, ["state"], {"pop": "sum"})
pdf = engine.to_pandas(result)
Each back-end uses lazy imports so the module itself never requires DuckDB, PySpark, or SQLAlchemy/psycopg2 to be installed.
- class siege_utilities.engines.dataframe_engine.Engine[source]
Bases:
EnumSupported DataFrame engine back-ends.
- PANDAS = 'pandas'
- DUCKDB = 'duckdb'
- SPARK = 'spark'
- POSTGIS = 'postgis'
- class siege_utilities.engines.dataframe_engine.DataFrameEngine[source]
Bases:
ABCAbstract base class for engine-agnostic DataFrame operations.
- abstractmethod read_csv(path, **kwargs)[source]
Read a CSV file and return a DataFrame-like object.
- abstractmethod read_parquet(path, **kwargs)[source]
Read a Parquet file and return a DataFrame-like object.
- abstractmethod query(sql, **kwargs)[source]
Execute sql and return a DataFrame-like object.
Back-end specific keyword arguments (e.g.
tablefor DuckDB,connectionfor PostGIS) are passed through via kwargs.
- abstractmethod groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- abstractmethod filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- abstractmethod join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- abstractmethod read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file (GeoJSON, Shapefile, GeoParquet, GPKG).
Returns an engine-native spatial DataFrame. crs defaults to
EPSG:4326(viageo.crs.get_default_crs).
- abstractmethod spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- abstractmethod buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.
- abstractmethod distance(df, other, geometry_col='geometry', other_geom='geometry')[source]
Compute distances between geometries.
other may be a DataFrame (row-aligned pairwise) or a single geometry (shapely object / WKT string).
- abstractmethod to_geodataframe(df, geometry_col='geometry', *, crs=None)[source]
Convert df to a GeoPandas
GeoDataFrame.Always returns a
geopandas.GeoDataFrameregardless of engine.
- from_geodataframe(gdf, geometry_col='geometry')[source]
Convert a GeoPandas
GeoDataFrameto this engine’s native format.The inverse of
to_geodataframe(). Geometry is serialized as WKB-hex (DuckDB) or WKT (Spark/PostGIS) for engine-native spatial operations.Subclasses should override this for engine-specific conversion. The default implementation returns the GeoDataFrame unchanged (suitable for PandasEngine).
- index_points(df, lat_col, lon_col, *, grid=None, resolution=None, level=None, as_token=True)[source]
Compute a grid (H3 or S2) cell ID for each row.
Default: round-trip through pandas. Engines that benefit from a native path (Spark via pandas UDF, DuckDB via SQL when the spatial extension is loaded) should override.
See
siege_utilities.geo.grids.index_points()for the kwarg inference rules – passresolution=for H3,level=for S2, or setgrid=explicitly.Returns the same engine-native frame type as
dfwould have been augmented with – for the default, that’s a column appended to a pandas-converted copy. Engines that override return their native shape.
- index_polygon(geometry, *, grid=None, resolution=None, level=None, max_cells=None, min_level=None, max_level=None, refine=True)[source]
Cover a polygon with grid cells.
Engine-agnostic by construction – the input is a single geometry, not a frame, so no engine-specific dispatch is required at this level. Provided here for API symmetry with
index_points()and so consumer code can acceptengineas a config knob and callengine.index_polygon(...)uniformly.
- point_in_polygon(points_df, polygons_df, *, points_geom='geometry', polygons_geom='geometry')[source]
Find which polygon each point falls within.
Delegates to
spatial_join()withpredicate='within'.
- dissolve(df, by, geometry_col='geometry', **agg_kwargs)[source]
Dissolve/merge geometries grouped by by columns.
Default converts to GeoDataFrame and uses
gdf.dissolve(). Engines may override for native performance.
- load_polygons(path, *, geometry_col='geometry', format='auto', crs=None)[source]
Load polygon geometries from any supported spatial format.
Handles GeoJSON, Shapefile, GeoParquet, GPKG, WKT CSV. Returns engine-native DataFrame with a geometry column.
- Parameters:
path (str) – Path to the spatial file.
geometry_col (str) – Name for the geometry column in the returned DataFrame. If the source data uses a different column name it will be renamed.
format (str) – Spatial format hint.
"auto"(default) infers from the file extension. Explicit values:"geojson","shapefile","gpkg","geopackage","parquet","geoparquet".crs (str or None) – Target CRS. Defaults to
EPSG:4326viaget_default_crs().
- Return type:
- load_points(df, lat_col='lat', lon_col='lon', geometry_col='geometry')[source]
Create point geometries from latitude/longitude columns.
Adds a
geometrycolumn with WKT POINT strings (or native geometry objects, depending on engine). Engines may override for native point construction.
- load_lines(path, *, geometry_col='geometry', format='auto', crs=None)[source]
Load line/multiline geometries from any supported spatial format.
Same interface as
load_polygons()– works for roads, rivers, transit routes, or any linear feature.- Parameters:
path (str) – Path to the spatial file.
geometry_col (str) – Name for the geometry column in the returned DataFrame. If the source data uses a different column name it will be renamed.
format (str) – Spatial format hint.
"auto"(default) infers from the file extension. Explicit values:"geojson","shapefile","gpkg","geopackage","parquet","geoparquet".crs (str or None) – Target CRS. Defaults to
EPSG:4326viaget_default_crs().
- Return type:
- assign_boundaries(points, polygons, *, point_geom='geometry', polygon_geom='geometry', how='left')[source]
Assign each point to its containing polygon(s).
Core DSTK replacement: given addresses (points) and boundaries (polygons), determine which boundary contains each address.
Default delegates to
spatial_join()with predicate=’contains’ (reversed: polygon contains point). Engines may override for broadcast optimization or native spatial indexing.
- intersect_boundaries(poly_a, poly_b, *, a_geom='geometry', b_geom='geometry')[source]
Find overlapping regions between two polygon boundary sets.
Returns all pairs of polygons that intersect, with both sets of attributes. Useful for county-to-district apportionment or boundary change analysis.
Default delegates to
spatial_join()with predicate=’intersects’. Engines may override to compute intersection geometry and area.
- apportion(source_polys, target_polys, weight_col, *, source_geom='geometry', target_geom='geometry')[source]
Apportion a value from source polygons to target polygons by overlap.
Distributes
weight_colfrom source to target proportional to the fraction of source area that overlaps each target. Example: apportion tract population to congressional districts.Default: intersect, compute area ratios via GeoPandas, aggregate. Engines may override for native area computation (Sedona ST_Area, PostGIS ST_Area, etc.).
- nearest(points, targets, *, k=1, max_distance=None, point_geom='geometry', target_geom='geometry')[source]
Find the k nearest targets for each point.
Default: brute-force via GeoPandas sjoin_nearest (supports k=1 only). Engines should override for native spatial indexing (Sedona KNN, PostGIS ST_DWithin + ORDER BY distance, DuckDB spatial) and k>1.
- multi_assign(points, boundary_layers, *, point_geom='geometry')[source]
Assign points to multiple boundary layers simultaneously.
Calls
assign_boundaries()for each layer and renames every polygon-side column to be{layer_name}_{column}so layers do not collide on shared schemas (TIGER layers sharegeoid/namecolumns – without prefixing, the second join silently overwrites the first layer’s columns).- Parameters:
- Returns:
Points enriched with
{layer_name}_*columns for every polygon-side column from each boundary layer. Bare polygon column names never appear in the output.- Return type:
DataFrame
- class siege_utilities.engines.dataframe_engine.PandasEngine[source]
Bases:
DataFrameEngineEngine backed by pandas (and optionally GeoPandas).
- read_csv(path, **kwargs)[source]
Read a CSV file.
**kwargs: Forwarded to
pandas.read_csv().
- read_parquet(path, **kwargs)[source]
Read a Parquet file.
**kwargs: Forwarded to
pandas.read_parquet().
- query(sql, **kwargs)[source]
Execute SQL via
pandas.read_sql.Requires a
connectionkeyword argument (a SQLAlchemy engine or DBAPI2 connection).
- groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file.
**kwargs: Forwarded to
geopandas.read_file().
- spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.
- class siege_utilities.engines.dataframe_engine.DuckDBEngine[source]
Bases:
DataFrameEngineEngine backed by DuckDB.
- read_csv(path, **kwargs)[source]
Read a CSV via DuckDB’s
read_csv_auto.Engine-specific kwargs (DuckDB’s
read_csv_autooptions likeheader,delim,columns) are not yet forwarded; pass the equivalent options inside a custom SQL viaquery()if you need to override auto-detection.
- read_parquet(path, **kwargs)[source]
Read a Parquet file via DuckDB’s
read_parquet.Engine-specific kwargs are not yet forwarded; use
query()for DuckDB-specific options.
- query(sql, **kwargs)[source]
Execute sql against the DuckDB connection.
If a
tablekeyword is supplied together with a pandas DataFrame value, that DataFrame is first registered as a virtual table so it can be referenced in sql.
- groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file via DuckDB’s
ST_Read.**kwargs: Forwarded to
geopandas.read_file()(unused in the current DuckDB path but accepted for interface compatibility).
- spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.
- distance(df, other, geometry_col='geometry', other_geom='geometry')[source]
Compute distances between geometries.
other may be a DataFrame (row-aligned pairwise) or a single geometry (shapely object / WKT string).
- class siege_utilities.engines.dataframe_engine.SparkEngine[source]
Bases:
DataFrameEngineEngine backed by PySpark.
- Parameters:
spark (optional) – An existing
SparkSession. When None the engine creates or retrieves the active session on first use.
- read_csv(path, **kwargs)[source]
Read a CSV file via Spark.
**kwargs: Forwarded to Spark’s
DataFrameReader.csv().
- read_parquet(path, **kwargs)[source]
Read a Parquet file via Spark.
**kwargs: Forwarded to Spark’s
DataFrameReader.parquet().
- query(sql, **kwargs)[source]
Execute sql via Spark SQL.
SparkEngine.query accepts no engine-specific kwargs;
self._session.sql()takes no options beyond the SQL text. The**kwargsparameter is present for interface symmetry with DuckDBEngine (which acceptstable/df) and PostGISEngine (which acceptsgeom_col). A non-empty kwargs is a caller error and is raised to surface the mistake rather than silently ignored.
- groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file via GeoPandas and convert to a Spark DataFrame.
**kwargs: Forwarded to
geopandas.read_file().
- spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.
- distance(df, other, geometry_col='geometry', other_geom='geometry')[source]
Compute distances between geometries.
other may be a DataFrame (row-aligned pairwise) or a single geometry (shapely object / WKT string).
- to_geodataframe(df, geometry_col='geometry', *, crs=None)[source]
Convert df to a GeoPandas
GeoDataFrame.Always returns a
geopandas.GeoDataFrameregardless of engine.
- load_polygons(path, *, geometry_col='geometry', format='auto', crs=None)[source]
Load polygons – Spark-native parquet/json reader.
- load_points(df, lat_col='lat', lon_col='lon', geometry_col='geometry')[source]
Create point geometries – Spark-native WKT string construction.
- class siege_utilities.engines.dataframe_engine.PostGISEngine[source]
Bases:
DataFrameEngineEngine backed by PostGIS via SQLAlchemy + GeoPandas.
- Parameters:
connection_string (str) – A SQLAlchemy connection string, e.g.
"postgresql://user:pass@host:5432/dbname".
- index_points(df, lat_col, lon_col, *, grid=None, resolution=None, level=None, as_token=True)[source]
PostGIS does not natively compute H3 / S2 cell IDs.
For S2-keyed warehouse joins, the recommended pattern is:
Compute cell IDs ONCE during ingest (e.g. via the PandasEngine or in your ETL job using
s2_index_points).Store as a
BIGINTcolumn on the row.For region queries, generate
(range_min, range_max)pairs vias2_cells_to_ranges()and filter withWHERE s2_cell BETWEEN range_min AND range_max.
Computing cell IDs at query time over PostGIS would defeat the whole point – the database can’t index a value it has to compute per-row. Hence this engine raises rather than silently degrade.
- read_csv(path, **kwargs)[source]
Read a CSV file.
**kwargs: Forwarded to
pandas.read_csv().
- read_parquet(path, **kwargs)[source]
Read a parquet file. Returns a
GeoDataFramewhen the file carries GeoParquet metadata, a plainDataFrameotherwise.The decision is by file inspection (pyarrow schema metadata), not by exception-fallback. An earlier implementation tried
gpd.read_parquetfirst and silently fell back topd.read_parqueton any exception – that pattern hid real gpd errors (a corrupt geoparquet file, an out-of-date geopandas) as “this must not be a geoparquet file.” The current path checks the parquet metadata directly so the decision is explicit.
- query(sql, **kwargs)[source]
Execute sql against the PostGIS database.
Pass
geom_col="<column>"to indicate a geometry column in the result; the query then returns aGeoDataFramewith that column parsed as geometry. Omitgeom_col(or pass an empty string) for queries that return no geometry; the result is a plainDataFrame.The Geo-vs-plain decision is made by the caller (via the presence of
geom_col), not by exception-fallback. An earlier implementation triedgpd.read_postgisfirst and silently fell back topd.read_sqlon any exception – that pattern hid real PostGIS errors (a malformed SQL, a missing column) as “this must not have a geometry column.”
- groupby_agg(df, group_cols, agg_dict)[source]
Group df by group_cols and aggregate according to agg_dict.
agg_dict maps column names to aggregation function names (
"sum","mean","count","min","max").
- filter(df, condition)[source]
Return rows of df that satisfy condition.
The type of condition is engine-specific (boolean Series for pandas, SQL expression string for DuckDB/Spark, etc.).
- join(left, right, on, how='inner')[source]
Join left and right on column(s) on using join type how.
- read_spatial(path, *, crs=None, **kwargs)[source]
Read a spatial file or execute a spatial SQL query.
**kwargs: Forwarded to
geopandas.read_file().
- spatial_join(left, right, how='inner', predicate='intersects', *, left_geom='geometry', right_geom='geometry')[source]
Spatial join using a geometric predicate.
predicate is one of: intersects, contains, within, touches, crosses, overlaps.
- buffer(df, distance, geometry_col='geometry', *, crs=None)[source]
Buffer geometries by distance (in CRS units).
Returns a new DataFrame with the geometry column replaced by buffered geometries.