Databricks
Databricks-specific connectors: bridge pattern (Spark to GeoPandas), secrets, artifacts, lakehouse federation.
Databricks and LakeBase utilities for siege_utilities.
Why this package is a sibling to `distributed/`, not a child.
distributed/ holds vendor-agnostic PySpark + HDFS utilities that work on any Spark cluster (EMR, Dataproc, self-hosted). databricks/ holds Databricks-only SDK features (Unity Catalog, LakeBase, workspace auth, secrets, DBFS, job-run URL construction for Asset Bundles) that only make sense inside a Databricks workspace.
The split is load-bearing because Azure Databricks lacks Apache Sedona and
the C-library stack that GeoPandas / Shapely / Fiona need. dataframe_bridge.py
is the workaround — Spark ↔ pandas and Spark ↔ GeoPandas bridges that let
geospatial work happen in environments where the usual stack is not installable.
Don’t merge these modules. A future reader “DRY”-ing them together will re-create a problem we already solved.
See also: docs/INTENT.md (D4), docs/adr/ (architecture decisions).
- siege_utilities.databricks.build_databricks_run_url(host, workspace_id, run_id)[source]
Build a direct Databricks run URL.
- siege_utilities.databricks.build_foreign_table_sql(catalog, schema, table, connection_name, source_schema, source_table=None)[source]
Build SQL for creating a Databricks Lakehouse Federation foreign table from a LakeBase (or other UC-Federation-registered) source.
Databricks Lakehouse Federation only. The generated
CREATE FOREIGN TABLE ... USING CONNECTIONsyntax is not parseable by OSS unitycatalog.io. (SU#519)All identifier fields (catalog, schema, table, connection_name, source_schema, source_table) are validated against the Postgres identifier allow-list before interpolation. An earlier version interpolated source_schema and source_table directly into the SQL string without validation, producing an injection risk if either contained a single quote.
Note: syntax can vary by Databricks workspace / feature flag. Keep this as a composable helper and validate generated SQL in the target Databricks environment.
- siege_utilities.databricks.build_jdbc_url(host, dbname, port=None)[source]
Build a PostgreSQL JDBC URL.
Validates
hostanddbnameagainst allow-lists to prevent URL-injection into the JDBC connection string.
- siege_utilities.databricks.build_lakebase_psql_command(host, user, dbname, port=None, sslmode=None)[source]
Build a psql command line for LakeBase/Postgres.
All caller-provided values are validated against strict allow-lists (hostname characters for host, identifier characters for user/dbname, Postgres sslmode enum for sslmode). The returned command is then safe to execute via subprocess.run(…, shell=True) or shell=False.
Raises ValueError if any field contains characters that could break out of the conninfo or the surrounding shell quoting.
- siege_utilities.databricks.build_pgpass_entry(host, port, dbname, user, password)[source]
Build a single .pgpass line.
Each field is backslash-escaped per the Postgres .pgpass format (
\->\\;:->\:). A password containing:or\previously produced a malformed entry that either failed to match or matched the wrong host/db.
- siege_utilities.databricks.build_schema_and_table_sync_sql(catalog, schema, connection_name, source_schema, tables)[source]
Build CREATE SCHEMA + CREATE FOREIGN TABLE statements for multiple tables.
Databricks Lakehouse Federation only — see module docstring.
- siege_utilities.databricks.ensure_secret_scope(scope, workspace_client)[source]
Ensure a secret scope exists using Databricks workspace APIs.
Returns the scope name on either the existed-already or created-now path. Logs at INFO level naming which path was taken (writing-code:11 floor (b)): an auditor can confirm from log output alone whether a new scope was created or an existing one was reused.
- siege_utilities.databricks.geopandas_to_spark(dataframe, spark=None, geometry_column='geometry', geometry_format='wkt', crs_column='geometry_crs')[source]
Convert a GeoPandas DataFrame to Spark using a serializable geometry column.
Geometry is serialized as WKT or WKB hex to avoid runtime-specific geospatial library requirements on Spark clusters.
- siege_utilities.databricks.get_active_spark_session(create_if_missing=False, app_name='siege_utilities_databricks')[source]
Get active Spark session from Databricks or PySpark runtime.
- siege_utilities.databricks.get_dbutils(spark=None)[source]
Get a Databricks DBUtils handle.
Dispatches between the pyspark-DBUtils path (Databricks Connect / runtime) and the notebook-namespace path (classic Databricks notebooks) by inspecting the runtime environment rather than by try/except chain. Each context is content-distinguishable per writing-code:14: pyspark.dbutils availability is a module-import check; notebook context is an IPython global-namespace check.
- siege_utilities.databricks.get_runtime_secret(scope, key, dbutils=None)[source]
Read a secret from Databricks runtime dbutils.
- siege_utilities.databricks.get_workspace_client(profile=None, host=None, token=None, azure_client_id=None, azure_client_secret=None, azure_tenant_id=None, **kwargs)[source]
Build a Databricks WorkspaceClient for PAT, profile, or Azure SP auth.
- Parameters:
profile (str | None) – Databricks CLI profile name.
host (str | None) – Databricks workspace host URL.
token (str | None) – Databricks PAT.
azure_client_id (str | None) – Azure service principal client ID.
azure_client_secret (str | None) – Azure service principal client secret.
azure_tenant_id (str | None) – Azure tenant ID.
**kwargs (Any) – Additional WorkspaceClient arguments.
- Return type:
- siege_utilities.databricks.pandas_to_spark(dataframe, spark=None)[source]
Convert a Pandas DataFrame to a Spark DataFrame.
- Parameters:
dataframe (pandas.DataFrame)
spark (Any | None)
- Return type:
- siege_utilities.databricks.parse_conninfo(conninfo)[source]
Parse a PostgreSQL-style conninfo string into key/value pairs.
Example
host=example.com user=alice dbname=mydb port=5432 sslmode=require
- siege_utilities.databricks.put_secret(scope, key, value, workspace_client)[source]
Create or update a secret key using Databricks workspace APIs.
Returns the scope-qualified key (
<scope>/<key>) on success. Logs at INFO level naming the scope and key (without the value). The combination satisfies writing-code:11 floor (a) inspectable return + (b) audit log: callers can both inspect the return value AND grep logs to verify the write happened.
- siege_utilities.databricks.runtime_secret_exists(scope, key, dbutils=None)[source]
Check whether a runtime secret key exists in a scope.
Returns True if the key exists in the scope, False if it does not. Lookup errors (scope missing, access denied) propagate as the underlying exception type so the caller can distinguish ‘key not present’ from ‘lookup failed’ – writing-code:7 territory.
- siege_utilities.databricks.spark_to_geopandas(dataframe, geometry_column='geometry', geometry_format='wkt', crs=None)[source]
Convert Spark DataFrame with serialized geometry into GeoPandas DataFrame.
Supports WKT and WKB hex serialization formats.
- siege_utilities.databricks.spark_to_pandas(dataframe, limit=None)[source]
Convert a Spark DataFrame to a Pandas DataFrame, with optional limit.
- Parameters:
- Return type:
Submodules
Bridges for Pandas/GeoPandas and Spark DataFrames.
- siege_utilities.databricks.dataframe_bridge.geopandas_to_spark(dataframe, spark=None, geometry_column='geometry', geometry_format='wkt', crs_column='geometry_crs')[source]
Convert a GeoPandas DataFrame to Spark using a serializable geometry column.
Geometry is serialized as WKT or WKB hex to avoid runtime-specific geospatial library requirements on Spark clusters.
- siege_utilities.databricks.dataframe_bridge.pandas_to_spark(dataframe, spark=None)[source]
Convert a Pandas DataFrame to a Spark DataFrame.
- Parameters:
dataframe (pandas.DataFrame)
spark (Any | None)
- Return type:
- siege_utilities.databricks.dataframe_bridge.spark_to_geopandas(dataframe, geometry_column='geometry', geometry_format='wkt', crs=None)[source]
Convert Spark DataFrame with serialized geometry into GeoPandas DataFrame.
Supports WKT and WKB hex serialization formats.
- siege_utilities.databricks.dataframe_bridge.spark_to_pandas(dataframe, limit=None)[source]
Convert a Spark DataFrame to a Pandas DataFrame, with optional limit.
- Parameters:
- Return type:
Databricks authentication helpers.
- siege_utilities.databricks.auth.get_workspace_client(profile=None, host=None, token=None, azure_client_id=None, azure_client_secret=None, azure_tenant_id=None, **kwargs)[source]
Build a Databricks WorkspaceClient for PAT, profile, or Azure SP auth.
- Parameters:
profile (str | None) – Databricks CLI profile name.
host (str | None) – Databricks workspace host URL.
token (str | None) – Databricks PAT.
azure_client_id (str | None) – Azure service principal client ID.
azure_client_secret (str | None) – Azure service principal client secret.
azure_tenant_id (str | None) – Azure tenant ID.
**kwargs (Any) – Additional WorkspaceClient arguments.
- Return type:
Databricks secret management helpers.
- siege_utilities.databricks.secrets.ensure_secret_scope(scope, workspace_client)[source]
Ensure a secret scope exists using Databricks workspace APIs.
Returns the scope name on either the existed-already or created-now path. Logs at INFO level naming which path was taken (writing-code:11 floor (b)): an auditor can confirm from log output alone whether a new scope was created or an existing one was reused.
- siege_utilities.databricks.secrets.get_runtime_secret(scope, key, dbutils=None)[source]
Read a secret from Databricks runtime dbutils.
- siege_utilities.databricks.secrets.put_secret(scope, key, value, workspace_client)[source]
Create or update a secret key using Databricks workspace APIs.
Returns the scope-qualified key (
<scope>/<key>) on success. Logs at INFO level naming the scope and key (without the value). The combination satisfies writing-code:11 floor (a) inspectable return + (b) audit log: callers can both inspect the return value AND grep logs to verify the write happened.
- siege_utilities.databricks.secrets.runtime_secret_exists(scope, key, dbutils=None)[source]
Check whether a runtime secret key exists in a scope.
Returns True if the key exists in the scope, False if it does not. Lookup errors (scope missing, access denied) propagate as the underlying exception type so the caller can distinguish ‘key not present’ from ‘lookup failed’ – writing-code:7 territory.
Databricks runtime session helpers.
- siege_utilities.databricks.session.get_active_spark_session(create_if_missing=False, app_name='siege_utilities_databricks')[source]
Get active Spark session from Databricks or PySpark runtime.
- siege_utilities.databricks.session.get_dbutils(spark=None)[source]
Get a Databricks DBUtils handle.
Dispatches between the pyspark-DBUtils path (Databricks Connect / runtime) and the notebook-namespace path (classic Databricks notebooks) by inspecting the runtime environment rather than by try/except chain. Each context is content-distinguishable per writing-code:14: pyspark.dbutils availability is a module-import check; notebook context is an IPython global-namespace check.
Databricks artifact helpers.
- siege_utilities.databricks.artifacts.build_databricks_run_url(host, workspace_id, run_id)[source]
Build a direct Databricks run URL.
SQL helpers for Databricks Lakehouse Federation foreign tables.
This module generates SQL of the form
CREATE FOREIGN TABLE ... USING CONNECTION ..., which is a
Databricks-managed Unity Catalog only feature (Lakehouse Federation).
It does NOT work against the open-source Unity Catalog implementation
(unitycatalog.io / unitycatalog/unitycatalog). OSS UC has no
CREATE CONNECTION / CREATE FOREIGN TABLE mechanism — federation
is handled at the compute layer (Trino, Spark) instead.
Previously named siege_utilities.databricks.unity_catalog. Renamed
in SU#519 because the old name implied the helpers worked against any
Unity Catalog and burned consumers planning OSS UC bridges. The old
module path remains importable as a deprecation shim.
See unity_catalog.py (sibling) for the back-compat re-export and
SU#519 for the rename rationale.
- siege_utilities.databricks.lakehouse_federation.build_foreign_table_sql(catalog, schema, table, connection_name, source_schema, source_table=None)[source]
Build SQL for creating a Databricks Lakehouse Federation foreign table from a LakeBase (or other UC-Federation-registered) source.
Databricks Lakehouse Federation only. The generated
CREATE FOREIGN TABLE ... USING CONNECTIONsyntax is not parseable by OSS unitycatalog.io. (SU#519)All identifier fields (catalog, schema, table, connection_name, source_schema, source_table) are validated against the Postgres identifier allow-list before interpolation. An earlier version interpolated source_schema and source_table directly into the SQL string without validation, producing an injection risk if either contained a single quote.
Note: syntax can vary by Databricks workspace / feature flag. Keep this as a composable helper and validate generated SQL in the target Databricks environment.
- siege_utilities.databricks.lakehouse_federation.build_schema_and_table_sync_sql(catalog, schema, connection_name, source_schema, tables)[source]
Build CREATE SCHEMA + CREATE FOREIGN TABLE statements for multiple tables.
Databricks Lakehouse Federation only — see module docstring.
- siege_utilities.databricks.lakehouse_federation.quote_ident(value)[source]
Quote an identifier with backticks for Databricks SQL.
LakeBase/Postgres connection helpers for Databricks workflows.
- siege_utilities.databricks.lakebase.build_jdbc_url(host, dbname, port=None)[source]
Build a PostgreSQL JDBC URL.
Validates
hostanddbnameagainst allow-lists to prevent URL-injection into the JDBC connection string.
- siege_utilities.databricks.lakebase.build_lakebase_psql_command(host, user, dbname, port=None, sslmode=None)[source]
Build a psql command line for LakeBase/Postgres.
All caller-provided values are validated against strict allow-lists (hostname characters for host, identifier characters for user/dbname, Postgres sslmode enum for sslmode). The returned command is then safe to execute via subprocess.run(…, shell=True) or shell=False.
Raises ValueError if any field contains characters that could break out of the conninfo or the surrounding shell quoting.
- siege_utilities.databricks.lakebase.build_pgpass_entry(host, port, dbname, user, password)[source]
Build a single .pgpass line.
Each field is backslash-escaped per the Postgres .pgpass format (
\->\\;:->\:). A password containing:or\previously produced a malformed entry that either failed to match or matched the wrong host/db.
- siege_utilities.databricks.lakebase.parse_conninfo(conninfo)[source]
Parse a PostgreSQL-style conninfo string into key/value pairs.
Example
host=example.com user=alice dbname=mydb port=5432 sslmode=require
Deprecated alias for siege_utilities.databricks.lakehouse_federation.
The helpers in this module generate CREATE FOREIGN TABLE ... USING
CONNECTION SQL, which is a Databricks Lakehouse Federation feature.
It does NOT work against the open-source Unity Catalog implementation
(unitycatalog.io). The old module name implied otherwise and burned
consumers planning OSS UC bridges. Renamed in SU#519.
Import from siege_utilities.databricks.lakehouse_federation
instead. This shim re-exports the public surface unchanged; importing
this module emits a single DeprecationWarning.
- siege_utilities.databricks.unity_catalog.build_foreign_table_sql(catalog, schema, table, connection_name, source_schema, source_table=None)[source]
Build SQL for creating a Databricks Lakehouse Federation foreign table from a LakeBase (or other UC-Federation-registered) source.
Databricks Lakehouse Federation only. The generated
CREATE FOREIGN TABLE ... USING CONNECTIONsyntax is not parseable by OSS unitycatalog.io. (SU#519)All identifier fields (catalog, schema, table, connection_name, source_schema, source_table) are validated against the Postgres identifier allow-list before interpolation. An earlier version interpolated source_schema and source_table directly into the SQL string without validation, producing an injection risk if either contained a single quote.
Note: syntax can vary by Databricks workspace / feature flag. Keep this as a composable helper and validate generated SQL in the target Databricks environment.