HDFS Operations

The HDFS operations module provides utilities for working with Hadoop Distributed File System operations, including file management, configuration, and environment setup.

Module Overview

Abstract HDFS Operations - Fully Configurable and Reusable Zero hard-coded project dependencies

class siege_utilities.distributed.hdfs_operations.AbstractHDFSOperations

Bases: object

Configurable HDFS operations for distributed pipelines.

Wraps the small set of HDFS interactions the library needs — status check, Spark session construction, distributed dependency staging — behind a single class parameterized by project-specific settings (cache locations, Hadoop timeout, hash functions).

Parameters:

config (HDFSConfig) – Project-specific configuration object. Must expose a cache_directory path, get_cache_path method, hdfs_timeout, and optional hash_func / quick_signature_func overrides. See create_hdfs_operations() and setup_distributed_environment() for common construction patterns.

Notes

Instantiation eagerly creates config.cache_directory if absent. All HDFS interactions log through config.log_info / config.log_error rather than raising, matching the pre-check convention used elsewhere in the distributed module.

__init__(config)

Initialize with HDFSConfig

check_hdfs_status()

Check if HDFS is accessible

Return type:

bool

create_spark_session()

Create Spark session using configuration.

Supports local, standalone cluster, and YARN deployments based on the master URL in the config.

sync_directory_to_hdfs(local_path=None, hdfs_subdir='inputs')

Sync local directory/file to HDFS with proper verification.

Raises:

• ValueError – If no data path is provided.

• FileNotFoundError – If the local path does not exist.

• RuntimeError – If HDFS is not accessible or sync verification fails.

• subprocess.CalledProcessError – If an HDFS command fails.

• subprocess.TimeoutExpired – If an HDFS command times out.

Parameters:

• local_path (str | None)

• hdfs_subdir (str)

Return type:

Tuple[str, Dict]

setup_distributed_environment(data_path=None, dependency_paths=None)

Main setup function with proper verification.

Returns:

Tuple of (spark_session, data_url, None).

Raises:

• ValueError – If no data path is provided.

• FileNotFoundError – If the data path does not exist.

• ImportError – If PySpark is not available.

• RuntimeError – If HDFS sync or Spark session creation fails.

Parameters:

• data_path (str | None)

• dependency_paths (List[str] | None)

siege_utilities.distributed.hdfs_operations.create_hdfs_operations(config)

Factory for AbstractHDFSOperations instances.

Wraps the constructor so callers can request an HDFS operations object without importing the class name directly — useful for keeping consumer code decoupled from the concrete implementation.

Parameters:

config (HDFSConfig) – Project-specific HDFS config (see AbstractHDFSOperations).

Returns:

A ready-to-use HDFS operations instance.

Return type:

AbstractHDFSOperations

siege_utilities.distributed.hdfs_operations.setup_distributed_environment(config, data_path=None, dependency_paths=None)

One-shot helper to construct HDFS ops + stage data + build a Spark session.

Equivalent to AbstractHDFSOperations(config).setup_distributed_environment(...) — provided as a top-level convenience so callers who only need a single distributed-setup call don’t have to instantiate the class explicitly.

Parameters:

• config (HDFSConfig) – Project-specific HDFS config (see AbstractHDFSOperations).

• data_path (str, optional) – Local path to a data payload that should be staged onto HDFS before the Spark session starts. When None, no staging happens.

• dependency_paths (list of str, optional) – Additional local paths (e.g. Python source directories) to zip and stage as PySpark dependencies. When None, only the library’s own default dependency bundle is staged.

Returns:

(spark_session, staged_data_url, dependency_zip_url) — matching the tuple returned by AbstractHDFSOperations.setup_distributed_environment(). Both URLs are None when their corresponding argument was omitted.

Return type:

tuple

Functions

siege_utilities.distributed.hdfs_operations.create_hdfs_operations(config)

Factory for AbstractHDFSOperations instances.

Wraps the constructor so callers can request an HDFS operations object without importing the class name directly — useful for keeping consumer code decoupled from the concrete implementation.

Parameters:

config (HDFSConfig) – Project-specific HDFS config (see AbstractHDFSOperations).

Returns:

A ready-to-use HDFS operations instance.

Return type:

AbstractHDFSOperations

siege_utilities.distributed.hdfs_operations.setup_distributed_environment(config, data_path=None, dependency_paths=None)

One-shot helper to construct HDFS ops + stage data + build a Spark session.

Equivalent to AbstractHDFSOperations(config).setup_distributed_environment(...) — provided as a top-level convenience so callers who only need a single distributed-setup call don’t have to instantiate the class explicitly.

Parameters:

• config (HDFSConfig) – Project-specific HDFS config (see AbstractHDFSOperations).

• data_path (str, optional) – Local path to a data payload that should be staged onto HDFS before the Spark session starts. When None, no staging happens.

• dependency_paths (list of str, optional) – Additional local paths (e.g. Python source directories) to zip and stage as PySpark dependencies. When None, only the library’s own default dependency bundle is staged.

Returns:

(spark_session, staged_data_url, dependency_zip_url) — matching the tuple returned by AbstractHDFSOperations.setup_distributed_environment(). Both URLs are None when their corresponding argument was omitted.

Return type:

tuple

Usage Examples

Basic HDFS operations setup:

File operations:

File transfer operations:

Batch operations:

Unit Tests

The HDFS operations module has comprehensive test coverage:

Test Results: All HDFS operations tests pass successfully with comprehensive coverage.