HDFS Operations

The HDFS operations module provides utilities for working with Hadoop Distributed File System operations, including file management, configuration, and environment setup.

Module Overview

Abstract HDFS Operations - Fully Configurable and Reusable Zero hard-coded project dependencies

class siege_utilities.distributed.hdfs_operations.AbstractHDFSOperations

Bases: object

Abstract HDFS Operations class that can be configured for any project

__init__(config)

Initialize with HDFSConfig

check_hdfs_status()

Check if HDFS is accessible

Return type:

bool

create_spark_session()

Create Spark session using configuration.

Supports local, standalone cluster, and YARN deployments based on the master URL in the config.

sync_directory_to_hdfs(local_path=None, hdfs_subdir='inputs')

Sync local directory/file to HDFS with proper verification.

Raises:

• ValueError – If no data path is provided.

• FileNotFoundError – If the local path does not exist.

• RuntimeError – If HDFS is not accessible or sync verification fails.

• subprocess.CalledProcessError – If an HDFS command fails.

• subprocess.TimeoutExpired – If an HDFS command times out.

Parameters:

• local_path (str | None)

• hdfs_subdir (str)

Return type:

Tuple[str, Dict]

setup_distributed_environment(data_path=None, dependency_paths=None)

Main setup function with proper verification.

Returns:

Tuple of (spark_session, data_url, None).

Raises:

• ValueError – If no data path is provided.

• FileNotFoundError – If the data path does not exist.

• ImportError – If PySpark is not available.

• RuntimeError – If HDFS sync or Spark session creation fails.

Parameters:

• data_path (str | None)

• dependency_paths (List[str] | None)

siege_utilities.distributed.hdfs_operations.create_hdfs_operations(config)

Factory function to create HDFS operations instance

siege_utilities.distributed.hdfs_operations.setup_distributed_environment(config, data_path=None, dependency_paths=None)

Convenience function to set up distributed environment

Parameters:

• data_path (str | None)

• dependency_paths (List[str] | None)

Functions

siege_utilities.distributed.hdfs_operations.create_hdfs_operations(config)

Factory function to create HDFS operations instance

siege_utilities.distributed.hdfs_operations.setup_distributed_environment(config, data_path=None, dependency_paths=None)

Convenience function to set up distributed environment

Parameters:

• data_path (str | None)

• dependency_paths (List[str] | None)

Usage Examples

Basic HDFS operations setup:

File operations:

File transfer operations:

Batch operations:

Unit Tests

The HDFS operations module has comprehensive test coverage:

Test Results: All HDFS operations tests pass successfully with comprehensive coverage.