Files

File and filesystem operations: hashing, path management, remote downloads, shell operations.

File utilities for siege_utilities.

This module provides comprehensive file operations including: - Path manipulation and directory management - File operations (copy, move, delete, etc.) - Remote file downloads with retry logic - File hashing and integrity verification

All functions are available at the top level for mutual availability across modules.

siege_utilities.files.ensure_path_exists(desired_path)[source]

Ensure a directory path exists, creating it if necessary.

SECURITY: Validates paths to prevent path traversal and sensitive file access.

Parameters:

desired_path (str | Path) – Path to ensure exists

Returns:

Path object for the created directory

Raises:

PathSecurityError – If path fails security validation

Return type:

Path

Example

>>> path = ensure_path_exists("logs/2024/01")
>>> print(f"Directory created: {path}")
>>> ensure_path_exists("../../../etc")
siege_utilities.files.unzip_file_to_directory(zip_file_path, extract_to=None, create_subdirectory=True)[source]

Extract a zip file to a directory.

SECURITY: Validates paths to prevent zip slip attacks and path traversal.

Parameters:
  • zip_file_path (str | Path) – Path to the zip file

  • extract_to (str | Path | None) – Directory to extract to (defaults to zip file’s parent)

  • create_subdirectory (bool) – Whether to create a subdirectory for extracted files

Returns:

Path to the extraction directory

Raises:
Return type:

Path

Example

>>> extract_dir = unzip_file_to_directory("data.zip")
>>> print(f"Files extracted to: {extract_dir}")
siege_utilities.files.get_file_extension(file_path)[source]

Get the file extension from a file path.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File extension (including the dot)

Raises:

PathSecurityError – If path fails security validation

Return type:

str

Example

>>> ext = get_file_extension("document.pdf")
>>> print(f"File extension: {ext}")
>>> get_file_extension("../../../etc/passwd")
siege_utilities.files.get_file_name_without_extension(file_path)[source]

Get the file name without its extension.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File name without extension

Raises:

PathSecurityError – If path fails security validation

Return type:

str

Example

>>> name = get_file_name_without_extension("document.pdf")
>>> print(f"File name: {name}")
>>> get_file_name_without_extension("../../../etc/shadow")
siege_utilities.files.is_hidden_file(file_path)[source]

Check if a file or directory is hidden.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to check

Returns:

True if the file/directory is hidden

Raises:

PathSecurityError – If path fails security validation

Return type:

bool

Example

>>> if is_hidden_file(".config"):
...     print("Hidden file detected")
>>> is_hidden_file("../../.ssh/id_rsa")
siege_utilities.files.get_relative_path(base_path, target_path)[source]

Get the relative path from base_path to target_path.

SECURITY: Validates both base and target paths to prevent path traversal attacks.

Parameters:
  • base_path (str | Path) – Base directory path

  • target_path (str | Path) – Target file/directory path

Returns:

Relative path from base to target

Raises:
Return type:

Path

Example

>>> rel_path = get_relative_path("/home/user", "/home/user/documents/file.txt")
>>> print(f"Relative path: {rel_path}")
siege_utilities.files.find_files_by_pattern(directory, pattern='*', recursive=False)[source]

Find files matching a pattern in a directory.

SECURITY: Validates directory path to prevent path traversal attacks.

Parameters:
  • directory (str | Path) – Directory to search in

  • pattern (str) – Glob pattern to match

  • recursive (bool) – Whether to search recursively

Returns:

List of matching file paths

Raises:
Return type:

list[Path]

Example

>>> files = find_files_by_pattern("data", "*.csv", recursive=True)
>>> print(f"Found {len(files)} CSV files")
>>> find_files_by_pattern("../../../etc", "*")
siege_utilities.files.create_backup_path(original_path, backup_suffix='.backup', backup_dir=None)[source]

Create a backup path for a file.

SECURITY: Validates both original and backup paths to prevent path traversal attacks.

Parameters:
  • original_path (str | Path) – Path to the original file

  • backup_suffix (str) – Suffix to add to the backup filename

  • backup_dir (str | Path | None) – Directory to place backup in (defaults to original file’s directory)

Returns:

Path for the backup file

Raises:

PathSecurityError – If paths fail security validation

Return type:

Path

Example

>>> backup_path = create_backup_path("config.yaml", ".bak")
>>> print(f"Backup will be saved to: {backup_path}")
>>> create_backup_path("/etc/passwd", ".bak")
siege_utilities.files.normalize_path(path)[source]

Normalize a file path, resolving any relative components.

This function expands ~ (home directory), resolves .. (parent directory), and converts to an absolute path. It does NOT validate for security - use validate_safe_path() if you need security validation.

Parameters:

path (str | Path) – Path to normalize

Returns:

Normalized absolute path

Return type:

Path

Example

>>> norm_path = normalize_path("~/../data/file.txt")
>>> print(f"Normalized path: {norm_path}")
>>> norm_path = normalize_path("./../data/file.txt")
siege_utilities.files.remove_tree(path)[source]

Remove a file or directory tree safely.

SECURITY: Validates paths to prevent removing sensitive system files.

Parameters:

path (str | Path) – Path to file or directory to remove

Raises:
Return type:

None

Example

>>> remove_tree("temp_directory")
>>> remove_tree(Path("old_files"))
siege_utilities.files.file_exists(path)[source]

Check if a file exists at the specified path.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

path (str | Path) – Path to check

Returns:

True if file exists, False otherwise

Raises:

PathSecurityError – If path fails security validation

Return type:

bool

Example

>>> if file_exists("config.yaml"):
...     print("Config file found")
>>> file_exists("../../../etc/passwd")
Security Changes:
  • Now validates paths to block path traversal

  • Blocks access to sensitive system files

siege_utilities.files.touch_file(path, create_parents=True)[source]

Create an empty file, creating parent directories if needed.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:
  • path (str | Path) – Path to the file to create

  • create_parents (bool) – Whether to create parent directories

Raises:
Return type:

None

Example

>>> touch_file("logs/app.log")
>>> touch_file(Path("data/output.txt"))
siege_utilities.files.count_lines(file_path, encoding='utf-8')[source]

Count the total number of lines in a text file.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:
  • file_path (str | Path) – Path to the text file

  • encoding (str) – File encoding to use

Returns:

Number of lines

Raises:
Return type:

int

Example

>>> line_count = count_lines("data.csv")
>>> print(f"File has {line_count} lines")
siege_utilities.files.copy_file(source, destination, overwrite=False)[source]

Copy a file from source to destination.

SECURITY: Validates both source and destination paths.

Parameters:
  • source (str | Path) – Source file path

  • destination (str | Path) – Destination file path

  • overwrite (bool) – Whether to overwrite existing files

Raises:
Return type:

None

Example

>>> copy_file("source.txt", "backup/source.txt")
>>> copy_file(Path("config.yaml"), Path("config.yaml.bak"))
siege_utilities.files.move_file(source, destination, overwrite=False)[source]

Move a file from source to destination.

SECURITY: Validates both source and destination paths.

Parameters:
  • source (str | Path) – Source file path

  • destination (str | Path) – Destination file path

  • overwrite (bool) – Whether to overwrite existing files

Raises:
Return type:

None

Example

>>> move_file("temp.txt", "archive/temp.txt")
>>> move_file(Path("old.log"), Path("logs/old.log"))
siege_utilities.files.get_file_size(file_path)[source]

Get the size of a file in bytes.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File size in bytes

Raises:
Return type:

int

Example

>>> size = get_file_size("large_file.zip")
>>> print(f"File size: {size} bytes")
siege_utilities.files.list_directory(path, pattern='*', include_dirs=True, include_files=True)[source]

List contents of a directory with optional filtering.

SECURITY: Validates directory path to prevent traversal attacks.

Parameters:
  • path (str | Path) – Directory path to list

  • pattern (str) – Glob pattern for filtering files

  • include_dirs (bool) – Whether to include directories

  • include_files (bool) – Whether to include files

Returns:

List of Path objects

Raises:
Return type:

List[Path]

Example

>>> files = list_directory("data", "*.csv")
>>> dirs = list_directory("logs", include_files=False)
siege_utilities.files.run_command(command, cwd=None, timeout=30, capture_output=True, allow_list=None, unsafe=False)[source]

Run a shell command with security validation and proper error handling.

SECURITY: This function now validates commands by default. Only whitelisted commands are allowed unless unsafe=True is explicitly set.

Parameters:
  • command (str | List[str]) – Command to run (string or list)

  • cwd (str | Path | None) – Working directory for the command

  • timeout (int | None) – Timeout in seconds (default: 30)

  • capture_output (bool) – Whether to capture command output

  • allow_list (set | None) – Optional custom whitelist of allowed commands

  • unsafe (bool) – If True, bypass the allow-list check. The command is still executed via argv (shell=False) — string commands are shlex-split and run without shell expansion. Use this when you legitimately need a binary that isn’t in the allow-list, not when you need shell features.

Returns:

CompletedProcess object on success or non-zero exit.

Raises:
Return type:

CompletedProcess

Example

>>> result = run_command("ls -la")
>>> if result and result.returncode == 0:
...     print("Command succeeded")
>>> result = run_command("rm -rf /")
>>> result = run_command("git status", allow_list={'git', 'ls'})
>>> result = run_command("custom_cmd", unsafe=True)
Security Changes:
  • Previously used shell=True with no validation (CRITICAL VULNERABILITY)

  • Now validates all commands against whitelist by default

  • Blocks command injection, path traversal, sensitive file access

  • Use unsafe=True ONLY with trusted input in trusted environments

siege_utilities.files.delete_existing_file_and_replace_it_with_an_empty_file(file_path, create_parents=True)[source]

Backward compatibility function that deletes existing file and replaces it with empty file.

Deprecated since version 3.19.0: Use touch_file() for creating empty files. Note that touch_file does not delete the existing file first; if you need delete-then-create semantics, call Path.unlink() before touch_file(). Will be removed in v4.0.0.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:
  • file_path (str | Path) – Path to the file

  • create_parents (bool) – Whether to create parent directories

Raises:
Return type:

None

siege_utilities.files.ensure_directory_exists(directory)[source]

Ensure a directory exists, creating it if necessary.

Parameters:

directory (str | Path) – Path to directory

Returns:

Path object for the directory

Raises:

PathSecurityError – If path fails security validation.

Return type:

Path

siege_utilities.files.safe_file_write(file_path, content, encoding='utf-8', create_dirs=True)[source]

Safely write content to a file, optionally creating parent directories.

Parameters:
  • file_path (str | Path) – Path to file

  • content (str) – Content to write

  • encoding (str) – File encoding (default: utf-8)

  • create_dirs (bool) – Create parent directories if they don’t exist

Returns:

True if successful, False otherwise

Raises:

PathSecurityError – If path fails security validation.

Return type:

bool

siege_utilities.files.safe_file_read(file_path, encoding='utf-8', default=None)[source]

Safely read content from a file.

Parameters:
  • file_path (str | Path) – Path to file

  • encoding (str) – File encoding (default: utf-8)

  • default (str | None) – Default value to return if file doesn’t exist or error occurs

Returns:

File content or default value

Raises:

PathSecurityError – If path fails security validation.

Return type:

str | None

siege_utilities.files.safe_json_write(file_path, data, indent=2, create_dirs=True)[source]

Safely write data to a JSON file.

Parameters:
  • file_path (str | Path) – Path to JSON file

  • data (Any) – Data to write (must be JSON serializable)

  • indent (int) – JSON indentation (default: 2)

  • create_dirs (bool) – Create parent directories if they don’t exist

Returns:

True if successful, False otherwise

Raises:

PathSecurityError – If path fails security validation.

Return type:

bool

siege_utilities.files.safe_json_read(file_path, default=None)[source]

Safely read data from a JSON file.

Parameters:
  • file_path (str | Path) – Path to JSON file

  • default (Any | None) – Default value to return if file doesn’t exist or error occurs

Returns:

Parsed JSON data or default value

Raises:

PathSecurityError – If path fails security validation.

Return type:

Any | None

siege_utilities.files.get_file_size_mb(file_path)[source]

Get file size in megabytes.

Parameters:

file_path (str | Path) – Path to file

Returns:

File size in MB.

Raises:
Return type:

float

siege_utilities.files.list_files_recursive(directory, pattern='*', exclude_dirs=True)[source]

List all files in a directory recursively.

Parameters:
  • directory (str | Path) – Directory to search

  • pattern (str) – Glob pattern (default: “*” for all files)

  • exclude_dirs (bool) – Exclude directories from results

Returns:

List of Path objects

Raises:
Return type:

list[Path]

siege_utilities.files.download_file(url, local_filename, chunk_size=8192, timeout=30, verify_ssl=True)[source]

Download a file from a URL to a local file with progress bar.

SECURITY: Validates local file path to prevent path traversal attacks.

Parameters:
  • url (str) – The URL to download from

  • local_filename (str | Path) – The local path where the file should be saved

  • chunk_size (int) – Size of chunks to download at once

  • timeout (int) – Request timeout in seconds

  • verify_ssl (bool) – Whether to verify SSL certificates

Returns:

The local filename as a string

Raises:
  • PathSecurityError – If local path fails security validation

  • ConnectionError – If HTTP request fails (non-2xx status)

  • requests.exceptions.Timeout – If request times out

  • requests.exceptions.RequestException – If request fails

  • OSError – If file write fails

Return type:

str

Example

>>> result = download_file("https://example.com/file.zip", "downloads/file.zip")
>>> print(f"Downloaded to {result}")
siege_utilities.files.generate_local_path_from_url(url, directory_path, as_string=True)[source]

Generate a local file path from a URL.

SECURITY: Validates directory path to prevent path traversal attacks.

Parameters:
  • url (str) – URL to extract filename from

  • directory_path (str | Path) – Directory where the file should be saved

  • as_string (bool) – Whether to return the result as a string

Returns:

Path object or string

Raises:
Return type:

Path | str

Example

>>> path = generate_local_path_from_url("https://example.com/file.zip", "downloads")
>>> print(f"Local path: {path}")
siege_utilities.files.download_file_with_retry(url, local_filename, max_retries=3, retry_delay=5, **kwargs)[source]

Download a file with automatic retry on failure.

Parameters:
  • url (str) – The URL to download from

  • local_filename (str | Path) – The local path where the file should be saved

  • max_retries (int) – Maximum number of retry attempts

  • retry_delay (int) – Delay between retries in seconds

  • **kwargs – Additional arguments passed to download_file

Returns:

The local filename as a string

Raises:
  • ConnectionError – If all retry attempts fail

  • requests.exceptions.RequestException – If all retry attempts fail

Return type:

str

Example

>>> result = download_file_with_retry("https://example.com/file.zip", "file.zip")
>>> print(f"Downloaded to {result}")
siege_utilities.files.get_file_info(url, timeout=10)[source]

Get information about a remote file without downloading it.

Parameters:
  • url (str) – URL to get information about

  • timeout (int) – Request timeout in seconds

Returns:

Dictionary with file information

Raises:
  • ConnectionError – If HTTP request fails (non-2xx status)

  • requests.exceptions.RequestException – If request fails

  • ImportError – If requests library is not available

Return type:

dict

Example

>>> info = get_file_info("https://example.com/file.zip")
>>> print(f"File size: {info['size']} bytes")
siege_utilities.files.is_downloadable(url, timeout=10)[source]

Check if a URL points to a downloadable file.

Parameters:
  • url (str) – URL to check

  • timeout (int) – Request timeout in seconds

Returns:

True if the URL points to a downloadable file, False otherwise

Return type:

bool

Example

>>> if is_downloadable("https://example.com/file.zip"):
...     print("URL is downloadable")
siege_utilities.files.get_download_directory()[source]

Get the default download directory for siege_utilities.

Deprecated since version 3.19.0: Use siege_utilities.config.user_config.get_download_directory() instead, or from siege_utilities import get_download_directory. Will be removed in v4.0.0.

Returns:

Path to the download directory

Return type:

Path

class siege_utilities.files.SpatialFormat[source]

Bases: str, Enum

Supported spatial output formats.

GEOPARQUET = 'geoparquet'
PARQUET = 'parquet'
GPKG = 'gpkg'
GEOJSON = 'geojson'
TOPOJSON = 'topojson'
CSV = 'csv'
SHAPEFILE = 'shp'
__new__(value)
class siege_utilities.files.TabularFormat[source]

Bases: str, Enum

Supported tabular output formats.

PARQUET = 'parquet'
CSV = 'csv'
EXCEL = 'excel'
JSON = 'json'
__new__(value)
siege_utilities.files.save_spatial(gdf, path, fmt=SpatialFormat.GEOPARQUET, **kwargs)[source]

Write a GeoDataFrame to path in the requested format.

Parameters:
Return type:

Path to the written file.

siege_utilities.files.save_tabular(df, path, fmt=TabularFormat.PARQUET, **kwargs)[source]

Write a DataFrame to path in the requested format.

Parameters:
  • df (pandas.DataFrame) – Data to write.

  • path (str or Path) – Destination file path.

  • fmt (TabularFormat) – Output format (default PARQUET).

  • **kwargs – Passed through to the underlying writer.

Return type:

Path to the written file.

siege_utilities.files.generate_sha256_hash_for_file(file_path)[source]

Generate SHA256 hash for a file - chunked reading for large files.

Deprecated since version 3.19.0: Use calculate_file_hash() or get_file_hash() instead. Will be removed in v4.0.0.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path – Path to the file (str or Path object)

Returns:

SHA256 hash as hexadecimal string

Raises:
Return type:

str

Example

>>> hash_val = generate_sha256_hash_for_file("data.txt")
siege_utilities.files.get_file_hash(file_path, algorithm='sha256')[source]

Generate hash for a file using specified algorithm.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:
  • file_path – Path to the file (str or Path object)

  • algorithm – Hash algorithm to use (‘sha256’, ‘md5’, ‘sha1’, etc.)

Returns:

Hash as hexadecimal string

Raises:
Return type:

str

Example

>>> hash_val = get_file_hash("data.txt", "sha256")
siege_utilities.files.calculate_file_hash(file_path)[source]

Alias for get_file_hash with SHA256 - for backward compatibility.

Return type:

str

siege_utilities.files.get_quick_file_signature(file_path)[source]

Generate a quick file signature using file stats + partial hash Faster for change detection, not cryptographically secure

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path – Path to the file

Returns:

Quick signature string (‘missing’, fallback stat string, or hash)

Raises:
  • PathSecurityError – If path fails security validation

  • Exception – If both primary and fallback signature generation fail (logged before re-raise)

Return type:

str

Example

>>> sig = get_quick_file_signature("data.txt")
>>> get_quick_file_signature("/etc/passwd")
Security Changes:
  • Now validates paths to block path traversal

  • Blocks access to sensitive system files

siege_utilities.files.verify_file_integrity(file_path, expected_hash, algorithm='sha256')[source]

Verify file integrity by comparing with expected hash.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files (through get_file_hash).

Parameters:
  • file_path – Path to the file

  • expected_hash – Expected hash value

  • algorithm – Hash algorithm used

Returns:

True if file matches expected hash, False if hash does not match

Raises:
Return type:

bool

Example

>>> expected = "abc123..."
>>> if verify_file_integrity("data.txt", expected):
...     print("File integrity verified")

Submodules

Modern file operations for siege_utilities. Provides clean, type-safe file manipulation utilities.

siege_utilities.files.operations.sanitize_filename(name)[source]

Convert an arbitrary string into a safe filename component.

Unsafe characters (path separators, control characters, punctuation other than . and -) are replaced with underscores, runs of underscores are collapsed, and leading/trailing dots and underscores are trimmed. The result is always a non-empty string and the operation is idempotent (sanitize_filename(sanitize_filename(x)) == sanitize_filename(x)).

Parameters:

name (str) – Arbitrary input string.

Returns:

A filesystem-safe filename component.

Return type:

str

Raises:

TypeError – If name is not a string.

siege_utilities.files.operations.atomic_write_path(target)[source]

Context manager that yields a temporary path for writing, then atomically replaces the target on successful exit.

Usage:

with atomic_write_path("/data/output.csv") as tmp:
    df.to_csv(tmp, index=False)
# target is now atomically updated

If the block raises, the temp file is cleaned up and the target is left unchanged.

Parameters:

target (str | Path)

Return type:

Generator[Path, None, None]

siege_utilities.files.operations.atomic_write_shapefile(gdf, target, **kwargs)[source]

Write a GeoDataFrame as a Shapefile atomically.

Shapefiles produce sidecar files (.shx, .dbf, .prj, .cpg). This writes to a temp stem then renames all produced files to the final stem.

Parameters:

target (str | Path)

Return type:

None

siege_utilities.files.operations.remove_tree(path)[source]

Remove a file or directory tree safely.

SECURITY: Validates paths to prevent removing sensitive system files.

Parameters:

path (str | Path) – Path to file or directory to remove

Raises:
Return type:

None

Example

>>> remove_tree("temp_directory")
>>> remove_tree(Path("old_files"))
siege_utilities.files.operations.file_exists(path)[source]

Check if a file exists at the specified path.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

path (str | Path) – Path to check

Returns:

True if file exists, False otherwise

Raises:

PathSecurityError – If path fails security validation

Return type:

bool

Example

>>> if file_exists("config.yaml"):
...     print("Config file found")
>>> file_exists("../../../etc/passwd")
Security Changes:
  • Now validates paths to block path traversal

  • Blocks access to sensitive system files

siege_utilities.files.operations.touch_file(path, create_parents=True)[source]

Create an empty file, creating parent directories if needed.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:
  • path (str | Path) – Path to the file to create

  • create_parents (bool) – Whether to create parent directories

Raises:
Return type:

None

Example

>>> touch_file("logs/app.log")
>>> touch_file(Path("data/output.txt"))
siege_utilities.files.operations.count_lines(file_path, encoding='utf-8')[source]

Count the total number of lines in a text file.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:
  • file_path (str | Path) – Path to the text file

  • encoding (str) – File encoding to use

Returns:

Number of lines

Raises:
Return type:

int

Example

>>> line_count = count_lines("data.csv")
>>> print(f"File has {line_count} lines")
siege_utilities.files.operations.copy_file(source, destination, overwrite=False)[source]

Copy a file from source to destination.

SECURITY: Validates both source and destination paths.

Parameters:
  • source (str | Path) – Source file path

  • destination (str | Path) – Destination file path

  • overwrite (bool) – Whether to overwrite existing files

Raises:
Return type:

None

Example

>>> copy_file("source.txt", "backup/source.txt")
>>> copy_file(Path("config.yaml"), Path("config.yaml.bak"))
siege_utilities.files.operations.move_file(source, destination, overwrite=False)[source]

Move a file from source to destination.

SECURITY: Validates both source and destination paths.

Parameters:
  • source (str | Path) – Source file path

  • destination (str | Path) – Destination file path

  • overwrite (bool) – Whether to overwrite existing files

Raises:
Return type:

None

Example

>>> move_file("temp.txt", "archive/temp.txt")
>>> move_file(Path("old.log"), Path("logs/old.log"))
siege_utilities.files.operations.get_file_size(file_path)[source]

Get the size of a file in bytes.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File size in bytes

Raises:
Return type:

int

Example

>>> size = get_file_size("large_file.zip")
>>> print(f"File size: {size} bytes")
siege_utilities.files.operations.list_directory(path, pattern='*', include_dirs=True, include_files=True)[source]

List contents of a directory with optional filtering.

SECURITY: Validates directory path to prevent traversal attacks.

Parameters:
  • path (str | Path) – Directory path to list

  • pattern (str) – Glob pattern for filtering files

  • include_dirs (bool) – Whether to include directories

  • include_files (bool) – Whether to include files

Returns:

List of Path objects

Raises:
Return type:

List[Path]

Example

>>> files = list_directory("data", "*.csv")
>>> dirs = list_directory("logs", include_files=False)
siege_utilities.files.operations.run_command(command, cwd=None, timeout=30, capture_output=True, allow_list=None, unsafe=False)[source]

Run a shell command with security validation and proper error handling.

SECURITY: This function now validates commands by default. Only whitelisted commands are allowed unless unsafe=True is explicitly set.

Parameters:
  • command (str | List[str]) – Command to run (string or list)

  • cwd (str | Path | None) – Working directory for the command

  • timeout (int | None) – Timeout in seconds (default: 30)

  • capture_output (bool) – Whether to capture command output

  • allow_list (set | None) – Optional custom whitelist of allowed commands

  • unsafe (bool) – If True, bypass the allow-list check. The command is still executed via argv (shell=False) — string commands are shlex-split and run without shell expansion. Use this when you legitimately need a binary that isn’t in the allow-list, not when you need shell features.

Returns:

CompletedProcess object on success or non-zero exit.

Raises:
Return type:

CompletedProcess

Example

>>> result = run_command("ls -la")
>>> if result and result.returncode == 0:
...     print("Command succeeded")
>>> result = run_command("rm -rf /")
>>> result = run_command("git status", allow_list={'git', 'ls'})
>>> result = run_command("custom_cmd", unsafe=True)
Security Changes:
  • Previously used shell=True with no validation (CRITICAL VULNERABILITY)

  • Now validates all commands against whitelist by default

  • Blocks command injection, path traversal, sensitive file access

  • Use unsafe=True ONLY with trusted input in trusted environments

siege_utilities.files.operations.ensure_directory_exists(directory)[source]

Ensure a directory exists, creating it if necessary.

Parameters:

directory (str | Path) – Path to directory

Returns:

Path object for the directory

Raises:

PathSecurityError – If path fails security validation.

Return type:

Path

siege_utilities.files.operations.safe_file_write(file_path, content, encoding='utf-8', create_dirs=True)[source]

Safely write content to a file, optionally creating parent directories.

Parameters:
  • file_path (str | Path) – Path to file

  • content (str) – Content to write

  • encoding (str) – File encoding (default: utf-8)

  • create_dirs (bool) – Create parent directories if they don’t exist

Returns:

True if successful, False otherwise

Raises:

PathSecurityError – If path fails security validation.

Return type:

bool

siege_utilities.files.operations.safe_file_read(file_path, encoding='utf-8', default=None)[source]

Safely read content from a file.

Parameters:
  • file_path (str | Path) – Path to file

  • encoding (str) – File encoding (default: utf-8)

  • default (str | None) – Default value to return if file doesn’t exist or error occurs

Returns:

File content or default value

Raises:

PathSecurityError – If path fails security validation.

Return type:

str | None

siege_utilities.files.operations.safe_json_write(file_path, data, indent=2, create_dirs=True)[source]

Safely write data to a JSON file.

Parameters:
  • file_path (str | Path) – Path to JSON file

  • data (Any) – Data to write (must be JSON serializable)

  • indent (int) – JSON indentation (default: 2)

  • create_dirs (bool) – Create parent directories if they don’t exist

Returns:

True if successful, False otherwise

Raises:

PathSecurityError – If path fails security validation.

Return type:

bool

siege_utilities.files.operations.safe_json_read(file_path, default=None)[source]

Safely read data from a JSON file.

Parameters:
  • file_path (str | Path) – Path to JSON file

  • default (Any | None) – Default value to return if file doesn’t exist or error occurs

Returns:

Parsed JSON data or default value

Raises:

PathSecurityError – If path fails security validation.

Return type:

Any | None

siege_utilities.files.operations.get_file_size_mb(file_path)[source]

Get file size in megabytes.

Parameters:

file_path (str | Path) – Path to file

Returns:

File size in MB.

Raises:
Return type:

float

siege_utilities.files.operations.list_files_recursive(directory, pattern='*', exclude_dirs=True)[source]

List all files in a directory recursively.

Parameters:
  • directory (str | Path) – Directory to search

  • pattern (str) – Glob pattern (default: “*” for all files)

  • exclude_dirs (bool) – Exclude directories from results

Returns:

List of Path objects

Raises:
Return type:

list[Path]

siege_utilities.files.operations.rmtree(path)

Remove a file or directory tree safely.

SECURITY: Validates paths to prevent removing sensitive system files.

Parameters:

path (str | Path) – Path to file or directory to remove

Raises:
Return type:

None

Example

>>> remove_tree("temp_directory")
>>> remove_tree(Path("old_files"))
siege_utilities.files.operations.check_if_file_exists_at_path(path)[source]

Check if a file exists at the specified path.

Deprecated since version 3.19.0: Use file_exists() instead. Will be removed in v4.0.0.

Parameters:

path (str | Path)

Return type:

bool

siege_utilities.files.operations.delete_existing_file_and_replace_it_with_an_empty_file(file_path, create_parents=True)[source]

Backward compatibility function that deletes existing file and replaces it with empty file.

Deprecated since version 3.19.0: Use touch_file() for creating empty files. Note that touch_file does not delete the existing file first; if you need delete-then-create semantics, call Path.unlink() before touch_file(). Will be removed in v4.0.0.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:
  • file_path (str | Path) – Path to the file

  • create_parents (bool) – Whether to create parent directories

Raises:
Return type:

None

siege_utilities.files.operations.count_total_rows_in_file_pythonically(file_path, encoding='utf-8')

Count the total number of lines in a text file.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:
  • file_path (str | Path) – Path to the text file

  • encoding (str) – File encoding to use

Returns:

Number of lines

Raises:
Return type:

int

Example

>>> line_count = count_lines("data.csv")
>>> print(f"File has {line_count} lines")

Hash Management Functions - Fixed Version Provides standardized hash functions that actually exist and work properly

siege_utilities.files.hashing.calculate_file_hash(file_path)[source]

Alias for get_file_hash with SHA256 - for backward compatibility.

Return type:

str

siege_utilities.files.hashing.generate_sha256_hash_for_file(file_path)[source]

Generate SHA256 hash for a file - chunked reading for large files.

Deprecated since version 3.19.0: Use calculate_file_hash() or get_file_hash() instead. Will be removed in v4.0.0.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path – Path to the file (str or Path object)

Returns:

SHA256 hash as hexadecimal string

Raises:
Return type:

str

Example

>>> hash_val = generate_sha256_hash_for_file("data.txt")
siege_utilities.files.hashing.get_file_hash(file_path, algorithm='sha256')[source]

Generate hash for a file using specified algorithm.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:
  • file_path – Path to the file (str or Path object)

  • algorithm – Hash algorithm to use (‘sha256’, ‘md5’, ‘sha1’, etc.)

Returns:

Hash as hexadecimal string

Raises:
Return type:

str

Example

>>> hash_val = get_file_hash("data.txt", "sha256")
siege_utilities.files.hashing.get_quick_file_signature(file_path)[source]

Generate a quick file signature using file stats + partial hash Faster for change detection, not cryptographically secure

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path – Path to the file

Returns:

Quick signature string (‘missing’, fallback stat string, or hash)

Raises:
  • PathSecurityError – If path fails security validation

  • Exception – If both primary and fallback signature generation fail (logged before re-raise)

Return type:

str

Example

>>> sig = get_quick_file_signature("data.txt")
>>> get_quick_file_signature("/etc/passwd")
Security Changes:
  • Now validates paths to block path traversal

  • Blocks access to sensitive system files

siege_utilities.files.hashing.verify_file_integrity(file_path, expected_hash, algorithm='sha256')[source]

Verify file integrity by comparing with expected hash.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files (through get_file_hash).

Parameters:
  • file_path – Path to the file

  • expected_hash – Expected hash value

  • algorithm – Hash algorithm used

Returns:

True if file matches expected hash, False if hash does not match

Raises:
Return type:

bool

Example

>>> expected = "abc123..."
>>> if verify_file_integrity("data.txt", expected):
...     print("File integrity verified")

Modern path utilities for siege_utilities. Provides clean, type-safe path manipulation utilities.

siege_utilities.files.paths.ensure_path_exists(desired_path)[source]

Ensure a directory path exists, creating it if necessary.

SECURITY: Validates paths to prevent path traversal and sensitive file access.

Parameters:

desired_path (str | Path) – Path to ensure exists

Returns:

Path object for the created directory

Raises:

PathSecurityError – If path fails security validation

Return type:

Path

Example

>>> path = ensure_path_exists("logs/2024/01")
>>> print(f"Directory created: {path}")
>>> ensure_path_exists("../../../etc")
siege_utilities.files.paths.unzip_file_to_directory(zip_file_path, extract_to=None, create_subdirectory=True)[source]

Extract a zip file to a directory.

SECURITY: Validates paths to prevent zip slip attacks and path traversal.

Parameters:
  • zip_file_path (str | Path) – Path to the zip file

  • extract_to (str | Path | None) – Directory to extract to (defaults to zip file’s parent)

  • create_subdirectory (bool) – Whether to create a subdirectory for extracted files

Returns:

Path to the extraction directory

Raises:
Return type:

Path

Example

>>> extract_dir = unzip_file_to_directory("data.zip")
>>> print(f"Files extracted to: {extract_dir}")
siege_utilities.files.paths.get_file_extension(file_path)[source]

Get the file extension from a file path.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File extension (including the dot)

Raises:

PathSecurityError – If path fails security validation

Return type:

str

Example

>>> ext = get_file_extension("document.pdf")
>>> print(f"File extension: {ext}")
>>> get_file_extension("../../../etc/passwd")
siege_utilities.files.paths.get_file_name_without_extension(file_path)[source]

Get the file name without its extension.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File name without extension

Raises:

PathSecurityError – If path fails security validation

Return type:

str

Example

>>> name = get_file_name_without_extension("document.pdf")
>>> print(f"File name: {name}")
>>> get_file_name_without_extension("../../../etc/shadow")
siege_utilities.files.paths.is_hidden_file(file_path)[source]

Check if a file or directory is hidden.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to check

Returns:

True if the file/directory is hidden

Raises:

PathSecurityError – If path fails security validation

Return type:

bool

Example

>>> if is_hidden_file(".config"):
...     print("Hidden file detected")
>>> is_hidden_file("../../.ssh/id_rsa")
siege_utilities.files.paths.get_relative_path(base_path, target_path)[source]

Get the relative path from base_path to target_path.

SECURITY: Validates both base and target paths to prevent path traversal attacks.

Parameters:
  • base_path (str | Path) – Base directory path

  • target_path (str | Path) – Target file/directory path

Returns:

Relative path from base to target

Raises:
Return type:

Path

Example

>>> rel_path = get_relative_path("/home/user", "/home/user/documents/file.txt")
>>> print(f"Relative path: {rel_path}")
siege_utilities.files.paths.find_files_by_pattern(directory, pattern='*', recursive=False)[source]

Find files matching a pattern in a directory.

SECURITY: Validates directory path to prevent path traversal attacks.

Parameters:
  • directory (str | Path) – Directory to search in

  • pattern (str) – Glob pattern to match

  • recursive (bool) – Whether to search recursively

Returns:

List of matching file paths

Raises:
Return type:

list[Path]

Example

>>> files = find_files_by_pattern("data", "*.csv", recursive=True)
>>> print(f"Found {len(files)} CSV files")
>>> find_files_by_pattern("../../../etc", "*")
siege_utilities.files.paths.create_backup_path(original_path, backup_suffix='.backup', backup_dir=None)[source]

Create a backup path for a file.

SECURITY: Validates both original and backup paths to prevent path traversal attacks.

Parameters:
  • original_path (str | Path) – Path to the original file

  • backup_suffix (str) – Suffix to add to the backup filename

  • backup_dir (str | Path | None) – Directory to place backup in (defaults to original file’s directory)

Returns:

Path for the backup file

Raises:

PathSecurityError – If paths fail security validation

Return type:

Path

Example

>>> backup_path = create_backup_path("config.yaml", ".bak")
>>> print(f"Backup will be saved to: {backup_path}")
>>> create_backup_path("/etc/passwd", ".bak")
siege_utilities.files.paths.normalize_path(path)[source]

Normalize a file path, resolving any relative components.

This function expands ~ (home directory), resolves .. (parent directory), and converts to an absolute path. It does NOT validate for security - use validate_safe_path() if you need security validation.

Parameters:

path (str | Path) – Path to normalize

Returns:

Normalized absolute path

Return type:

Path

Example

>>> norm_path = normalize_path("~/../data/file.txt")
>>> print(f"Normalized path: {norm_path}")
>>> norm_path = normalize_path("./../data/file.txt")
siege_utilities.files.paths.unzip_file_to_its_own_directory(zip_file_path, extract_to=None, create_subdirectory=True)

Extract a zip file to a directory.

SECURITY: Validates paths to prevent zip slip attacks and path traversal.

Parameters:
  • zip_file_path (str | Path) – Path to the zip file

  • extract_to (str | Path | None) – Directory to extract to (defaults to zip file’s parent)

  • create_subdirectory (bool) – Whether to create a subdirectory for extracted files

Returns:

Path to the extraction directory

Raises:
Return type:

Path

Example

>>> extract_dir = unzip_file_to_directory("data.zip")
>>> print(f"Files extracted to: {extract_dir}")

Modern remote file operations for siege_utilities. Provides clean, type-safe file download utilities.

siege_utilities.files.remote.download_file(url, local_filename, chunk_size=8192, timeout=30, verify_ssl=True)[source]

Download a file from a URL to a local file with progress bar.

SECURITY: Validates local file path to prevent path traversal attacks.

Parameters:
  • url (str) – The URL to download from

  • local_filename (str | Path) – The local path where the file should be saved

  • chunk_size (int) – Size of chunks to download at once

  • timeout (int) – Request timeout in seconds

  • verify_ssl (bool) – Whether to verify SSL certificates

Returns:

The local filename as a string

Raises:
  • PathSecurityError – If local path fails security validation

  • ConnectionError – If HTTP request fails (non-2xx status)

  • requests.exceptions.Timeout – If request times out

  • requests.exceptions.RequestException – If request fails

  • OSError – If file write fails

Return type:

str

Example

>>> result = download_file("https://example.com/file.zip", "downloads/file.zip")
>>> print(f"Downloaded to {result}")
siege_utilities.files.remote.generate_local_path_from_url(url, directory_path, as_string=True)[source]

Generate a local file path from a URL.

SECURITY: Validates directory path to prevent path traversal attacks.

Parameters:
  • url (str) – URL to extract filename from

  • directory_path (str | Path) – Directory where the file should be saved

  • as_string (bool) – Whether to return the result as a string

Returns:

Path object or string

Raises:
Return type:

Path | str

Example

>>> path = generate_local_path_from_url("https://example.com/file.zip", "downloads")
>>> print(f"Local path: {path}")
siege_utilities.files.remote.download_file_with_retry(url, local_filename, max_retries=3, retry_delay=5, **kwargs)[source]

Download a file with automatic retry on failure.

Parameters:
  • url (str) – The URL to download from

  • local_filename (str | Path) – The local path where the file should be saved

  • max_retries (int) – Maximum number of retry attempts

  • retry_delay (int) – Delay between retries in seconds

  • **kwargs – Additional arguments passed to download_file

Returns:

The local filename as a string

Raises:
  • ConnectionError – If all retry attempts fail

  • requests.exceptions.RequestException – If all retry attempts fail

Return type:

str

Example

>>> result = download_file_with_retry("https://example.com/file.zip", "file.zip")
>>> print(f"Downloaded to {result}")
siege_utilities.files.remote.get_file_info(url, timeout=10)[source]

Get information about a remote file without downloading it.

Parameters:
  • url (str) – URL to get information about

  • timeout (int) – Request timeout in seconds

Returns:

Dictionary with file information

Raises:
  • ConnectionError – If HTTP request fails (non-2xx status)

  • requests.exceptions.RequestException – If request fails

  • ImportError – If requests library is not available

Return type:

dict

Example

>>> info = get_file_info("https://example.com/file.zip")
>>> print(f"File size: {info['size']} bytes")
siege_utilities.files.remote.is_downloadable(url, timeout=10)[source]

Check if a URL points to a downloadable file.

Parameters:
  • url (str) – URL to check

  • timeout (int) – Request timeout in seconds

Returns:

True if the URL points to a downloadable file, False otherwise

Return type:

bool

Example

>>> if is_downloadable("https://example.com/file.zip"):
...     print("URL is downloadable")

Secure shell command execution utilities.

SECURITY NOTE: This module previously contained critical vulnerabilities. All shell execution now includes proper validation and sanitization.

siege_utilities.files.shell.run_subprocess(command_list, allow_list=None, timeout=30)[source]

Run a shell command as a subprocess with security validation.

SECURITY: This function now validates all commands against a whitelist and blocks dangerous operations. Only read-only commands are allowed by default.

Parameters:
  • command_list (str | List[str]) – The command to run, as a list or string

  • allow_list (set | None) – Optional custom whitelist of allowed commands

  • timeout (int) – Command timeout in seconds (default: 30)

Returns:

The command output (stdout if successful, stderr if failed)

Raises:
Return type:

str

Example

>>> output = run_subprocess("ls -la")
>>> output = run_subprocess("rm -rf /")
Security Changes:
  • Previously used shell=True with no validation (CRITICAL VULNERABILITY)

  • Now validates all commands against whitelist

  • Blocks command injection, path traversal, sensitive file access

  • Uses shell=False by default for safety

siege_utilities.files.shell.validate_command_safety(command, allow_list=None)[source]

Validate that a command is safe to execute.

Parameters:
  • command (str | List[str]) – Command string or list to validate

  • allow_list (set | None) – Optional custom whitelist of allowed commands

Returns:

Validated command as list of strings

Raises:
Return type:

List[str]

Security checks: - Validates command against whitelist - Blocks command injection characters (;, &, |, $, `, newlines) - Blocks path traversal attempts - Blocks access to sensitive system files

exception siege_utilities.files.shell.SecurityError[source]

Bases: Exception

Raised when a security violation is detected.

Unified output format support for spatial and tabular data.

Provides save_spatial() and save_tabular() dispatchers that write data to disk in the requested format. Enum types SpatialFormat and TabularFormat enumerate supported file types.

Usage:

from siege_utilities.files.formats import save_spatial, SpatialFormat

save_spatial(gdf, "/tmp/districts.parquet", SpatialFormat.GEOPARQUET)
class siege_utilities.files.formats.SpatialFormat[source]

Bases: str, Enum

Supported spatial output formats.

GEOPARQUET = 'geoparquet'
PARQUET = 'parquet'
GPKG = 'gpkg'
GEOJSON = 'geojson'
TOPOJSON = 'topojson'
CSV = 'csv'
SHAPEFILE = 'shp'
__new__(value)
class siege_utilities.files.formats.TabularFormat[source]

Bases: str, Enum

Supported tabular output formats.

PARQUET = 'parquet'
CSV = 'csv'
EXCEL = 'excel'
JSON = 'json'
__new__(value)
siege_utilities.files.formats.save_spatial(gdf, path, fmt=SpatialFormat.GEOPARQUET, **kwargs)[source]

Write a GeoDataFrame to path in the requested format.

Parameters:
Return type:

Path to the written file.

siege_utilities.files.formats.save_tabular(df, path, fmt=TabularFormat.PARQUET, **kwargs)[source]

Write a DataFrame to path in the requested format.

Parameters:
  • df (pandas.DataFrame) – Data to write.

  • path (str or Path) – Destination file path.

  • fmt (TabularFormat) – Output format (default PARQUET).

  • **kwargs – Passed through to the underlying writer.

Return type:

Path to the written file.

Path validation and sanitization utilities for siege_utilities.

This module provides security-focused path validation to prevent: - Path traversal attacks (../, ~/../, etc.) - Access to sensitive system files and directories - Symlink-based attacks - Null byte injection

Use these validators in all file operations to ensure secure path handling.

exception siege_utilities.files.validation.PathSecurityError[source]

Bases: Exception

Raised when a path fails security validation.

siege_utilities.files.validation.is_path_traversal_attempt(path)[source]

Check if a path contains traversal patterns.

Parameters:

path (str | Path) – Path to check

Returns:

True if path contains traversal patterns

Return type:

bool

Example

>>> is_path_traversal_attempt("../../../etc/passwd")
True
>>> is_path_traversal_attempt("data/file.txt")
False
siege_utilities.files.validation.is_sensitive_path(path)[source]

Check if a path points to or contains a sensitive system location.

Parameters:

path (str | Path) – Path to check

Returns:

True if path is sensitive

Return type:

bool

Example

>>> is_sensitive_path("/etc/passwd")
True
>>> is_sensitive_path("/tmp/data.txt")
False
siege_utilities.files.validation.validate_safe_path(path, allow_absolute=True, base_directory=None)[source]

Validate that a path is safe to access.

Performs comprehensive security checks: 1. Blocks path traversal attempts 2. Blocks access to sensitive system paths 3. Optionally restricts to a base directory 4. Blocks null byte injection 5. Resolves symlinks and checks final destination

Parameters:
  • path (str | Path) – Path to validate

  • allow_absolute (bool) – Whether to allow absolute paths

  • base_directory (str | Path | None) – If provided, path must be within this directory

Returns:

Validated Path object

Raises:
Return type:

Path

Example

>>> # Safe path
>>> validated = validate_safe_path("data/file.txt")
>>>
>>> try:
...     validate_safe_path("../../../etc/passwd")
... except PathSecurityError as e:
...     pass  # raises PathSecurityError
>>>
>>> validated = validate_safe_path("subdir/file.txt", base_directory="/safe/dir")
siege_utilities.files.validation.validate_file_path(file_path, must_exist=False, base_directory=None)[source]

Validate a file path for read/write operations.

Parameters:
  • file_path (str | Path) – Path to validate

  • must_exist (bool) – If True, path must exist

  • base_directory (str | Path | None) – If provided, path must be within this directory

Returns:

Validated Path object

Raises:
Return type:

Path

Example

>>> path = validate_file_path("data.csv", must_exist=True)
>>>
>>> path = validate_file_path("output.txt", must_exist=False)
siege_utilities.files.validation.validate_directory_path(dir_path, must_exist=False, base_directory=None)[source]

Validate a directory path.

Parameters:
  • dir_path (str | Path) – Path to validate

  • must_exist (bool) – If True, directory must exist

  • base_directory (str | Path | None) – If provided, path must be within this directory

Returns:

Validated Path object

Raises:
Return type:

Path

Example

>>> # Validate directory
>>> path = validate_directory_path("logs", must_exist=True)
siege_utilities.files.validation.safe_join_paths(*paths, base_directory=None)[source]

Safely join multiple path components with security validation.

Parameters:
  • *paths (str | Path) – Path components to join

  • base_directory (str | Path | None) – If provided, result must be within this directory

Returns:

Validated joined Path object

Raises:
Return type:

Path

Example

>>> # Safe join
>>> path = safe_join_paths("data", "2024", "file.txt")
>>>
>>> path = safe_join_paths("subdir", "file.txt", base_directory="/safe/dir")