Files

File and filesystem operations: hashing, path management, remote downloads, shell operations.

File utilities for siege_utilities.

This module provides comprehensive file operations including: - Path manipulation and directory management - File operations (copy, move, delete, etc.) - Remote file downloads with retry logic - File hashing and integrity verification

All functions are available at the top level for mutual availability across modules.

siege_utilities.files.ensure_path_exists(desired_path)

Ensure a directory path exists, creating it if necessary.

SECURITY: Validates paths to prevent path traversal and sensitive file access.

Parameters:

desired_path (str | Path) – Path to ensure exists

Returns:

Path object for the created directory

Raises:

PathSecurityError – If path fails security validation

Return type:

Path

Example

>>> path = ensure_path_exists("logs/2024/01")
>>> print(f"Directory created: {path}")
>>> ensure_path_exists("../../../etc")

siege_utilities.files.unzip_file_to_directory(zip_file_path, extract_to=None, create_subdirectory=True)

Extract a zip file to a directory.

SECURITY: Validates paths to prevent zip slip attacks and path traversal.

Parameters:

• zip_file_path (str | Path) – Path to the zip file

• extract_to (str | Path | None) – Directory to extract to (defaults to zip file’s parent)

• create_subdirectory (bool) – Whether to create a subdirectory for extracted files

Returns:

Path to the extraction directory

Raises:

• FileNotFoundError – If zip file does not exist or is not a file

• zipfile.BadZipFile – If file is not a valid zip

• PathSecurityError – If paths fail security validation

• OSError – If extraction fails

Return type:

Path

Example

>>> extract_dir = unzip_file_to_directory("data.zip")
>>> print(f"Files extracted to: {extract_dir}")

siege_utilities.files.get_file_extension(file_path)

Get the file extension from a file path.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File extension (including the dot)

Raises:

PathSecurityError – If path fails security validation

Return type:

str

Example

>>> ext = get_file_extension("document.pdf")
>>> print(f"File extension: {ext}")
>>> get_file_extension("../../../etc/passwd")

siege_utilities.files.get_file_name_without_extension(file_path)

Get the file name without its extension.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File name without extension

Raises:

PathSecurityError – If path fails security validation

Return type:

str

Example

>>> name = get_file_name_without_extension("document.pdf")
>>> print(f"File name: {name}")
>>> get_file_name_without_extension("../../../etc/shadow")

siege_utilities.files.is_hidden_file(file_path)

Check if a file or directory is hidden.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to check

Returns:

True if the file/directory is hidden

Raises:

PathSecurityError – If path fails security validation

Return type:

bool

Example

>>> if is_hidden_file(".config"):
...     print("Hidden file detected")
>>> is_hidden_file("../../.ssh/id_rsa")

siege_utilities.files.get_relative_path(base_path, target_path)

Get the relative path from base_path to target_path.

SECURITY: Validates both base and target paths to prevent path traversal attacks.

Parameters:

• base_path (str | Path) – Base directory path

• target_path (str | Path) – Target file/directory path

Returns:

Relative path from base to target

Raises:

• ValueError – If target is not relative to base

• PathSecurityError – If paths fail security validation

Return type:

Path

Example

>>> rel_path = get_relative_path("/home/user", "/home/user/documents/file.txt")
>>> print(f"Relative path: {rel_path}")

siege_utilities.files.find_files_by_pattern(directory, pattern='*', recursive=False)

Find files matching a pattern in a directory.

SECURITY: Validates directory path to prevent path traversal attacks.

Parameters:

• directory (str | Path) – Directory to search in

• pattern (str) – Glob pattern to match

• recursive (bool) – Whether to search recursively

Returns:

List of matching file paths

Raises:

• PathSecurityError – If path fails security validation

• OSError – If the directory cannot be read

Return type:

list[Path]

Example

>>> files = find_files_by_pattern("data", "*.csv", recursive=True)
>>> print(f"Found {len(files)} CSV files")
>>> find_files_by_pattern("../../../etc", "*")

siege_utilities.files.create_backup_path(original_path, backup_suffix='.backup', backup_dir=None)

Create a backup path for a file.

SECURITY: Validates both original and backup paths to prevent path traversal attacks.

Parameters:

• original_path (str | Path) – Path to the original file

• backup_suffix (str) – Suffix to add to the backup filename

• backup_dir (str | Path | None) – Directory to place backup in (defaults to original file’s directory)

Returns:

Path for the backup file

Raises:

PathSecurityError – If paths fail security validation

Return type:

Path

Example

>>> backup_path = create_backup_path("config.yaml", ".bak")
>>> print(f"Backup will be saved to: {backup_path}")
>>> create_backup_path("/etc/passwd", ".bak")

siege_utilities.files.normalize_path(path)

Normalize a file path, resolving any relative components.

This function expands ~ (home directory), resolves .. (parent directory), and converts to an absolute path. It does NOT validate for security - use validate_safe_path() if you need security validation.

Parameters:

path (str | Path) – Path to normalize

Returns:

Normalized absolute path

Return type:

Path

Example

>>> norm_path = normalize_path("~/../data/file.txt")
>>> print(f"Normalized path: {norm_path}")
>>> norm_path = normalize_path("./../data/file.txt")

siege_utilities.files.remove_tree(path)

Remove a file or directory tree safely.

SECURITY: Validates paths to prevent removing sensitive system files.

Parameters:

path (str | Path) – Path to file or directory to remove

Raises:

• FileNotFoundError – If path does not exist

• PathSecurityError – If path fails security validation

• OSError – If removal fails

Return type:

None

Example

>>> remove_tree("temp_directory")
>>> remove_tree(Path("old_files"))

siege_utilities.files.file_exists(path)

Check if a file exists at the specified path.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

path (str | Path) – Path to check

Returns:

True if file exists, False otherwise

Raises:

PathSecurityError – If path fails security validation

Return type:

bool

Example

>>> if file_exists("config.yaml"):
...     print("Config file found")
>>> file_exists("../../../etc/passwd")

Security Changes:

• Now validates paths to block path traversal

• Blocks access to sensitive system files

siege_utilities.files.touch_file(path, create_parents=True)

Create an empty file, creating parent directories if needed.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

• path (str | Path) – Path to the file to create

• create_parents (bool) – Whether to create parent directories

Raises:

• PathSecurityError – If path fails security validation

• OSError – If file creation fails

Return type:

None

Example

>>> touch_file("logs/app.log")
>>> touch_file(Path("data/output.txt"))

siege_utilities.files.count_lines(file_path, encoding='utf-8')

Count the total number of lines in a text file.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

• file_path (str | Path) – Path to the text file

• encoding (str) – File encoding to use

Returns:

Number of lines

Raises:

• FileNotFoundError – If file does not exist or is not a file

• PathSecurityError – If path fails security validation

• OSError – If file cannot be read

• UnicodeDecodeError – If file cannot be decoded with given encoding

Return type:

int

Example

>>> line_count = count_lines("data.csv")
>>> print(f"File has {line_count} lines")

siege_utilities.files.copy_file(source, destination, overwrite=False)

Copy a file from source to destination.

SECURITY: Validates both source and destination paths.

Parameters:

• source (str | Path) – Source file path

• destination (str | Path) – Destination file path

• overwrite (bool) – Whether to overwrite existing files

Raises:

• FileNotFoundError – If source does not exist or is not a file

• FileExistsError – If destination exists and overwrite=False

• PathSecurityError – If paths fail security validation

• OSError – If copy fails

Return type:

None

Example

>>> copy_file("source.txt", "backup/source.txt")
>>> copy_file(Path("config.yaml"), Path("config.yaml.bak"))

siege_utilities.files.move_file(source, destination, overwrite=False)

Move a file from source to destination.

SECURITY: Validates both source and destination paths.

Parameters:

• source (str | Path) – Source file path

• destination (str | Path) – Destination file path

• overwrite (bool) – Whether to overwrite existing files

Raises:

• FileNotFoundError – If source does not exist or is not a file

• FileExistsError – If destination exists and overwrite=False

• PathSecurityError – If paths fail security validation

• OSError – If move fails

Return type:

None

Example

>>> move_file("temp.txt", "archive/temp.txt")
>>> move_file(Path("old.log"), Path("logs/old.log"))

siege_utilities.files.get_file_size(file_path)

Get the size of a file in bytes.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File size in bytes

Raises:

• FileNotFoundError – If file does not exist or is not a file

• PathSecurityError – If path fails security validation

• OSError – If file size cannot be determined

Return type:

int

Example

>>> size = get_file_size("large_file.zip")
>>> print(f"File size: {size} bytes")

siege_utilities.files.list_directory(path, pattern='*', include_dirs=True, include_files=True)

List contents of a directory with optional filtering.

SECURITY: Validates directory path to prevent traversal attacks.

Parameters:

• path (str | Path) – Directory path to list

• pattern (str) – Glob pattern for filtering files

• include_dirs (bool) – Whether to include directories

• include_files (bool) – Whether to include files

Returns:

List of Path objects

Raises:

• FileNotFoundError – If directory does not exist or is not a directory

• PathSecurityError – If path fails security validation

• OSError – If directory cannot be read

Return type:

List[Path]

Example

>>> files = list_directory("data", "*.csv")
>>> dirs = list_directory("logs", include_files=False)

siege_utilities.files.run_command(command, cwd=None, timeout=30, capture_output=True, allow_list=None, unsafe=False)

Run an argv command with optional command and argument validation.

SECURITY: This function now validates commands by default. Only whitelisted commands are allowed unless unsafe=True is explicitly set. Validation rejects selected characters and known sensitive paths; it is not a sandbox and does not guarantee read-only behavior.

Parameters:

• command (str | List[str]) – Command to run (string or list)

• cwd (str | Path | None) – Working directory for execution and relative operand validation

• timeout (int | None) – Timeout in seconds (default: 30)

• capture_output (bool) – Whether to capture command output

• allow_list (set | None) – Optional custom whitelist of allowed commands

• unsafe (bool) – If True, bypass command and argument validation. The command is still executed via argv (shell=False) — string commands are shlex-split and run without shell expansion. Use this when you legitimately need a binary that isn’t in the allow-list, not when you need shell features.

Returns:

CompletedProcess object on success or non-zero exit. Arguments and output are omitted from logs. Raised diagnostics omit command details and captured output, which may contain credentials.

Raises:

• SecurityError – If command fails security validation (when unsafe=False).

• subprocess.TimeoutExpired – If the command exceeds timeout.

• OSError – If the command binary cannot be found or executed.

Return type:

CompletedProcess

Example

>>> result = run_command("ls -la")
>>> if result and result.returncode == 0:
...     print("Command succeeded")
>>> result = run_command("rm -rf /")
>>> result = run_command("git status", allow_list={'git', 'ls'})
>>> result = run_command("custom_cmd", unsafe=True)

Security Changes:

• Previously used shell=True with no validation (CRITICAL VULNERABILITY)

• Now validates all commands against whitelist by default

• Rejects selected metacharacters, ‘..’, and known sensitive paths

• Use unsafe=True ONLY with trusted input in trusted environments

siege_utilities.files.delete_existing_file_and_replace_it_with_an_empty_file(file_path, create_parents=True)

Backward compatibility function that deletes existing file and replaces it with empty file.

Deprecated since version 3.19.0: Use touch_file() for creating empty files. Note that touch_file does not delete the existing file first; if you need delete-then-create semantics, call Path.unlink() before touch_file(). Will be removed in v4.0.0.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

• file_path (str | Path) – Path to the file

• create_parents (bool) – Whether to create parent directories

Raises:

• PathSecurityError – If path fails security validation

• OSError – If file creation fails

Return type:

None

siege_utilities.files.ensure_directory_exists(directory)

Ensure a directory exists, creating it if necessary.

Parameters:

directory (str | Path) – Path to directory

Returns:

Path object for the directory

Raises:

PathSecurityError – If path fails security validation.

Return type:

Path

siege_utilities.files.safe_file_write(file_path, content, encoding='utf-8', create_dirs=True)

Atomically write content to a file, optionally creating parent directories.

Parameters:

• file_path (str | Path) – Path to file

• content (str) – Content to write

• encoding (str) – File encoding (default: utf-8)

• create_dirs (bool) – Create parent directories if they don’t exist

Returns:

True if successful, False otherwise

Raises:

PathSecurityError – If path fails security validation.

Return type:

bool

siege_utilities.files.safe_file_read(file_path, encoding='utf-8', default=None)

Safely read content from a file.

Parameters:

• file_path (str | Path) – Path to file

• encoding (str) – File encoding (default: utf-8)

• default (str | None) – Default value to return if file doesn’t exist or error occurs

Returns:

File content or default value

Raises:

PathSecurityError – If path fails security validation.

Return type:

str | None

siege_utilities.files.safe_json_write(file_path, data, indent=2, create_dirs=True)

Atomically write data to a JSON file.

Parameters:

• file_path (str | Path) – Path to JSON file

• data (Any) – Data to write (must be JSON serializable)

• indent (int) – JSON indentation (default: 2)

• create_dirs (bool) – Create parent directories if they don’t exist

Returns:

True if successful, False otherwise

Raises:

PathSecurityError – If path fails security validation.

Return type:

bool

siege_utilities.files.safe_json_read(file_path, default=None)

Safely read data from a JSON file.

Parameters:

• file_path (str | Path) – Path to JSON file

• default (Any | None) – Default value to return if file doesn’t exist or error occurs

Returns:

Parsed JSON data or default value

Raises:

PathSecurityError – If path fails security validation.

Return type:

Any | None

siege_utilities.files.get_file_size_mb(file_path)

Get file size in megabytes.

Parameters:

file_path (str | Path) – Path to file

Returns:

File size in MB.

Raises:

• FileNotFoundError – If the file does not exist.

• PathSecurityError – If path fails security validation.

• OSError – If the file size cannot be determined.

Return type:

float

siege_utilities.files.list_files_recursive(directory, pattern='*', exclude_dirs=True)

List all files in a directory recursively.

Parameters:

• directory (str | Path) – Directory to search

• pattern (str) – Glob pattern (default: “*” for all files)

• exclude_dirs (bool) – Exclude directories from results

Returns:

List of Path objects

Raises:

• FileNotFoundError – If the directory does not exist.

• PathSecurityError – If path fails security validation.

• OSError – If the directory cannot be read.

Return type:

list[Path]

siege_utilities.files.download_file(url, local_filename, chunk_size=8192, timeout=30, verify_ssl=True, allow_insecure_fallback=False)

Download a file from a URL to a local file with progress bar.

SECURITY: Validates local file path to prevent path traversal attacks.

SSL behavior: when verify_ssl is True (the default), SSL certificate verification is enforced. If verification fails the original SSLError is re-raised and nothing is written – there is no silent retry with verification disabled. An insecure retry only happens when the caller explicitly sets allow_insecure_fallback=True, which downgrades to an unverified connection (vulnerable to man-in-the-middle) and is logged at ERROR. Leave it at the default False unless you fully accept that risk.

Parameters:

• url (str) – The URL to download from

• local_filename (str | Path) – The local path where the file should be saved

• chunk_size (int) – Size of chunks to download at once

• timeout (int) – Request timeout in seconds

• verify_ssl (bool) – Whether to verify SSL certificates

• allow_insecure_fallback (bool) – If True, an SSL failure triggers an explicit retry with verification disabled (insecure; logged at ERROR). Defaults to False, in which case SSL failures are raised.

Returns:

The local filename as a string

Raises:

• PathSecurityError – If local path fails security validation

• ConnectionError – If HTTP request fails (non-2xx status)

• requests.exceptions.SSLError – If SSL verification fails and allow_insecure_fallback is False

• requests.exceptions.Timeout – If request times out

• requests.exceptions.RequestException – If request fails

• OSError – If file write fails

Return type:

str

Example

>>> result = download_file("https://example.com/file.zip", "downloads/file.zip")
>>> print(f"Downloaded to {result}")

siege_utilities.files.generate_local_path_from_url(url, directory_path, as_string=True)

Generate a local file path from a URL.

SECURITY: Validates directory path to prevent path traversal attacks.

Parameters:

• url (str) – URL to extract filename from

• directory_path (str | Path) – Directory where the file should be saved

• as_string (bool) – Whether to return the result as a string

Returns:

Path object or string

Raises:

• ValueError – If filename cannot be extracted from URL

• PathSecurityError – If directory path fails security validation

• OSError – If directory creation fails

Return type:

Path | str

Example

>>> path = generate_local_path_from_url("https://example.com/file.zip", "downloads")
>>> print(f"Local path: {path}")

siege_utilities.files.download_file_with_retry(url, local_filename, max_retries=3, retry_delay=5, **kwargs)

Download a file with automatic retry on failure.

Parameters:

• url (str) – The URL to download from

• local_filename (str | Path) – The local path where the file should be saved

• max_retries (int) – Maximum number of retry attempts

• retry_delay (int) – Delay between retries in seconds

• **kwargs – Additional arguments passed to download_file. SSL behavior is inherited: verify_ssl defaults to True and SSL failures raise unless allow_insecure_fallback=True is passed through explicitly.

Returns:

The local filename as a string

Raises:

• requests.exceptions.SSLError – If SSL verification fails; raised immediately without retry unless download_file handles an explicitly allowed insecure fallback

• ConnectionError – If all retry attempts fail

• requests.exceptions.RequestException – If all retry attempts fail

Return type:

str

Example

>>> result = download_file_with_retry("https://example.com/file.zip", "file.zip")
>>> print(f"Downloaded to {result}")

siege_utilities.files.get_file_info(url, timeout=10)

Get information about a remote file without downloading it.

Parameters:

• url (str) – URL to get information about

• timeout (int) – Request timeout in seconds

Returns:

Dictionary with file information

Raises:

• ConnectionError – If HTTP request fails (non-2xx status)

• requests.exceptions.RequestException – If request fails

• ImportError – If requests library is not available

Return type:

dict

Example

>>> info = get_file_info("https://example.com/file.zip")
>>> print(f"File size: {info['size']} bytes")

siege_utilities.files.is_downloadable(url, timeout=10)

Check if a URL points to a downloadable file.

Parameters:

• url (str) – URL to check

• timeout (int) – Request timeout in seconds

Returns:

True if the URL points to a downloadable file, False otherwise

Return type:

bool

Example

>>> if is_downloadable("https://example.com/file.zip"):
...     print("URL is downloadable")

siege_utilities.files.get_download_directory()

Get the default download directory for siege_utilities.

Deprecated since version 3.19.0: Use siege_utilities.config.user_config.get_download_directory() instead, or from siege_utilities import get_download_directory. Will be removed in v4.0.0.

Returns:

Path to the download directory

Return type:

Path

class siege_utilities.files.SpatialFormat

Bases: str, Enum

Supported spatial output formats.

GEOPARQUET = 'geoparquet'

PARQUET = 'parquet'

GPKG = 'gpkg'

GEOJSON = 'geojson'

TOPOJSON = 'topojson'

CSV = 'csv'

SHAPEFILE = 'shp'

__new__(value)

class siege_utilities.files.TabularFormat

Bases: str, Enum

Supported tabular output formats.

PARQUET = 'parquet'

CSV = 'csv'

EXCEL = 'excel'

JSON = 'json'

__new__(value)

siege_utilities.files.save_spatial(gdf, path, fmt=SpatialFormat.GEOPARQUET, **kwargs)

Write a GeoDataFrame to path in the requested format.

Parameters:

• gdf (geopandas.GeoDataFrame) – Data to write.

• path (str or Path) – Destination file path.

• fmt (SpatialFormat) – Output format (default GEOPARQUET).

• **kwargs – Passed through to the underlying writer.

Return type:

Path to the written file.

siege_utilities.files.save_tabular(df, path, fmt=TabularFormat.PARQUET, **kwargs)

Write a DataFrame to path in the requested format.

Parameters:

• df (pandas.DataFrame) – Data to write.

• path (str or Path) – Destination file path.

• fmt (TabularFormat) – Output format (default PARQUET).

• **kwargs – Passed through to the underlying writer.

Return type:

Path to the written file.

siege_utilities.files.generate_sha256_hash_for_file(file_path)

Generate SHA256 hash for a file - chunked reading for large files.

Deprecated since version 3.19.0: Use calculate_file_hash() or get_file_hash() instead. Will be removed in v4.0.0.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path – Path to the file (str or Path object)

Returns:

SHA256 hash as hexadecimal string

Raises:

• FileNotFoundError – If file does not exist or is not a file

• PathSecurityError – If path fails security validation

• OSError – If file cannot be read

Return type:

str

Example

>>> hash_val = generate_sha256_hash_for_file("data.txt")

siege_utilities.files.get_file_hash(file_path, algorithm='sha256')

Generate hash for a file using specified algorithm.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

• file_path – Path to the file (str or Path object)

• algorithm – Hash algorithm to use (‘sha256’, ‘md5’, ‘sha1’, etc.)

Returns:

Hash as hexadecimal string

Raises:

• FileNotFoundError – If file does not exist or is not a file

• PathSecurityError – If path fails security validation

• OSError – If file cannot be read

• ValueError – If algorithm is not supported

Return type:

str

Example

>>> hash_val = get_file_hash("data.txt", "sha256")

siege_utilities.files.calculate_file_hash(file_path)

Alias for get_file_hash with SHA256 - for backward compatibility.

Return type:

str

siege_utilities.files.get_quick_file_signature(file_path)

Generate a quick file signature using file stats + partial hash Faster for change detection, not cryptographically secure

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path – Path to the file

Returns:

Quick signature string (‘missing’, fallback stat string, or hash)

Raises:

• PathSecurityError – If path fails security validation

• Exception – If both primary and fallback signature generation fail (logged before re-raise)

Return type:

str

Example

>>> sig = get_quick_file_signature("data.txt")
>>> get_quick_file_signature("/etc/passwd")

Security Changes:

• Now validates paths to block path traversal

• Blocks access to sensitive system files

siege_utilities.files.verify_file_integrity(file_path, expected_hash, algorithm='sha256')

Verify file integrity by comparing with expected hash.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files (through get_file_hash).

Parameters:

• file_path – Path to the file

• expected_hash – Expected hash value

• algorithm – Hash algorithm used

Returns:

True if file matches expected hash, False if hash does not match

Raises:

• FileNotFoundError – If file does not exist

• PathSecurityError – If path fails security validation

• OSError – If file cannot be read

Return type:

bool

Example

>>> expected = "abc123..."
>>> if verify_file_integrity("data.txt", expected):
...     print("File integrity verified")

Submodules

Modern file operations for siege_utilities. Provides clean, type-safe file manipulation utilities.

siege_utilities.files.operations.sanitize_filename(name)

Convert an arbitrary string into a safe filename component.

Unsafe characters (path separators, control characters, punctuation other than . and -) are replaced with underscores, runs of underscores are collapsed, and leading/trailing dots and underscores are trimmed. The result is always a non-empty string and the operation is idempotent (sanitize_filename(sanitize_filename(x)) == sanitize_filename(x)).

Parameters:

name (str) – Arbitrary input string.

Returns:

A filesystem-safe filename component.

Return type:

str

Raises:

TypeError – If name is not a string.

siege_utilities.files.operations.atomic_write_path(target)

Context manager that yields a temporary path for writing, then atomically replaces the target on successful exit.

Usage:

with atomic_write_path("/data/output.csv") as tmp:
    df.to_csv(tmp, index=False)
# target is now atomically updated

If writing or syncing the file raises, the temp file is cleaned up and the target is left unchanged. Files use mode 0666 filtered by the process umask, including replacements. This is for data, not credentials: use secure_write_text for secrets. Flush and close writers before exit.

Parameters:

target (str | Path)

Return type:

Generator[Path, None, None]

siege_utilities.files.operations.atomic_write_shapefile(gdf, target, **kwargs)

Write a GeoDataFrame as a Shapefile atomically.

Shapefiles produce sidecar files (.shx, .dbf, .prj, .cpg). This writes to a temp stem then renames all produced files to the final stem.

Parameters:

target (str | Path)

Return type:

None

siege_utilities.files.operations.remove_tree(path)

Remove a file or directory tree safely.

SECURITY: Validates paths to prevent removing sensitive system files.

Parameters:

path (str | Path) – Path to file or directory to remove

Raises:

• FileNotFoundError – If path does not exist

• PathSecurityError – If path fails security validation

• OSError – If removal fails

Return type:

None

Example

>>> remove_tree("temp_directory")
>>> remove_tree(Path("old_files"))

siege_utilities.files.operations.file_exists(path)

Check if a file exists at the specified path.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

path (str | Path) – Path to check

Returns:

True if file exists, False otherwise

Raises:

PathSecurityError – If path fails security validation

Return type:

bool

Example

>>> if file_exists("config.yaml"):
...     print("Config file found")
>>> file_exists("../../../etc/passwd")

Security Changes:

• Now validates paths to block path traversal

• Blocks access to sensitive system files

siege_utilities.files.operations.touch_file(path, create_parents=True)

Create an empty file, creating parent directories if needed.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

• path (str | Path) – Path to the file to create

• create_parents (bool) – Whether to create parent directories

Raises:

• PathSecurityError – If path fails security validation

• OSError – If file creation fails

Return type:

None

Example

>>> touch_file("logs/app.log")
>>> touch_file(Path("data/output.txt"))

siege_utilities.files.operations.count_lines(file_path, encoding='utf-8')

Count the total number of lines in a text file.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

• file_path (str | Path) – Path to the text file

• encoding (str) – File encoding to use

Returns:

Number of lines

Raises:

• FileNotFoundError – If file does not exist or is not a file

• PathSecurityError – If path fails security validation

• OSError – If file cannot be read

• UnicodeDecodeError – If file cannot be decoded with given encoding

Return type:

int

Example

>>> line_count = count_lines("data.csv")
>>> print(f"File has {line_count} lines")

siege_utilities.files.operations.copy_file(source, destination, overwrite=False)

Copy a file from source to destination.

SECURITY: Validates both source and destination paths.

Parameters:

• source (str | Path) – Source file path

• destination (str | Path) – Destination file path

• overwrite (bool) – Whether to overwrite existing files

Raises:

• FileNotFoundError – If source does not exist or is not a file

• FileExistsError – If destination exists and overwrite=False

• PathSecurityError – If paths fail security validation

• OSError – If copy fails

Return type:

None

Example

>>> copy_file("source.txt", "backup/source.txt")
>>> copy_file(Path("config.yaml"), Path("config.yaml.bak"))

siege_utilities.files.operations.move_file(source, destination, overwrite=False)

Move a file from source to destination.

SECURITY: Validates both source and destination paths.

Parameters:

• source (str | Path) – Source file path

• destination (str | Path) – Destination file path

• overwrite (bool) – Whether to overwrite existing files

Raises:

• FileNotFoundError – If source does not exist or is not a file

• FileExistsError – If destination exists and overwrite=False

• PathSecurityError – If paths fail security validation

• OSError – If move fails

Return type:

None

Example

>>> move_file("temp.txt", "archive/temp.txt")
>>> move_file(Path("old.log"), Path("logs/old.log"))

siege_utilities.files.operations.get_file_size(file_path)

Get the size of a file in bytes.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File size in bytes

Raises:

• FileNotFoundError – If file does not exist or is not a file

• PathSecurityError – If path fails security validation

• OSError – If file size cannot be determined

Return type:

int

Example

>>> size = get_file_size("large_file.zip")
>>> print(f"File size: {size} bytes")

siege_utilities.files.operations.list_directory(path, pattern='*', include_dirs=True, include_files=True)

List contents of a directory with optional filtering.

SECURITY: Validates directory path to prevent traversal attacks.

Parameters:

• path (str | Path) – Directory path to list

• pattern (str) – Glob pattern for filtering files

• include_dirs (bool) – Whether to include directories

• include_files (bool) – Whether to include files

Returns:

List of Path objects

Raises:

• FileNotFoundError – If directory does not exist or is not a directory

• PathSecurityError – If path fails security validation

• OSError – If directory cannot be read

Return type:

List[Path]

Example

>>> files = list_directory("data", "*.csv")
>>> dirs = list_directory("logs", include_files=False)

siege_utilities.files.operations.run_command(command, cwd=None, timeout=30, capture_output=True, allow_list=None, unsafe=False)

Run an argv command with optional command and argument validation.

SECURITY: This function now validates commands by default. Only whitelisted commands are allowed unless unsafe=True is explicitly set. Validation rejects selected characters and known sensitive paths; it is not a sandbox and does not guarantee read-only behavior.

Parameters:

• command (str | List[str]) – Command to run (string or list)

• cwd (str | Path | None) – Working directory for execution and relative operand validation

• timeout (int | None) – Timeout in seconds (default: 30)

• capture_output (bool) – Whether to capture command output

• allow_list (set | None) – Optional custom whitelist of allowed commands

• unsafe (bool) – If True, bypass command and argument validation. The command is still executed via argv (shell=False) — string commands are shlex-split and run without shell expansion. Use this when you legitimately need a binary that isn’t in the allow-list, not when you need shell features.

Returns:

CompletedProcess object on success or non-zero exit. Arguments and output are omitted from logs. Raised diagnostics omit command details and captured output, which may contain credentials.

Raises:

• SecurityError – If command fails security validation (when unsafe=False).

• subprocess.TimeoutExpired – If the command exceeds timeout.

• OSError – If the command binary cannot be found or executed.

Return type:

CompletedProcess

Example

>>> result = run_command("ls -la")
>>> if result and result.returncode == 0:
...     print("Command succeeded")
>>> result = run_command("rm -rf /")
>>> result = run_command("git status", allow_list={'git', 'ls'})
>>> result = run_command("custom_cmd", unsafe=True)

Security Changes:

• Previously used shell=True with no validation (CRITICAL VULNERABILITY)

• Now validates all commands against whitelist by default

• Rejects selected metacharacters, ‘..’, and known sensitive paths

• Use unsafe=True ONLY with trusted input in trusted environments

siege_utilities.files.operations.ensure_directory_exists(directory)

Ensure a directory exists, creating it if necessary.

Parameters:

directory (str | Path) – Path to directory

Returns:

Path object for the directory

Raises:

PathSecurityError – If path fails security validation.

Return type:

Path

siege_utilities.files.operations.safe_file_write(file_path, content, encoding='utf-8', create_dirs=True)

Atomically write content to a file, optionally creating parent directories.

Parameters:

• file_path (str | Path) – Path to file

• content (str) – Content to write

• encoding (str) – File encoding (default: utf-8)

• create_dirs (bool) – Create parent directories if they don’t exist

Returns:

True if successful, False otherwise

Raises:

PathSecurityError – If path fails security validation.

Return type:

bool

siege_utilities.files.operations.safe_file_read(file_path, encoding='utf-8', default=None)

Safely read content from a file.

Parameters:

• file_path (str | Path) – Path to file

• encoding (str) – File encoding (default: utf-8)

• default (str | None) – Default value to return if file doesn’t exist or error occurs

Returns:

File content or default value

Raises:

PathSecurityError – If path fails security validation.

Return type:

str | None

siege_utilities.files.operations.safe_json_write(file_path, data, indent=2, create_dirs=True)

Atomically write data to a JSON file.

Parameters:

• file_path (str | Path) – Path to JSON file

• data (Any) – Data to write (must be JSON serializable)

• indent (int) – JSON indentation (default: 2)

• create_dirs (bool) – Create parent directories if they don’t exist

Returns:

True if successful, False otherwise

Raises:

PathSecurityError – If path fails security validation.

Return type:

bool

siege_utilities.files.operations.safe_json_read(file_path, default=None)

Safely read data from a JSON file.

Parameters:

• file_path (str | Path) – Path to JSON file

• default (Any | None) – Default value to return if file doesn’t exist or error occurs

Returns:

Parsed JSON data or default value

Raises:

PathSecurityError – If path fails security validation.

Return type:

Any | None

siege_utilities.files.operations.get_file_size_mb(file_path)

Get file size in megabytes.

Parameters:

file_path (str | Path) – Path to file

Returns:

File size in MB.

Raises:

• FileNotFoundError – If the file does not exist.

• PathSecurityError – If path fails security validation.

• OSError – If the file size cannot be determined.

Return type:

float

siege_utilities.files.operations.list_files_recursive(directory, pattern='*', exclude_dirs=True)

List all files in a directory recursively.

Parameters:

• directory (str | Path) – Directory to search

• pattern (str) – Glob pattern (default: “*” for all files)

• exclude_dirs (bool) – Exclude directories from results

Returns:

List of Path objects

Raises:

• FileNotFoundError – If the directory does not exist.

• PathSecurityError – If path fails security validation.

• OSError – If the directory cannot be read.

Return type:

list[Path]

siege_utilities.files.operations.rmtree(path)

Remove a file or directory tree safely.

SECURITY: Validates paths to prevent removing sensitive system files.

Parameters:

path (str | Path) – Path to file or directory to remove

Raises:

• FileNotFoundError – If path does not exist

• PathSecurityError – If path fails security validation

• OSError – If removal fails

Return type:

None

Example

>>> remove_tree("temp_directory")
>>> remove_tree(Path("old_files"))

siege_utilities.files.operations.check_if_file_exists_at_path(path)

Check if a file exists at the specified path.

Deprecated since version 3.19.0: Use file_exists() instead. Will be removed in v4.0.0.

Parameters:

path (str | Path)

Return type:

bool

siege_utilities.files.operations.delete_existing_file_and_replace_it_with_an_empty_file(file_path, create_parents=True)

Backward compatibility function that deletes existing file and replaces it with empty file.

Deprecated since version 3.19.0: Use touch_file() for creating empty files. Note that touch_file does not delete the existing file first; if you need delete-then-create semantics, call Path.unlink() before touch_file(). Will be removed in v4.0.0.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

• file_path (str | Path) – Path to the file

• create_parents (bool) – Whether to create parent directories

Raises:

• PathSecurityError – If path fails security validation

• OSError – If file creation fails

Return type:

None

siege_utilities.files.operations.count_total_rows_in_file_pythonically(file_path, encoding='utf-8')

Count the total number of lines in a text file.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

• file_path (str | Path) – Path to the text file

• encoding (str) – File encoding to use

Returns:

Number of lines

Raises:

• FileNotFoundError – If file does not exist or is not a file

• PathSecurityError – If path fails security validation

• OSError – If file cannot be read

• UnicodeDecodeError – If file cannot be decoded with given encoding

Return type:

int

Example

>>> line_count = count_lines("data.csv")
>>> print(f"File has {line_count} lines")

Hash Management Functions - Fixed Version Provides standardized hash functions that actually exist and work properly

siege_utilities.files.hashing.calculate_file_hash(file_path)

Alias for get_file_hash with SHA256 - for backward compatibility.

Return type:

str

siege_utilities.files.hashing.generate_sha256_hash_for_file(file_path)

Generate SHA256 hash for a file - chunked reading for large files.

Deprecated since version 3.19.0: Use calculate_file_hash() or get_file_hash() instead. Will be removed in v4.0.0.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path – Path to the file (str or Path object)

Returns:

SHA256 hash as hexadecimal string

Raises:

• FileNotFoundError – If file does not exist or is not a file

• PathSecurityError – If path fails security validation

• OSError – If file cannot be read

Return type:

str

Example

>>> hash_val = generate_sha256_hash_for_file("data.txt")

siege_utilities.files.hashing.get_file_hash(file_path, algorithm='sha256')

Generate hash for a file using specified algorithm.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

• file_path – Path to the file (str or Path object)

• algorithm – Hash algorithm to use (‘sha256’, ‘md5’, ‘sha1’, etc.)

Returns:

Hash as hexadecimal string

Raises:

• FileNotFoundError – If file does not exist or is not a file

• PathSecurityError – If path fails security validation

• OSError – If file cannot be read

• ValueError – If algorithm is not supported

Return type:

str

Example

>>> hash_val = get_file_hash("data.txt", "sha256")

siege_utilities.files.hashing.get_quick_file_signature(file_path)

Generate a quick file signature using file stats + partial hash Faster for change detection, not cryptographically secure

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.

Parameters:

file_path – Path to the file

Returns:

Quick signature string (‘missing’, fallback stat string, or hash)

Raises:

• PathSecurityError – If path fails security validation

• Exception – If both primary and fallback signature generation fail (logged before re-raise)

Return type:

str

Example

>>> sig = get_quick_file_signature("data.txt")
>>> get_quick_file_signature("/etc/passwd")

Security Changes:

• Now validates paths to block path traversal

• Blocks access to sensitive system files

siege_utilities.files.hashing.verify_file_integrity(file_path, expected_hash, algorithm='sha256')

Verify file integrity by comparing with expected hash.

SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files (through get_file_hash).

Parameters:

• file_path – Path to the file

• expected_hash – Expected hash value

• algorithm – Hash algorithm used

Returns:

True if file matches expected hash, False if hash does not match

Raises:

• FileNotFoundError – If file does not exist

• PathSecurityError – If path fails security validation

• OSError – If file cannot be read

Return type:

bool

Example

>>> expected = "abc123..."
>>> if verify_file_integrity("data.txt", expected):
...     print("File integrity verified")

Modern path utilities for siege_utilities. Provides clean, type-safe path manipulation utilities.

siege_utilities.files.paths.ensure_path_exists(desired_path)

Ensure a directory path exists, creating it if necessary.

SECURITY: Validates paths to prevent path traversal and sensitive file access.

Parameters:

desired_path (str | Path) – Path to ensure exists

Returns:

Path object for the created directory

Raises:

PathSecurityError – If path fails security validation

Return type:

Path

Example

>>> path = ensure_path_exists("logs/2024/01")
>>> print(f"Directory created: {path}")
>>> ensure_path_exists("../../../etc")

siege_utilities.files.paths.unzip_file_to_directory(zip_file_path, extract_to=None, create_subdirectory=True)

Extract a zip file to a directory.

SECURITY: Validates paths to prevent zip slip attacks and path traversal.

Parameters:

• zip_file_path (str | Path) – Path to the zip file

• extract_to (str | Path | None) – Directory to extract to (defaults to zip file’s parent)

• create_subdirectory (bool) – Whether to create a subdirectory for extracted files

Returns:

Path to the extraction directory

Raises:

• FileNotFoundError – If zip file does not exist or is not a file

• zipfile.BadZipFile – If file is not a valid zip

• PathSecurityError – If paths fail security validation

• OSError – If extraction fails

Return type:

Path

Example

>>> extract_dir = unzip_file_to_directory("data.zip")
>>> print(f"Files extracted to: {extract_dir}")

siege_utilities.files.paths.get_file_extension(file_path)

Get the file extension from a file path.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File extension (including the dot)

Raises:

PathSecurityError – If path fails security validation

Return type:

str

Example

>>> ext = get_file_extension("document.pdf")
>>> print(f"File extension: {ext}")
>>> get_file_extension("../../../etc/passwd")

siege_utilities.files.paths.get_file_name_without_extension(file_path)

Get the file name without its extension.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to the file

Returns:

File name without extension

Raises:

PathSecurityError – If path fails security validation

Return type:

str

Example

>>> name = get_file_name_without_extension("document.pdf")
>>> print(f"File name: {name}")
>>> get_file_name_without_extension("../../../etc/shadow")

siege_utilities.files.paths.is_hidden_file(file_path)

Check if a file or directory is hidden.

SECURITY: Validates paths to prevent path traversal attacks.

Parameters:

file_path (str | Path) – Path to check

Returns:

True if the file/directory is hidden

Raises:

PathSecurityError – If path fails security validation

Return type:

bool

Example

>>> if is_hidden_file(".config"):
...     print("Hidden file detected")
>>> is_hidden_file("../../.ssh/id_rsa")

siege_utilities.files.paths.get_relative_path(base_path, target_path)

Get the relative path from base_path to target_path.

SECURITY: Validates both base and target paths to prevent path traversal attacks.

Parameters:

• base_path (str | Path) – Base directory path

• target_path (str | Path) – Target file/directory path

Returns:

Relative path from base to target

Raises:

• ValueError – If target is not relative to base

• PathSecurityError – If paths fail security validation

Return type:

Path

Example

>>> rel_path = get_relative_path("/home/user", "/home/user/documents/file.txt")
>>> print(f"Relative path: {rel_path}")

siege_utilities.files.paths.find_files_by_pattern(directory, pattern='*', recursive=False)

Find files matching a pattern in a directory.

SECURITY: Validates directory path to prevent path traversal attacks.

Parameters:

• directory (str | Path) – Directory to search in

• pattern (str) – Glob pattern to match

• recursive (bool) – Whether to search recursively

Returns:

List of matching file paths

Raises:

• PathSecurityError – If path fails security validation

• OSError – If the directory cannot be read

Return type:

list[Path]

Example

>>> files = find_files_by_pattern("data", "*.csv", recursive=True)
>>> print(f"Found {len(files)} CSV files")
>>> find_files_by_pattern("../../../etc", "*")

siege_utilities.files.paths.create_backup_path(original_path, backup_suffix='.backup', backup_dir=None)

Create a backup path for a file.

SECURITY: Validates both original and backup paths to prevent path traversal attacks.

Parameters:

• original_path (str | Path) – Path to the original file

• backup_suffix (str) – Suffix to add to the backup filename

• backup_dir (str | Path | None) – Directory to place backup in (defaults to original file’s directory)

Returns:

Path for the backup file

Raises:

PathSecurityError – If paths fail security validation

Return type:

Path

Example

>>> backup_path = create_backup_path("config.yaml", ".bak")
>>> print(f"Backup will be saved to: {backup_path}")
>>> create_backup_path("/etc/passwd", ".bak")

siege_utilities.files.paths.normalize_path(path)

Normalize a file path, resolving any relative components.

This function expands ~ (home directory), resolves .. (parent directory), and converts to an absolute path. It does NOT validate for security - use validate_safe_path() if you need security validation.

Parameters:

path (str | Path) – Path to normalize

Returns:

Normalized absolute path

Return type:

Path

Example

>>> norm_path = normalize_path("~/../data/file.txt")
>>> print(f"Normalized path: {norm_path}")
>>> norm_path = normalize_path("./../data/file.txt")

siege_utilities.files.paths.unzip_file_to_its_own_directory(zip_file_path, extract_to=None, create_subdirectory=True)

Extract a zip file to a directory.

SECURITY: Validates paths to prevent zip slip attacks and path traversal.

Parameters:

• zip_file_path (str | Path) – Path to the zip file

• extract_to (str | Path | None) – Directory to extract to (defaults to zip file’s parent)

• create_subdirectory (bool) – Whether to create a subdirectory for extracted files

Returns:

Path to the extraction directory

Raises:

• FileNotFoundError – If zip file does not exist or is not a file

• zipfile.BadZipFile – If file is not a valid zip

• PathSecurityError – If paths fail security validation

• OSError – If extraction fails

Return type:

Path

Example

>>> extract_dir = unzip_file_to_directory("data.zip")
>>> print(f"Files extracted to: {extract_dir}")

Modern remote file operations for siege_utilities. Provides clean, type-safe file download utilities.

siege_utilities.files.remote.download_file(url, local_filename, chunk_size=8192, timeout=30, verify_ssl=True, allow_insecure_fallback=False)

Download a file from a URL to a local file with progress bar.

SECURITY: Validates local file path to prevent path traversal attacks.

SSL behavior: when verify_ssl is True (the default), SSL certificate verification is enforced. If verification fails the original SSLError is re-raised and nothing is written – there is no silent retry with verification disabled. An insecure retry only happens when the caller explicitly sets allow_insecure_fallback=True, which downgrades to an unverified connection (vulnerable to man-in-the-middle) and is logged at ERROR. Leave it at the default False unless you fully accept that risk.

Parameters:

• url (str) – The URL to download from

• local_filename (str | Path) – The local path where the file should be saved

• chunk_size (int) – Size of chunks to download at once

• timeout (int) – Request timeout in seconds

• verify_ssl (bool) – Whether to verify SSL certificates

• allow_insecure_fallback (bool) – If True, an SSL failure triggers an explicit retry with verification disabled (insecure; logged at ERROR). Defaults to False, in which case SSL failures are raised.

Returns:

The local filename as a string

Raises:

• PathSecurityError – If local path fails security validation

• ConnectionError – If HTTP request fails (non-2xx status)

• requests.exceptions.SSLError – If SSL verification fails and allow_insecure_fallback is False

• requests.exceptions.Timeout – If request times out

• requests.exceptions.RequestException – If request fails

• OSError – If file write fails

Return type:

str

Example

>>> result = download_file("https://example.com/file.zip", "downloads/file.zip")
>>> print(f"Downloaded to {result}")

siege_utilities.files.remote.generate_local_path_from_url(url, directory_path, as_string=True)

Generate a local file path from a URL.

SECURITY: Validates directory path to prevent path traversal attacks.

Parameters:

• url (str) – URL to extract filename from

• directory_path (str | Path) – Directory where the file should be saved

• as_string (bool) – Whether to return the result as a string

Returns:

Path object or string

Raises:

• ValueError – If filename cannot be extracted from URL

• PathSecurityError – If directory path fails security validation

• OSError – If directory creation fails

Return type:

Path | str

Example

>>> path = generate_local_path_from_url("https://example.com/file.zip", "downloads")
>>> print(f"Local path: {path}")

siege_utilities.files.remote.download_file_with_retry(url, local_filename, max_retries=3, retry_delay=5, **kwargs)

Download a file with automatic retry on failure.

Parameters:

• url (str) – The URL to download from

• local_filename (str | Path) – The local path where the file should be saved

• max_retries (int) – Maximum number of retry attempts

• retry_delay (int) – Delay between retries in seconds

• **kwargs – Additional arguments passed to download_file. SSL behavior is inherited: verify_ssl defaults to True and SSL failures raise unless allow_insecure_fallback=True is passed through explicitly.

Returns:

The local filename as a string

Raises:

• requests.exceptions.SSLError – If SSL verification fails; raised immediately without retry unless download_file handles an explicitly allowed insecure fallback

• ConnectionError – If all retry attempts fail

• requests.exceptions.RequestException – If all retry attempts fail

Return type:

str

Example

>>> result = download_file_with_retry("https://example.com/file.zip", "file.zip")
>>> print(f"Downloaded to {result}")

siege_utilities.files.remote.get_file_info(url, timeout=10)

Get information about a remote file without downloading it.

Parameters:

• url (str) – URL to get information about

• timeout (int) – Request timeout in seconds

Returns:

Dictionary with file information

Raises:

• ConnectionError – If HTTP request fails (non-2xx status)

• requests.exceptions.RequestException – If request fails

• ImportError – If requests library is not available

Return type:

dict

Example

>>> info = get_file_info("https://example.com/file.zip")
>>> print(f"File size: {info['size']} bytes")

siege_utilities.files.remote.is_downloadable(url, timeout=10)

Check if a URL points to a downloadable file.

Parameters:

• url (str) – URL to check

• timeout (int) – Request timeout in seconds

Returns:

True if the URL points to a downloadable file, False otherwise

Return type:

bool

Example

>>> if is_downloadable("https://example.com/file.zip"):
...     print("URL is downloadable")

Execute argv commands with an allow-list and conservative argument checks.

These checks are a guardrail, not a sandbox or a guarantee of read-only behavior.

siege_utilities.files.shell.run_subprocess(command_list, allow_list=None, timeout=30)

Run an argv command after applying validate_command_safety.

Execution uses shell=False, so shell syntax is not interpreted. Validation restricts command names and selected argument patterns; it does not promise read-only behavior or exhaustive validation of each program’s options.

Parameters:

• command_list (str or list[str]) – Command string or argv list to run.

• allow_list (set, optional) – Custom allowed command names, replacing the default allow-list.

• timeout (int, default 30) – Command timeout in seconds.

Returns:

The command output (stdout if successful, stderr if failed, with an error logged for a nonzero exit status). Arguments and output are omitted from logs. Raised diagnostics omit command details and captured output, which may contain credentials.

Return type:

str

Raises:

• SecurityError – If the command fails validation.

• ValueError – If the command is invalid.

• subprocess.TimeoutExpired – If the command times out.

• OSError – If the command cannot be executed.

Examples

>>> output = run_subprocess("ls -la")

siege_utilities.files.shell.validate_command_safety(command, allow_list=None, cwd=None)

Apply command-name, character, traversal, and known sensitive-path checks.

Operands and attached option values are checked using literal, expanded, normalized and resolved path forms relative to cwd, but returned unchanged. Process-relative /proc links (/proc/<pid>/cwd, root, fd, task, map_files) are rejected because they resolve differently in the child. This is not a filesystem sandbox: command options, recursive reads and filesystem changes after validation are not controlled. A custom allow-list delegates command semantics to the caller.

Parameters:

• command (str or list[str]) – Command string or argv list to validate.

• allow_list (set, optional) – Custom allowed command names, replacing the default allow-list.

• cwd (str or os.PathLike, optional) – Working directory used for execution and relative path checks. Defaults to the current process directory.

Returns:

Validated argv with argument text unchanged.

Return type:

list[str]

Raises:

• SecurityError – If the command contains forbidden elements.

• ValueError – If the command is empty or has invalid quoting.

Notes

• Validates command against whitelist

• Rejects selected shell metacharacters, including redirection characters

• Rejects ‘..’ path components

• Rejects known sensitive paths and .ssh path components, including aliases

exception siege_utilities.files.shell.SecurityError

Bases: Exception

Raised when a security violation is detected.

Unified output format support for spatial and tabular data.

Provides save_spatial() and save_tabular() dispatchers that write data to disk in the requested format. Enum types SpatialFormat and TabularFormat enumerate supported file types.

Usage:

from siege_utilities.files.formats import save_spatial, SpatialFormat

save_spatial(gdf, "/tmp/districts.parquet", SpatialFormat.GEOPARQUET)

class siege_utilities.files.formats.SpatialFormat

Bases: str, Enum

Supported spatial output formats.

GEOPARQUET = 'geoparquet'

PARQUET = 'parquet'

GPKG = 'gpkg'

GEOJSON = 'geojson'

TOPOJSON = 'topojson'

CSV = 'csv'

SHAPEFILE = 'shp'

__new__(value)

class siege_utilities.files.formats.TabularFormat

Bases: str, Enum

Supported tabular output formats.

PARQUET = 'parquet'

CSV = 'csv'

EXCEL = 'excel'

JSON = 'json'

__new__(value)

siege_utilities.files.formats.save_spatial(gdf, path, fmt=SpatialFormat.GEOPARQUET, **kwargs)

Write a GeoDataFrame to path in the requested format.

Parameters:

• gdf (geopandas.GeoDataFrame) – Data to write.

• path (str or Path) – Destination file path.

• fmt (SpatialFormat) – Output format (default GEOPARQUET).

• **kwargs – Passed through to the underlying writer.

Return type:

Path to the written file.

siege_utilities.files.formats.save_tabular(df, path, fmt=TabularFormat.PARQUET, **kwargs)

Write a DataFrame to path in the requested format.

Parameters:

• df (pandas.DataFrame) – Data to write.

• path (str or Path) – Destination file path.

• fmt (TabularFormat) – Output format (default PARQUET).

• **kwargs – Passed through to the underlying writer.

Return type:

Path to the written file.

Path validation and sanitization utilities for siege_utilities.

This module provides security-focused path validation to prevent: - Path traversal attacks (../, ~/../, etc.) - Access to sensitive system files and directories - Symlink-based attacks - Null byte injection

Use base_directory to confine paths to a trusted directory. Without a base, the sensitive-path blocklist is not a sandbox.

exception siege_utilities.files.validation.PathSecurityError

Bases: Exception

Raised when a path fails security validation.

siege_utilities.files.validation.is_path_traversal_attempt(path)

Check for parent components, home expansion, or null bytes in a local path.

Both slash styles are treated as separators for this check. Percent sequences such as %2f and %2e%2e are literal filename characters; they are neither decoded nor rejected. Dots within a name are also literal.

Parameters:

path (str or Path) – Local filesystem path to check.

Returns:

Whether the path contains a prohibited component or null byte.

Return type:

bool

Examples

>>> is_path_traversal_attempt("../../../etc/passwd")
True
>>> is_path_traversal_attempt("data/file.txt")
False

siege_utilities.files.validation.is_sensitive_path(path, base_directory=None)

Check if a path points to or contains a sensitive system location.

Under /dev, only /dev/shm and its descendants and the exact pseudo-devices /dev/null, /dev/zero, /dev/random, and /dev/urandom escape the directory ban. The /root tree is blocked except for paths contained in a resolved cwd or base_directory that is /root or one of its descendants. Other sensitive-path checks, including root’s credential files, still apply in both cases.

Parameters:

• path (str or Path) – Path to check, resolved before applying the blocklist.

• base_directory (str or Path, optional) – Operating base for the /root exception; this function does not enforce containment. Relative paths still resolve against the cwd.

Returns:

• bool – True if the path is sensitive or cannot be resolved.

• Example – >>> is_sensitive_path(“/etc/passwd”) True >>> is_sensitive_path(“/tmp/data.txt”) False

Return type:

bool

siege_utilities.files.validation.validate_safe_path(path, allow_absolute=True, base_directory=None)

Validate that a path is safe to access.

Performs the following checks: 1. Blocks path traversal attempts 2. Blocks access to sensitive system paths 3. Optionally restricts to a base directory 4. Blocks null byte injection 5. Resolves symlinks and checks final destination

Without base_directory, the blocklist is not a sandbox: absolute paths outside the listed sensitive locations are allowed by default. Supply a trusted base for untrusted paths. Percent sequences are kept literal. The /dev and /root exceptions follow is_sensitive_path, including the requirement that a path be contained in a resolved cwd or base at or below /root for its tree exception. Root’s credential files remain blocked. Checks describe the filesystem at validation time; they do not prevent concurrent symlink changes between validation and a subsequent operation.

Parameters:

• path (str or Path) – Path to validate. Relative paths are resolved against the current working directory, not joined onto base_directory.

• allow_absolute (bool, default True) – Whether to allow absolute paths.

• base_directory (str or Path, optional) – Resolved containment boundary, including for symlink destinations.

Returns:

Validated absolute path.

Return type:

Path

Raises:

• PathSecurityError – If the path fails security validation.

• ValueError – If the path is invalid.

Examples

>>> # Safe path
>>> validated = validate_safe_path("data/file.txt")
>>>
>>> try:
...     validate_safe_path("../../../etc/passwd")
... except PathSecurityError as e:
...     pass  # raises PathSecurityError
>>>
>>> validated = validate_safe_path("/safe/dir/subdir/file.txt", base_directory="/safe/dir")

siege_utilities.files.validation.validate_file_path(file_path, must_exist=False, base_directory=None)

Validate a file path for read/write operations.

Parameters:

• file_path (str | Path) – Path to validate

• must_exist (bool) – If True, path must exist

• base_directory (str | Path | None) – If provided, path must be within this directory

Returns:

Validated Path object

Raises:

• PathSecurityError – If path fails security validation

• FileNotFoundError – If must_exist=True and file doesn’t exist

• ValueError – If path is invalid

Return type:

Path

Example

>>> path = validate_file_path("data.csv", must_exist=True)
>>>
>>> path = validate_file_path("output.txt", must_exist=False)

siege_utilities.files.validation.validate_directory_path(dir_path, must_exist=False, base_directory=None)

Validate a directory path.

When must_exist is False the directory may be about to be created, so any component that does not exist yet must be free of control characters (C0, DEL, C1). Existing directories with such names still validate, provided they can be stat’ed; one under an unreadable parent counts as new.

Parameters:

• dir_path (str | Path) – Path to validate

• must_exist (bool) – If True, directory must exist

• base_directory (str | Path | None) – If provided, path must be within this directory

Returns:

Validated Path object

Raises:

• PathSecurityError – If path fails security validation, or a component that would be created contains a control character

• FileNotFoundError – If must_exist=True and directory doesn’t exist

• ValueError – If path is invalid or not a directory

Return type:

Path

Example

>>> # Validate directory
>>> path = validate_directory_path("logs", must_exist=True)

siege_utilities.files.validation.reject_unsafe_new_directory_names(directory)

Refuse a directory path whose not-yet-existing components would be unsafe names.

Call this immediately before Path(directory).mkdir(parents=True), for example on a file’s parent. The path is checked as written, not resolved, because that is what mkdir creates: in a/<new>/../b it creates <new> on the way, so <new> is checked even though .. cancels it. Components that already exist (including symlinks) are accepted whatever their name; a new one containing a C0, DEL or C1 control, or a lone surrogate (undecodable bytes), raises. Nothing is created.

Returns:

directory as a Path, unchanged.

Raises:

PathSecurityError – If a component to be created has an unsafe name.

Parameters:

directory (str | Path)

Return type:

Path

siege_utilities.files.validation.reject_unsafe_new_file_path(file_path)

Refuse a file path whose new components, the file name included, would be unsafe names.

Call this immediately before creating file_path. It applies the same as-written check as reject_unsafe_new_directory_names() to every component, so a new file name containing a C0, DEL or C1 control, or a lone surrogate, raises, as does any new parent directory. An existing file may be overwritten whatever its name (#1737). Nothing is created.

Returns:

file_path as a Path, unchanged.

Raises:

PathSecurityError – If a component to be created has an unsafe name.

Parameters:

file_path (str | Path)

Return type:

Path

siege_utilities.files.validation.safe_join_paths(*paths, base_directory=None)

Join paths within a resolved base and apply security validation.

Absolute, rooted, and drive-qualified components after the first are rejected, including Windows forms on other platforms. The first component is the default containment base. A supplied base overrides that boundary; it does not get prepended to the joined path. Symlink destinations must remain within the resolved base, using path-component boundaries.

Parameters:

• *paths (str or Path) – Path components to join. Relative paths use the working directory.

• base_directory (str or Path, optional) – Containment boundary, defaulting to the first component.

Returns:

Validated absolute joined path.

Return type:

Path

Raises:

• PathSecurityError – If a component or the result fails security validation.

• ValueError – If no components are provided or a path is invalid.

Examples

>>> # Safe join
>>> path = safe_join_paths("data", "2024", "file.txt")
>>>
>>> path = safe_join_paths("/safe/dir/subdir", "file.txt", base_directory="/safe/dir")