Files
File and filesystem operations: hashing, path management, remote downloads, shell operations.
File utilities for siege_utilities.
This module provides comprehensive file operations including: - Path manipulation and directory management - File operations (copy, move, delete, etc.) - Remote file downloads with retry logic - File hashing and integrity verification
All functions are available at the top level for mutual availability across modules.
- siege_utilities.files.ensure_path_exists(desired_path)[source]
Ensure a directory path exists, creating it if necessary.
SECURITY: Validates paths to prevent path traversal and sensitive file access.
- Parameters:
- Returns:
Path object for the created directory
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> path = ensure_path_exists("logs/2024/01") >>> print(f"Directory created: {path}") >>> ensure_path_exists("../../../etc")
- siege_utilities.files.unzip_file_to_directory(zip_file_path, extract_to=None, create_subdirectory=True)[source]
Extract a zip file to a directory.
SECURITY: Validates paths to prevent zip slip attacks and path traversal.
- Parameters:
- Returns:
Path to the extraction directory
- Raises:
FileNotFoundError – If zip file does not exist or is not a file
zipfile.BadZipFile – If file is not a valid zip
PathSecurityError – If paths fail security validation
OSError – If extraction fails
- Return type:
Example
>>> extract_dir = unzip_file_to_directory("data.zip") >>> print(f"Files extracted to: {extract_dir}")
- siege_utilities.files.get_file_extension(file_path)[source]
Get the file extension from a file path.
SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Returns:
File extension (including the dot)
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> ext = get_file_extension("document.pdf") >>> print(f"File extension: {ext}") >>> get_file_extension("../../../etc/passwd")
- siege_utilities.files.get_file_name_without_extension(file_path)[source]
Get the file name without its extension.
SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Returns:
File name without extension
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> name = get_file_name_without_extension("document.pdf") >>> print(f"File name: {name}") >>> get_file_name_without_extension("../../../etc/shadow")
Check if a file or directory is hidden.
SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Returns:
True if the file/directory is hidden
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> if is_hidden_file(".config"): ... print("Hidden file detected") >>> is_hidden_file("../../.ssh/id_rsa")
- siege_utilities.files.get_relative_path(base_path, target_path)[source]
Get the relative path from base_path to target_path.
SECURITY: Validates both base and target paths to prevent path traversal attacks.
- Parameters:
- Returns:
Relative path from base to target
- Raises:
ValueError – If target is not relative to base
PathSecurityError – If paths fail security validation
- Return type:
Example
>>> rel_path = get_relative_path("/home/user", "/home/user/documents/file.txt") >>> print(f"Relative path: {rel_path}")
- siege_utilities.files.find_files_by_pattern(directory, pattern='*', recursive=False)[source]
Find files matching a pattern in a directory.
SECURITY: Validates directory path to prevent path traversal attacks.
- Parameters:
- Returns:
List of matching file paths
- Raises:
PathSecurityError – If path fails security validation
OSError – If the directory cannot be read
- Return type:
Example
>>> files = find_files_by_pattern("data", "*.csv", recursive=True) >>> print(f"Found {len(files)} CSV files") >>> find_files_by_pattern("../../../etc", "*")
- siege_utilities.files.create_backup_path(original_path, backup_suffix='.backup', backup_dir=None)[source]
Create a backup path for a file.
SECURITY: Validates both original and backup paths to prevent path traversal attacks.
- Parameters:
- Returns:
Path for the backup file
- Raises:
PathSecurityError – If paths fail security validation
- Return type:
Example
>>> backup_path = create_backup_path("config.yaml", ".bak") >>> print(f"Backup will be saved to: {backup_path}") >>> create_backup_path("/etc/passwd", ".bak")
- siege_utilities.files.normalize_path(path)[source]
Normalize a file path, resolving any relative components.
This function expands ~ (home directory), resolves .. (parent directory), and converts to an absolute path. It does NOT validate for security - use validate_safe_path() if you need security validation.
- Parameters:
- Returns:
Normalized absolute path
- Return type:
Example
>>> norm_path = normalize_path("~/../data/file.txt") >>> print(f"Normalized path: {norm_path}") >>> norm_path = normalize_path("./../data/file.txt")
- siege_utilities.files.remove_tree(path)[source]
Remove a file or directory tree safely.
SECURITY: Validates paths to prevent removing sensitive system files.
- Parameters:
- Raises:
FileNotFoundError – If path does not exist
PathSecurityError – If path fails security validation
OSError – If removal fails
- Return type:
None
Example
>>> remove_tree("temp_directory") >>> remove_tree(Path("old_files"))
- siege_utilities.files.file_exists(path)[source]
Check if a file exists at the specified path.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
- Returns:
True if file exists, False otherwise
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> if file_exists("config.yaml"): ... print("Config file found") >>> file_exists("../../../etc/passwd")
- Security Changes:
Now validates paths to block path traversal
Blocks access to sensitive system files
- siege_utilities.files.touch_file(path, create_parents=True)[source]
Create an empty file, creating parent directories if needed.
SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Raises:
PathSecurityError – If path fails security validation
OSError – If file creation fails
- Return type:
None
Example
>>> touch_file("logs/app.log") >>> touch_file(Path("data/output.txt"))
- siege_utilities.files.count_lines(file_path, encoding='utf-8')[source]
Count the total number of lines in a text file.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
- Returns:
Number of lines
- Raises:
FileNotFoundError – If file does not exist or is not a file
PathSecurityError – If path fails security validation
OSError – If file cannot be read
UnicodeDecodeError – If file cannot be decoded with given encoding
- Return type:
Example
>>> line_count = count_lines("data.csv") >>> print(f"File has {line_count} lines")
- siege_utilities.files.copy_file(source, destination, overwrite=False)[source]
Copy a file from source to destination.
SECURITY: Validates both source and destination paths.
- Parameters:
- Raises:
FileNotFoundError – If source does not exist or is not a file
FileExistsError – If destination exists and overwrite=False
PathSecurityError – If paths fail security validation
OSError – If copy fails
- Return type:
None
Example
>>> copy_file("source.txt", "backup/source.txt") >>> copy_file(Path("config.yaml"), Path("config.yaml.bak"))
- siege_utilities.files.move_file(source, destination, overwrite=False)[source]
Move a file from source to destination.
SECURITY: Validates both source and destination paths.
- Parameters:
- Raises:
FileNotFoundError – If source does not exist or is not a file
FileExistsError – If destination exists and overwrite=False
PathSecurityError – If paths fail security validation
OSError – If move fails
- Return type:
None
Example
>>> move_file("temp.txt", "archive/temp.txt") >>> move_file(Path("old.log"), Path("logs/old.log"))
- siege_utilities.files.get_file_size(file_path)[source]
Get the size of a file in bytes.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
- Returns:
File size in bytes
- Raises:
FileNotFoundError – If file does not exist or is not a file
PathSecurityError – If path fails security validation
OSError – If file size cannot be determined
- Return type:
Example
>>> size = get_file_size("large_file.zip") >>> print(f"File size: {size} bytes")
- siege_utilities.files.list_directory(path, pattern='*', include_dirs=True, include_files=True)[source]
List contents of a directory with optional filtering.
SECURITY: Validates directory path to prevent traversal attacks.
- Parameters:
- Returns:
List of Path objects
- Raises:
FileNotFoundError – If directory does not exist or is not a directory
PathSecurityError – If path fails security validation
OSError – If directory cannot be read
- Return type:
Example
>>> files = list_directory("data", "*.csv") >>> dirs = list_directory("logs", include_files=False)
- siege_utilities.files.run_command(command, cwd=None, timeout=30, capture_output=True, allow_list=None, unsafe=False)[source]
Run a shell command with security validation and proper error handling.
SECURITY: This function now validates commands by default. Only whitelisted commands are allowed unless unsafe=True is explicitly set.
- Parameters:
timeout (int | None) – Timeout in seconds (default: 30)
capture_output (bool) – Whether to capture command output
allow_list (set | None) – Optional custom whitelist of allowed commands
unsafe (bool) – If True, bypass the allow-list check. The command is still executed via argv (shell=False) — string commands are shlex-split and run without shell expansion. Use this when you legitimately need a binary that isn’t in the allow-list, not when you need shell features.
- Returns:
CompletedProcess object on success or non-zero exit.
- Raises:
SecurityError – If command fails security validation (when unsafe=False).
subprocess.TimeoutExpired – If the command exceeds timeout.
OSError – If the command binary cannot be found or executed.
- Return type:
Example
>>> result = run_command("ls -la") >>> if result and result.returncode == 0: ... print("Command succeeded") >>> result = run_command("rm -rf /") >>> result = run_command("git status", allow_list={'git', 'ls'}) >>> result = run_command("custom_cmd", unsafe=True)
- Security Changes:
Previously used shell=True with no validation (CRITICAL VULNERABILITY)
Now validates all commands against whitelist by default
Blocks command injection, path traversal, sensitive file access
Use unsafe=True ONLY with trusted input in trusted environments
- siege_utilities.files.delete_existing_file_and_replace_it_with_an_empty_file(file_path, create_parents=True)[source]
Backward compatibility function that deletes existing file and replaces it with empty file.
Deprecated since version 3.19.0: Use
touch_file()for creating empty files. Note thattouch_filedoes not delete the existing file first; if you need delete-then-create semantics, callPath.unlink()beforetouch_file(). Will be removed in v4.0.0.SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Raises:
PathSecurityError – If path fails security validation
OSError – If file creation fails
- Return type:
None
- siege_utilities.files.ensure_directory_exists(directory)[source]
Ensure a directory exists, creating it if necessary.
- Parameters:
- Returns:
Path object for the directory
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
- siege_utilities.files.safe_file_write(file_path, content, encoding='utf-8', create_dirs=True)[source]
Safely write content to a file, optionally creating parent directories.
- Parameters:
- Returns:
True if successful, False otherwise
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
- siege_utilities.files.safe_file_read(file_path, encoding='utf-8', default=None)[source]
Safely read content from a file.
- Parameters:
- Returns:
File content or default value
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
str | None
- siege_utilities.files.safe_json_write(file_path, data, indent=2, create_dirs=True)[source]
Safely write data to a JSON file.
- Parameters:
- Returns:
True if successful, False otherwise
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
- siege_utilities.files.safe_json_read(file_path, default=None)[source]
Safely read data from a JSON file.
- Parameters:
- Returns:
Parsed JSON data or default value
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
Any | None
- siege_utilities.files.get_file_size_mb(file_path)[source]
Get file size in megabytes.
- Parameters:
- Returns:
File size in MB.
- Raises:
FileNotFoundError – If the file does not exist.
PathSecurityError – If path fails security validation.
OSError – If the file size cannot be determined.
- Return type:
- siege_utilities.files.list_files_recursive(directory, pattern='*', exclude_dirs=True)[source]
List all files in a directory recursively.
- Parameters:
- Returns:
List of Path objects
- Raises:
FileNotFoundError – If the directory does not exist.
PathSecurityError – If path fails security validation.
OSError – If the directory cannot be read.
- Return type:
- siege_utilities.files.download_file(url, local_filename, chunk_size=8192, timeout=30, verify_ssl=True)[source]
Download a file from a URL to a local file with progress bar.
SECURITY: Validates local file path to prevent path traversal attacks.
- Parameters:
- Returns:
The local filename as a string
- Raises:
PathSecurityError – If local path fails security validation
ConnectionError – If HTTP request fails (non-2xx status)
requests.exceptions.Timeout – If request times out
requests.exceptions.RequestException – If request fails
OSError – If file write fails
- Return type:
Example
>>> result = download_file("https://example.com/file.zip", "downloads/file.zip") >>> print(f"Downloaded to {result}")
- siege_utilities.files.generate_local_path_from_url(url, directory_path, as_string=True)[source]
Generate a local file path from a URL.
SECURITY: Validates directory path to prevent path traversal attacks.
- Parameters:
- Returns:
Path object or string
- Raises:
ValueError – If filename cannot be extracted from URL
PathSecurityError – If directory path fails security validation
OSError – If directory creation fails
- Return type:
Example
>>> path = generate_local_path_from_url("https://example.com/file.zip", "downloads") >>> print(f"Local path: {path}")
- siege_utilities.files.download_file_with_retry(url, local_filename, max_retries=3, retry_delay=5, **kwargs)[source]
Download a file with automatic retry on failure.
- Parameters:
- Returns:
The local filename as a string
- Raises:
ConnectionError – If all retry attempts fail
requests.exceptions.RequestException – If all retry attempts fail
- Return type:
Example
>>> result = download_file_with_retry("https://example.com/file.zip", "file.zip") >>> print(f"Downloaded to {result}")
- siege_utilities.files.get_file_info(url, timeout=10)[source]
Get information about a remote file without downloading it.
- Parameters:
- Returns:
Dictionary with file information
- Raises:
ConnectionError – If HTTP request fails (non-2xx status)
requests.exceptions.RequestException – If request fails
ImportError – If requests library is not available
- Return type:
Example
>>> info = get_file_info("https://example.com/file.zip") >>> print(f"File size: {info['size']} bytes")
- siege_utilities.files.is_downloadable(url, timeout=10)[source]
Check if a URL points to a downloadable file.
- Parameters:
- Returns:
True if the URL points to a downloadable file, False otherwise
- Return type:
Example
>>> if is_downloadable("https://example.com/file.zip"): ... print("URL is downloadable")
- siege_utilities.files.get_download_directory()[source]
Get the default download directory for siege_utilities.
Deprecated since version 3.19.0: Use
siege_utilities.config.user_config.get_download_directory()instead, orfrom siege_utilities import get_download_directory. Will be removed in v4.0.0.- Returns:
Path to the download directory
- Return type:
- class siege_utilities.files.SpatialFormat[source]
-
Supported spatial output formats.
- GEOPARQUET = 'geoparquet'
- PARQUET = 'parquet'
- GPKG = 'gpkg'
- GEOJSON = 'geojson'
- TOPOJSON = 'topojson'
- CSV = 'csv'
- SHAPEFILE = 'shp'
- __new__(value)
- class siege_utilities.files.TabularFormat[source]
-
Supported tabular output formats.
- PARQUET = 'parquet'
- CSV = 'csv'
- EXCEL = 'excel'
- JSON = 'json'
- __new__(value)
- siege_utilities.files.save_spatial(gdf, path, fmt=SpatialFormat.GEOPARQUET, **kwargs)[source]
Write a GeoDataFrame to path in the requested format.
- Parameters:
gdf (geopandas.GeoDataFrame) – Data to write.
fmt (SpatialFormat) – Output format (default
GEOPARQUET).**kwargs – Passed through to the underlying writer.
- Return type:
Path to the written file.
- siege_utilities.files.save_tabular(df, path, fmt=TabularFormat.PARQUET, **kwargs)[source]
Write a DataFrame to path in the requested format.
- Parameters:
df (pandas.DataFrame) – Data to write.
fmt (TabularFormat) – Output format (default
PARQUET).**kwargs – Passed through to the underlying writer.
- Return type:
Path to the written file.
- siege_utilities.files.generate_sha256_hash_for_file(file_path)[source]
Generate SHA256 hash for a file - chunked reading for large files.
Deprecated since version 3.19.0: Use
calculate_file_hash()orget_file_hash()instead. Will be removed in v4.0.0.SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
file_path – Path to the file (str or Path object)
- Returns:
SHA256 hash as hexadecimal string
- Raises:
FileNotFoundError – If file does not exist or is not a file
PathSecurityError – If path fails security validation
OSError – If file cannot be read
- Return type:
Example
>>> hash_val = generate_sha256_hash_for_file("data.txt")
- siege_utilities.files.get_file_hash(file_path, algorithm='sha256')[source]
Generate hash for a file using specified algorithm.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
file_path – Path to the file (str or Path object)
algorithm – Hash algorithm to use (‘sha256’, ‘md5’, ‘sha1’, etc.)
- Returns:
Hash as hexadecimal string
- Raises:
FileNotFoundError – If file does not exist or is not a file
PathSecurityError – If path fails security validation
OSError – If file cannot be read
ValueError – If algorithm is not supported
- Return type:
Example
>>> hash_val = get_file_hash("data.txt", "sha256")
- siege_utilities.files.calculate_file_hash(file_path)[source]
Alias for get_file_hash with SHA256 - for backward compatibility.
- Return type:
- siege_utilities.files.get_quick_file_signature(file_path)[source]
Generate a quick file signature using file stats + partial hash Faster for change detection, not cryptographically secure
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
file_path – Path to the file
- Returns:
Quick signature string (‘missing’, fallback stat string, or hash)
- Raises:
PathSecurityError – If path fails security validation
Exception – If both primary and fallback signature generation fail (logged before re-raise)
- Return type:
Example
>>> sig = get_quick_file_signature("data.txt") >>> get_quick_file_signature("/etc/passwd")
- Security Changes:
Now validates paths to block path traversal
Blocks access to sensitive system files
- siege_utilities.files.verify_file_integrity(file_path, expected_hash, algorithm='sha256')[source]
Verify file integrity by comparing with expected hash.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files (through get_file_hash).
- Parameters:
file_path – Path to the file
expected_hash – Expected hash value
algorithm – Hash algorithm used
- Returns:
True if file matches expected hash, False if hash does not match
- Raises:
FileNotFoundError – If file does not exist
PathSecurityError – If path fails security validation
OSError – If file cannot be read
- Return type:
Example
>>> expected = "abc123..." >>> if verify_file_integrity("data.txt", expected): ... print("File integrity verified")
Submodules
Modern file operations for siege_utilities. Provides clean, type-safe file manipulation utilities.
- siege_utilities.files.operations.sanitize_filename(name)[source]
Convert an arbitrary string into a safe filename component.
Unsafe characters (path separators, control characters, punctuation other than
.and-) are replaced with underscores, runs of underscores are collapsed, and leading/trailing dots and underscores are trimmed. The result is always a non-empty string and the operation is idempotent (sanitize_filename(sanitize_filename(x)) == sanitize_filename(x)).
- siege_utilities.files.operations.atomic_write_path(target)[source]
Context manager that yields a temporary path for writing, then atomically replaces the target on successful exit.
Usage:
with atomic_write_path("/data/output.csv") as tmp: df.to_csv(tmp, index=False) # target is now atomically updated
If the block raises, the temp file is cleaned up and the target is left unchanged.
- siege_utilities.files.operations.atomic_write_shapefile(gdf, target, **kwargs)[source]
Write a GeoDataFrame as a Shapefile atomically.
Shapefiles produce sidecar files (.shx, .dbf, .prj, .cpg). This writes to a temp stem then renames all produced files to the final stem.
- siege_utilities.files.operations.remove_tree(path)[source]
Remove a file or directory tree safely.
SECURITY: Validates paths to prevent removing sensitive system files.
- Parameters:
- Raises:
FileNotFoundError – If path does not exist
PathSecurityError – If path fails security validation
OSError – If removal fails
- Return type:
None
Example
>>> remove_tree("temp_directory") >>> remove_tree(Path("old_files"))
- siege_utilities.files.operations.file_exists(path)[source]
Check if a file exists at the specified path.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
- Returns:
True if file exists, False otherwise
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> if file_exists("config.yaml"): ... print("Config file found") >>> file_exists("../../../etc/passwd")
- Security Changes:
Now validates paths to block path traversal
Blocks access to sensitive system files
- siege_utilities.files.operations.touch_file(path, create_parents=True)[source]
Create an empty file, creating parent directories if needed.
SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Raises:
PathSecurityError – If path fails security validation
OSError – If file creation fails
- Return type:
None
Example
>>> touch_file("logs/app.log") >>> touch_file(Path("data/output.txt"))
- siege_utilities.files.operations.count_lines(file_path, encoding='utf-8')[source]
Count the total number of lines in a text file.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
- Returns:
Number of lines
- Raises:
FileNotFoundError – If file does not exist or is not a file
PathSecurityError – If path fails security validation
OSError – If file cannot be read
UnicodeDecodeError – If file cannot be decoded with given encoding
- Return type:
Example
>>> line_count = count_lines("data.csv") >>> print(f"File has {line_count} lines")
- siege_utilities.files.operations.copy_file(source, destination, overwrite=False)[source]
Copy a file from source to destination.
SECURITY: Validates both source and destination paths.
- Parameters:
- Raises:
FileNotFoundError – If source does not exist or is not a file
FileExistsError – If destination exists and overwrite=False
PathSecurityError – If paths fail security validation
OSError – If copy fails
- Return type:
None
Example
>>> copy_file("source.txt", "backup/source.txt") >>> copy_file(Path("config.yaml"), Path("config.yaml.bak"))
- siege_utilities.files.operations.move_file(source, destination, overwrite=False)[source]
Move a file from source to destination.
SECURITY: Validates both source and destination paths.
- Parameters:
- Raises:
FileNotFoundError – If source does not exist or is not a file
FileExistsError – If destination exists and overwrite=False
PathSecurityError – If paths fail security validation
OSError – If move fails
- Return type:
None
Example
>>> move_file("temp.txt", "archive/temp.txt") >>> move_file(Path("old.log"), Path("logs/old.log"))
- siege_utilities.files.operations.get_file_size(file_path)[source]
Get the size of a file in bytes.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
- Returns:
File size in bytes
- Raises:
FileNotFoundError – If file does not exist or is not a file
PathSecurityError – If path fails security validation
OSError – If file size cannot be determined
- Return type:
Example
>>> size = get_file_size("large_file.zip") >>> print(f"File size: {size} bytes")
- siege_utilities.files.operations.list_directory(path, pattern='*', include_dirs=True, include_files=True)[source]
List contents of a directory with optional filtering.
SECURITY: Validates directory path to prevent traversal attacks.
- Parameters:
- Returns:
List of Path objects
- Raises:
FileNotFoundError – If directory does not exist or is not a directory
PathSecurityError – If path fails security validation
OSError – If directory cannot be read
- Return type:
Example
>>> files = list_directory("data", "*.csv") >>> dirs = list_directory("logs", include_files=False)
- siege_utilities.files.operations.run_command(command, cwd=None, timeout=30, capture_output=True, allow_list=None, unsafe=False)[source]
Run a shell command with security validation and proper error handling.
SECURITY: This function now validates commands by default. Only whitelisted commands are allowed unless unsafe=True is explicitly set.
- Parameters:
timeout (int | None) – Timeout in seconds (default: 30)
capture_output (bool) – Whether to capture command output
allow_list (set | None) – Optional custom whitelist of allowed commands
unsafe (bool) – If True, bypass the allow-list check. The command is still executed via argv (shell=False) — string commands are shlex-split and run without shell expansion. Use this when you legitimately need a binary that isn’t in the allow-list, not when you need shell features.
- Returns:
CompletedProcess object on success or non-zero exit.
- Raises:
SecurityError – If command fails security validation (when unsafe=False).
subprocess.TimeoutExpired – If the command exceeds timeout.
OSError – If the command binary cannot be found or executed.
- Return type:
Example
>>> result = run_command("ls -la") >>> if result and result.returncode == 0: ... print("Command succeeded") >>> result = run_command("rm -rf /") >>> result = run_command("git status", allow_list={'git', 'ls'}) >>> result = run_command("custom_cmd", unsafe=True)
- Security Changes:
Previously used shell=True with no validation (CRITICAL VULNERABILITY)
Now validates all commands against whitelist by default
Blocks command injection, path traversal, sensitive file access
Use unsafe=True ONLY with trusted input in trusted environments
- siege_utilities.files.operations.ensure_directory_exists(directory)[source]
Ensure a directory exists, creating it if necessary.
- Parameters:
- Returns:
Path object for the directory
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
- siege_utilities.files.operations.safe_file_write(file_path, content, encoding='utf-8', create_dirs=True)[source]
Safely write content to a file, optionally creating parent directories.
- Parameters:
- Returns:
True if successful, False otherwise
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
- siege_utilities.files.operations.safe_file_read(file_path, encoding='utf-8', default=None)[source]
Safely read content from a file.
- Parameters:
- Returns:
File content or default value
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
str | None
- siege_utilities.files.operations.safe_json_write(file_path, data, indent=2, create_dirs=True)[source]
Safely write data to a JSON file.
- Parameters:
- Returns:
True if successful, False otherwise
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
- siege_utilities.files.operations.safe_json_read(file_path, default=None)[source]
Safely read data from a JSON file.
- Parameters:
- Returns:
Parsed JSON data or default value
- Raises:
PathSecurityError – If path fails security validation.
- Return type:
Any | None
- siege_utilities.files.operations.get_file_size_mb(file_path)[source]
Get file size in megabytes.
- Parameters:
- Returns:
File size in MB.
- Raises:
FileNotFoundError – If the file does not exist.
PathSecurityError – If path fails security validation.
OSError – If the file size cannot be determined.
- Return type:
- siege_utilities.files.operations.list_files_recursive(directory, pattern='*', exclude_dirs=True)[source]
List all files in a directory recursively.
- Parameters:
- Returns:
List of Path objects
- Raises:
FileNotFoundError – If the directory does not exist.
PathSecurityError – If path fails security validation.
OSError – If the directory cannot be read.
- Return type:
- siege_utilities.files.operations.rmtree(path)
Remove a file or directory tree safely.
SECURITY: Validates paths to prevent removing sensitive system files.
- Parameters:
- Raises:
FileNotFoundError – If path does not exist
PathSecurityError – If path fails security validation
OSError – If removal fails
- Return type:
None
Example
>>> remove_tree("temp_directory") >>> remove_tree(Path("old_files"))
- siege_utilities.files.operations.check_if_file_exists_at_path(path)[source]
Check if a file exists at the specified path.
Deprecated since version 3.19.0: Use
file_exists()instead. Will be removed in v4.0.0.
- siege_utilities.files.operations.delete_existing_file_and_replace_it_with_an_empty_file(file_path, create_parents=True)[source]
Backward compatibility function that deletes existing file and replaces it with empty file.
Deprecated since version 3.19.0: Use
touch_file()for creating empty files. Note thattouch_filedoes not delete the existing file first; if you need delete-then-create semantics, callPath.unlink()beforetouch_file(). Will be removed in v4.0.0.SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Raises:
PathSecurityError – If path fails security validation
OSError – If file creation fails
- Return type:
None
- siege_utilities.files.operations.count_total_rows_in_file_pythonically(file_path, encoding='utf-8')
Count the total number of lines in a text file.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
- Returns:
Number of lines
- Raises:
FileNotFoundError – If file does not exist or is not a file
PathSecurityError – If path fails security validation
OSError – If file cannot be read
UnicodeDecodeError – If file cannot be decoded with given encoding
- Return type:
Example
>>> line_count = count_lines("data.csv") >>> print(f"File has {line_count} lines")
Hash Management Functions - Fixed Version Provides standardized hash functions that actually exist and work properly
- siege_utilities.files.hashing.calculate_file_hash(file_path)[source]
Alias for get_file_hash with SHA256 - for backward compatibility.
- Return type:
- siege_utilities.files.hashing.generate_sha256_hash_for_file(file_path)[source]
Generate SHA256 hash for a file - chunked reading for large files.
Deprecated since version 3.19.0: Use
calculate_file_hash()orget_file_hash()instead. Will be removed in v4.0.0.SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
file_path – Path to the file (str or Path object)
- Returns:
SHA256 hash as hexadecimal string
- Raises:
FileNotFoundError – If file does not exist or is not a file
PathSecurityError – If path fails security validation
OSError – If file cannot be read
- Return type:
Example
>>> hash_val = generate_sha256_hash_for_file("data.txt")
- siege_utilities.files.hashing.get_file_hash(file_path, algorithm='sha256')[source]
Generate hash for a file using specified algorithm.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
file_path – Path to the file (str or Path object)
algorithm – Hash algorithm to use (‘sha256’, ‘md5’, ‘sha1’, etc.)
- Returns:
Hash as hexadecimal string
- Raises:
FileNotFoundError – If file does not exist or is not a file
PathSecurityError – If path fails security validation
OSError – If file cannot be read
ValueError – If algorithm is not supported
- Return type:
Example
>>> hash_val = get_file_hash("data.txt", "sha256")
- siege_utilities.files.hashing.get_quick_file_signature(file_path)[source]
Generate a quick file signature using file stats + partial hash Faster for change detection, not cryptographically secure
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files.
- Parameters:
file_path – Path to the file
- Returns:
Quick signature string (‘missing’, fallback stat string, or hash)
- Raises:
PathSecurityError – If path fails security validation
Exception – If both primary and fallback signature generation fail (logged before re-raise)
- Return type:
Example
>>> sig = get_quick_file_signature("data.txt") >>> get_quick_file_signature("/etc/passwd")
- Security Changes:
Now validates paths to block path traversal
Blocks access to sensitive system files
- siege_utilities.files.hashing.verify_file_integrity(file_path, expected_hash, algorithm='sha256')[source]
Verify file integrity by comparing with expected hash.
SECURITY: This function validates paths to prevent path traversal attacks and access to sensitive system files (through get_file_hash).
- Parameters:
file_path – Path to the file
expected_hash – Expected hash value
algorithm – Hash algorithm used
- Returns:
True if file matches expected hash, False if hash does not match
- Raises:
FileNotFoundError – If file does not exist
PathSecurityError – If path fails security validation
OSError – If file cannot be read
- Return type:
Example
>>> expected = "abc123..." >>> if verify_file_integrity("data.txt", expected): ... print("File integrity verified")
Modern path utilities for siege_utilities. Provides clean, type-safe path manipulation utilities.
- siege_utilities.files.paths.ensure_path_exists(desired_path)[source]
Ensure a directory path exists, creating it if necessary.
SECURITY: Validates paths to prevent path traversal and sensitive file access.
- Parameters:
- Returns:
Path object for the created directory
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> path = ensure_path_exists("logs/2024/01") >>> print(f"Directory created: {path}") >>> ensure_path_exists("../../../etc")
- siege_utilities.files.paths.unzip_file_to_directory(zip_file_path, extract_to=None, create_subdirectory=True)[source]
Extract a zip file to a directory.
SECURITY: Validates paths to prevent zip slip attacks and path traversal.
- Parameters:
- Returns:
Path to the extraction directory
- Raises:
FileNotFoundError – If zip file does not exist or is not a file
zipfile.BadZipFile – If file is not a valid zip
PathSecurityError – If paths fail security validation
OSError – If extraction fails
- Return type:
Example
>>> extract_dir = unzip_file_to_directory("data.zip") >>> print(f"Files extracted to: {extract_dir}")
- siege_utilities.files.paths.get_file_extension(file_path)[source]
Get the file extension from a file path.
SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Returns:
File extension (including the dot)
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> ext = get_file_extension("document.pdf") >>> print(f"File extension: {ext}") >>> get_file_extension("../../../etc/passwd")
- siege_utilities.files.paths.get_file_name_without_extension(file_path)[source]
Get the file name without its extension.
SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Returns:
File name without extension
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> name = get_file_name_without_extension("document.pdf") >>> print(f"File name: {name}") >>> get_file_name_without_extension("../../../etc/shadow")
Check if a file or directory is hidden.
SECURITY: Validates paths to prevent path traversal attacks.
- Parameters:
- Returns:
True if the file/directory is hidden
- Raises:
PathSecurityError – If path fails security validation
- Return type:
Example
>>> if is_hidden_file(".config"): ... print("Hidden file detected") >>> is_hidden_file("../../.ssh/id_rsa")
- siege_utilities.files.paths.get_relative_path(base_path, target_path)[source]
Get the relative path from base_path to target_path.
SECURITY: Validates both base and target paths to prevent path traversal attacks.
- Parameters:
- Returns:
Relative path from base to target
- Raises:
ValueError – If target is not relative to base
PathSecurityError – If paths fail security validation
- Return type:
Example
>>> rel_path = get_relative_path("/home/user", "/home/user/documents/file.txt") >>> print(f"Relative path: {rel_path}")
- siege_utilities.files.paths.find_files_by_pattern(directory, pattern='*', recursive=False)[source]
Find files matching a pattern in a directory.
SECURITY: Validates directory path to prevent path traversal attacks.
- Parameters:
- Returns:
List of matching file paths
- Raises:
PathSecurityError – If path fails security validation
OSError – If the directory cannot be read
- Return type:
Example
>>> files = find_files_by_pattern("data", "*.csv", recursive=True) >>> print(f"Found {len(files)} CSV files") >>> find_files_by_pattern("../../../etc", "*")
- siege_utilities.files.paths.create_backup_path(original_path, backup_suffix='.backup', backup_dir=None)[source]
Create a backup path for a file.
SECURITY: Validates both original and backup paths to prevent path traversal attacks.
- Parameters:
- Returns:
Path for the backup file
- Raises:
PathSecurityError – If paths fail security validation
- Return type:
Example
>>> backup_path = create_backup_path("config.yaml", ".bak") >>> print(f"Backup will be saved to: {backup_path}") >>> create_backup_path("/etc/passwd", ".bak")
- siege_utilities.files.paths.normalize_path(path)[source]
Normalize a file path, resolving any relative components.
This function expands ~ (home directory), resolves .. (parent directory), and converts to an absolute path. It does NOT validate for security - use validate_safe_path() if you need security validation.
- Parameters:
- Returns:
Normalized absolute path
- Return type:
Example
>>> norm_path = normalize_path("~/../data/file.txt") >>> print(f"Normalized path: {norm_path}") >>> norm_path = normalize_path("./../data/file.txt")
- siege_utilities.files.paths.unzip_file_to_its_own_directory(zip_file_path, extract_to=None, create_subdirectory=True)
Extract a zip file to a directory.
SECURITY: Validates paths to prevent zip slip attacks and path traversal.
- Parameters:
- Returns:
Path to the extraction directory
- Raises:
FileNotFoundError – If zip file does not exist or is not a file
zipfile.BadZipFile – If file is not a valid zip
PathSecurityError – If paths fail security validation
OSError – If extraction fails
- Return type:
Example
>>> extract_dir = unzip_file_to_directory("data.zip") >>> print(f"Files extracted to: {extract_dir}")
Modern remote file operations for siege_utilities. Provides clean, type-safe file download utilities.
- siege_utilities.files.remote.download_file(url, local_filename, chunk_size=8192, timeout=30, verify_ssl=True)[source]
Download a file from a URL to a local file with progress bar.
SECURITY: Validates local file path to prevent path traversal attacks.
- Parameters:
- Returns:
The local filename as a string
- Raises:
PathSecurityError – If local path fails security validation
ConnectionError – If HTTP request fails (non-2xx status)
requests.exceptions.Timeout – If request times out
requests.exceptions.RequestException – If request fails
OSError – If file write fails
- Return type:
Example
>>> result = download_file("https://example.com/file.zip", "downloads/file.zip") >>> print(f"Downloaded to {result}")
- siege_utilities.files.remote.generate_local_path_from_url(url, directory_path, as_string=True)[source]
Generate a local file path from a URL.
SECURITY: Validates directory path to prevent path traversal attacks.
- Parameters:
- Returns:
Path object or string
- Raises:
ValueError – If filename cannot be extracted from URL
PathSecurityError – If directory path fails security validation
OSError – If directory creation fails
- Return type:
Example
>>> path = generate_local_path_from_url("https://example.com/file.zip", "downloads") >>> print(f"Local path: {path}")
- siege_utilities.files.remote.download_file_with_retry(url, local_filename, max_retries=3, retry_delay=5, **kwargs)[source]
Download a file with automatic retry on failure.
- Parameters:
- Returns:
The local filename as a string
- Raises:
ConnectionError – If all retry attempts fail
requests.exceptions.RequestException – If all retry attempts fail
- Return type:
Example
>>> result = download_file_with_retry("https://example.com/file.zip", "file.zip") >>> print(f"Downloaded to {result}")
- siege_utilities.files.remote.get_file_info(url, timeout=10)[source]
Get information about a remote file without downloading it.
- Parameters:
- Returns:
Dictionary with file information
- Raises:
ConnectionError – If HTTP request fails (non-2xx status)
requests.exceptions.RequestException – If request fails
ImportError – If requests library is not available
- Return type:
Example
>>> info = get_file_info("https://example.com/file.zip") >>> print(f"File size: {info['size']} bytes")
- siege_utilities.files.remote.is_downloadable(url, timeout=10)[source]
Check if a URL points to a downloadable file.
- Parameters:
- Returns:
True if the URL points to a downloadable file, False otherwise
- Return type:
Example
>>> if is_downloadable("https://example.com/file.zip"): ... print("URL is downloadable")
Secure shell command execution utilities.
SECURITY NOTE: This module previously contained critical vulnerabilities. All shell execution now includes proper validation and sanitization.
- siege_utilities.files.shell.run_subprocess(command_list, allow_list=None, timeout=30)[source]
Run a shell command as a subprocess with security validation.
SECURITY: This function now validates all commands against a whitelist and blocks dangerous operations. Only read-only commands are allowed by default.
- Parameters:
- Returns:
The command output (stdout if successful, stderr if failed)
- Raises:
SecurityError – If command fails security validation
ValueError – If command is invalid
subprocess.TimeoutExpired – If command times out
- Return type:
Example
>>> output = run_subprocess("ls -la") >>> output = run_subprocess("rm -rf /")
- Security Changes:
Previously used shell=True with no validation (CRITICAL VULNERABILITY)
Now validates all commands against whitelist
Blocks command injection, path traversal, sensitive file access
Uses shell=False by default for safety
- siege_utilities.files.shell.validate_command_safety(command, allow_list=None)[source]
Validate that a command is safe to execute.
- Parameters:
- Returns:
Validated command as list of strings
- Raises:
SecurityError – If command contains forbidden elements
ValueError – If command is invalid
- Return type:
Security checks: - Validates command against whitelist - Blocks command injection characters (;, &, |, $, `, newlines) - Blocks path traversal attempts - Blocks access to sensitive system files
- exception siege_utilities.files.shell.SecurityError[source]
Bases:
ExceptionRaised when a security violation is detected.
Unified output format support for spatial and tabular data.
Provides save_spatial() and save_tabular() dispatchers that write
data to disk in the requested format. Enum types SpatialFormat and
TabularFormat enumerate supported file types.
Usage:
from siege_utilities.files.formats import save_spatial, SpatialFormat
save_spatial(gdf, "/tmp/districts.parquet", SpatialFormat.GEOPARQUET)
- class siege_utilities.files.formats.SpatialFormat[source]
-
Supported spatial output formats.
- GEOPARQUET = 'geoparquet'
- PARQUET = 'parquet'
- GPKG = 'gpkg'
- GEOJSON = 'geojson'
- TOPOJSON = 'topojson'
- CSV = 'csv'
- SHAPEFILE = 'shp'
- __new__(value)
- class siege_utilities.files.formats.TabularFormat[source]
-
Supported tabular output formats.
- PARQUET = 'parquet'
- CSV = 'csv'
- EXCEL = 'excel'
- JSON = 'json'
- __new__(value)
- siege_utilities.files.formats.save_spatial(gdf, path, fmt=SpatialFormat.GEOPARQUET, **kwargs)[source]
Write a GeoDataFrame to path in the requested format.
- Parameters:
gdf (geopandas.GeoDataFrame) – Data to write.
fmt (SpatialFormat) – Output format (default
GEOPARQUET).**kwargs – Passed through to the underlying writer.
- Return type:
Path to the written file.
- siege_utilities.files.formats.save_tabular(df, path, fmt=TabularFormat.PARQUET, **kwargs)[source]
Write a DataFrame to path in the requested format.
- Parameters:
df (pandas.DataFrame) – Data to write.
fmt (TabularFormat) – Output format (default
PARQUET).**kwargs – Passed through to the underlying writer.
- Return type:
Path to the written file.
Path validation and sanitization utilities for siege_utilities.
This module provides security-focused path validation to prevent: - Path traversal attacks (../, ~/../, etc.) - Access to sensitive system files and directories - Symlink-based attacks - Null byte injection
Use these validators in all file operations to ensure secure path handling.
- exception siege_utilities.files.validation.PathSecurityError[source]
Bases:
ExceptionRaised when a path fails security validation.
- siege_utilities.files.validation.is_path_traversal_attempt(path)[source]
Check if a path contains traversal patterns.
- Parameters:
- Returns:
True if path contains traversal patterns
- Return type:
Example
>>> is_path_traversal_attempt("../../../etc/passwd") True >>> is_path_traversal_attempt("data/file.txt") False
- siege_utilities.files.validation.is_sensitive_path(path)[source]
Check if a path points to or contains a sensitive system location.
Example
>>> is_sensitive_path("/etc/passwd") True >>> is_sensitive_path("/tmp/data.txt") False
- siege_utilities.files.validation.validate_safe_path(path, allow_absolute=True, base_directory=None)[source]
Validate that a path is safe to access.
Performs comprehensive security checks: 1. Blocks path traversal attempts 2. Blocks access to sensitive system paths 3. Optionally restricts to a base directory 4. Blocks null byte injection 5. Resolves symlinks and checks final destination
- Parameters:
- Returns:
Validated Path object
- Raises:
PathSecurityError – If path fails security validation
ValueError – If path is invalid
- Return type:
Example
>>> # Safe path >>> validated = validate_safe_path("data/file.txt") >>> >>> try: ... validate_safe_path("../../../etc/passwd") ... except PathSecurityError as e: ... pass # raises PathSecurityError >>> >>> validated = validate_safe_path("subdir/file.txt", base_directory="/safe/dir")
- siege_utilities.files.validation.validate_file_path(file_path, must_exist=False, base_directory=None)[source]
Validate a file path for read/write operations.
- Parameters:
- Returns:
Validated Path object
- Raises:
PathSecurityError – If path fails security validation
FileNotFoundError – If must_exist=True and file doesn’t exist
ValueError – If path is invalid
- Return type:
Example
>>> path = validate_file_path("data.csv", must_exist=True) >>> >>> path = validate_file_path("output.txt", must_exist=False)
- siege_utilities.files.validation.validate_directory_path(dir_path, must_exist=False, base_directory=None)[source]
Validate a directory path.
- Parameters:
- Returns:
Validated Path object
- Raises:
PathSecurityError – If path fails security validation
FileNotFoundError – If must_exist=True and directory doesn’t exist
ValueError – If path is invalid or not a directory
- Return type:
Example
>>> # Validate directory >>> path = validate_directory_path("logs", must_exist=True)
- siege_utilities.files.validation.safe_join_paths(*paths, base_directory=None)[source]
Safely join multiple path components with security validation.
- Parameters:
- Returns:
Validated joined Path object
- Raises:
PathSecurityError – If any component or result fails security validation
ValueError – If components are invalid
- Return type:
Example
>>> # Safe join >>> path = safe_join_paths("data", "2024", "file.txt") >>> >>> path = safe_join_paths("subdir", "file.txt", base_directory="/safe/dir")