ryan-tools

ryan-tools is a collection of Python utilities for TUFLOW, RORB, 12D, GDAL, QGIS and general data-processing workflows. Reusable code lives in ryan_library; files under ryan-scripts are human-facing wrappers and standalone utilities.

The distribution is named ryan_functions, while maintained imports normally use ryan_library.

Project status

The repository targets Python 3.14. Maintained library code, orchestrators and unversioned wrappers follow the current repository standards; older standalone, versioned and compatibility files remain where they still support migration or narrow workflows.

Start with:

• Development guide for architecture, code categories, lifecycle terms and validation.
• Environment guide for Python, VS Code, installed-wheel and QGIS/OSGeo4W setup.
• Ryan Scripts guide for choosing and safely running scripts.
• Maintained wrapper standard when changing a library-backed wrapper.
• Project documentation index for the complete topic map, including specialised guides kept beside scripts, processors, examples and maintenance tools.
• Examples for direct library use when an existing wrapper is not the right fit.

Repository map

ryan-tools/
|-- ryan_library/              # Maintained Python package
|   |-- classes/               # Configuration, metadata and filename parsing
|   |-- functions/             # Reusable algorithms and focused I/O helpers
|   |-- orchestrators/         # Complete workflow controllers
|   |-- processors/            # Stateful processors for supported result formats
|   `-- scripts/               # Deprecated import-compatibility wrappers
|-- ryan-scripts/              # Human-facing wrappers and standalone utilities
|-- docs/                      # Development guidance, plans and setup documentation
|-- examples/                  # Example notebooks and supporting demonstrations
|-- repo-scripts/              # Build, environment and repository maintenance tools
|-- tests/                     # Unit, integration and regression tests
|   `-- test_data/             # Required synthetic test-data submodule
|-- vendor/
|   |-- run_hy8/               # HY-8 submodule
|   `-- ryan_culverts/         # Culvert hydraulics submodule
|-- excel-resources/           # Excel workbook resources submodule
|-- qgis-resources/            # QGIS resources submodule
|-- unsorted/                  # Separate holding-area submodule
|-- pyproject.toml             # Package metadata and tool configuration
`-- setup.py                   # Setuptools hook that stages QGIS styles into wheels

Set up the repository

This repository uses Git submodules and Git LFS for workbook resources. Install Git LFS, then clone recursively:

git lfs install
git clone --recurse-submodules https://github.com/Chain-Frost/ryan-tools.git
cd ryan-tools
git lfs pull
git -C qgis-resources lfs pull
git -C excel-resources lfs pull

For an existing clone:

git submodule update --init --recursive
git lfs pull
git -C qgis-resources lfs pull
git -C excel-resources lfs pull

The test suite requires tests/test_data; it does not download or substitute those fixtures automatically. Because run_hy8 and culvert_solver are not PyPI dependencies, their vendor/run_hy8 and vendor/ryan_culverts submodules are bundled in ryan_functions wheels and installed automatically as normal top-level packages.

Install the repository requirements into the user's normal Python 3.14 installation. ryan-tools does not require or assume that users know how to create or activate a virtual environment:

py -3.14 -m pip install --upgrade pip
py -3.14 repo-scripts\install_latest_wheel.py --dependencies-only
py -3.14 -m pip install -r requirements.txt

requirements.txt installs the checkout and development tools into that Python installation. Installing the project is important when running wrappers copied outside the repository because they import the shared implementation from the installed ryan_functions distribution. The dependency bootstrap installs binary Fiona, Rasterio and GDAL packages from the configured geospatial wheel index, avoiding a local source build on Windows.

Build and install

After changing ryan_library or package metadata, rebuild from the repository root:

python repo-scripts/build_library.py

By default, the build script advances the normalized calendar version in pyproject.toml. Use --no-bump to rebuild and verify the current version without changing it, --skip-pip when the build dependency is already installed, or --skip-artifacts in an environment that cannot create or retain wheel artifacts. The wheel is built in temporary storage, checked against the project metadata and bundled resource sources, and only then promoted into dist/; a failed build or verification leaves the previous wheel and package version intact. Wheel builds require the QGIS resource submodule because setup.py stages the pinned TUFLOW QML styles into the package.

To re-check the retained wheel without rebuilding it:

python repo-scripts/verify_wheel.py

Windows convenience entry points are:

.\package_and_install.bat
.\install-latest-wheel.bat

The first builds and installs the package; the second installs the newest existing wheel.

License

This project is licensed under the Sustainable Use License v1.0 (SUL-1.0).

The software may be used and modified for personal, non-commercial and internal business purposes, including commercial professional and consulting work where the software itself is not provided as a commercial product or service. Commercial distribution, incorporation into software supplied commercially to third parties, or provision of the software or its functionality as a paid hosted service, SaaS or API is not permitted under the licence.

Test changes

Use focused pytest commands for a bounded change. Keep temporary files under the repository on this Windows checkout:

python -m pytest tests\path\to\test_file.py --basetemp=.pytest_cache\basetemp

Run the complete suite through the repository runner:

cmd.exe /C repo-scripts\run_tests.bat

The runner configures the source and bundled HY-8 import paths, uses a repository-local base temporary directory and generates terminal, HTML and XML coverage reports. See the development guide for proportional validation expectations.

Choose a workflow

Need	Start here
Run or adapt a human-facing script	ryan-scripts/README.md
Process TUFLOW results with maintained wrappers	ryan-scripts/TUFLOW-python/README.md
Extend the TUFLOW processor framework	ryan_library/processors/tuflow/README.md
Use reusable Python APIs directly	examples/README.md
Run maintained GDAL workflows	ryan-scripts/gdal-python/README.md
Convert geospatial, point-cloud, CAD or model files	Supported file converters and formats
Map a remaining legacy GDAL BAT file to its replacement	ryan-scripts/gdal-bat/README.md
Connect the repository MCP server	docs/MCP_SETUP.md

Parse a TUFLOW result filename

TuflowStringParser extracts the configured data type and run metadata from a result path:

from pathlib import Path

from ryan_library.classes.tuflow_string_classes import TuflowStringParser

parser = TuflowStringParser(Path("M11_01p_00120m_TP01_1d_Q.csv"))

print(parser.data_type)
print(parser.raw_run_code)
print(parser.clean_run_code)
print(parser.aep)
print(parser.duration)
print(parser.tp)

The authoritative suffix registry is ryan_library/classes/tuflow_results_validation_and_datatypes.json. It contains both processor-backed tabular types and raster classifications used for discovery.

Load TUFLOW data for exploration

load_tuflow_data() handles discovery, logging, serial or parallel processing and optional location filtering:

from ryan_library.functions.tuflow.notebook_helpers import load_tuflow_data

collection = load_tuflow_data(
    paths=["results"],
    data_types=["Q", "V", "H", "Nmx", "Cmx", "Chan", "EOF"],
    parallel=True,
    locations=["Culvert_01"],
)

timeseries = collection.combine_1d_timeseries()
maximums = collection.combine_1d_maximums()

Processor-backed types currently include POMM, PO, Cmx, Nmx, Chan, ccA, RLL_Qmx, Q, H, CF, V, EOF and TLF. Use the processor development notes for combination behavior, caching and extension guidance.

Run a ready-made TUFLOW workflow

Orchestrators are callable workflow controllers; wrappers add editable defaults, CLI handling, banners, exit codes and optional pauses:

from pathlib import Path

from ryan_library.orchestrators.tuflow.tuflow_culverts_merge import main_processing

main_processing(
    paths_to_process=[Path("results")],
    include_data_types=["Nmx", "Cmx", "Chan", "ccA", "RLL_Qmx", "EOF"],
    locations_to_include=["Culvert_01"],
    output_dir=Path("outputs"),
    export_mode="both",
)

Representative orchestrators cover culvert maximums and time series, PO/POMM combination, closure durations, peak and stability checks, TUFLOW log summaries and result styling. The corresponding maintained wrappers are under ryan-scripts/TUFLOW-python; use that folder's README and each wrapper's --help output as the current interface.

Use other library helpers

RORB hydrograph discovery and parsing:

from pathlib import Path

from ryan_library.functions.RORB.read_rorb_files import find_batch_files, parse_batch_output

batch_files = find_batch_files([Path("rorb_outputs")])
runs = [parse_batch_output(path) for path in batch_files]

12D culvert export processing:

from pathlib import Path

from ryan_library.functions.process_12D_culverts import get_combined_df_from_files

culverts = get_combined_df_from_files(Path("12d_exports"))

For GDAL raster conversion, mosaics, flood extents, footprints, metadata and point-cloud conversion, use the maintained GDAL Python wrappers. The BAT files that remain under ryan-scripts/gdal-bat are legacy migration references and should not be selected for new work.

Supported file converters and formats

The library provides Python-native and GDAL-backed conversion functions, batch orchestrators, and maintained wrappers across geospatial, point-cloud, CAD, hydrologic, and tabular formats:

Domain	Source formats	Destination formats	Key library APIs and entry points	Notes
Raster translation & compression	Any GDAL raster (.flt, .asc, .rst, .xyz, .tif, .ecw, etc.)	.tif (GeoTIFF), .ovr (pyramid overviews)	translate_to_geotiff(), convert_rasters(), gdal_translate_TIF_ovr.py	Lossless conversion with choice of tuflow (DEFLATE) or efficient (tiled ZSTD) profiles.
Raster mosaics & virtual datasets	Multiple raster tiles (.tif, .xyz, .asc, .flt)	.tif (GeoTIFF mosaic), .vrt (VRT dataset)	merge_directory(), create_grouped_mosaics(), gdal_merge.py, build_VRT.py	Supports vector extent clipping, assigned CRS/NoData, and grouped TUFLOW result sets.
Vector dataset translation	.gpkg, .shp, .fgb, .geojson, .sqlite	.gpkg (GeoPackage), .shp (ESRI Shapefile), .fgb (FlatGeobuf), .geojson (GeoJSON), .sqlite (SQLite)	translate_vector_dataset(), require_vector_driver()	Atomic vector translation via GDAL; multi-layer support for GPKG and SQLite.
File Geodatabase export	.gdb (ESRI File Geodatabase directories)	.gpkg, .shp, .fgb, .geojson, .sqlite	export_file_geodatabase(), discover_file_geodatabases()	Batch GDB discovery; exports each layer to separate files or one multi-layer database.
Raster to vector classification	Depth, water level, or DEM rasters (.tif, .ecw, etc.)	.gpkg, .shp, .geojson, .fgb, .sqlite (flood extent polygons, raster footprints)	main_processing(), polygonize_flood_extent(), gdal_flood_extent.py, gdal_raster_footprint.py	Thresholding, band selection, optional GDAL sieve filtering, and boundary vectorization.
LiDAR & point clouds	.laz (compressed LiDAR)	.las (uncompressed ASPRS LAS point clouds)	convert_laz_to_las(), convert_laz_directory(), laz_to_las.py	Memory-efficient chunked streaming preserving headers, point formats, scales, offsets, and VLRs.
Elevation rasters to tables/points	.tif (GeoTIFF DEMs)	Tabular X, Y, Z DataFrames, .csv, .xyz, .las	read_geotiff(), tile_data(), thin-raster-terrain-for-12D_v5.py, tif-to-LAS-valid-only_v6.py	Spatial tiling, NoData masking, and point thinning for 12D Model or tabular workflows.
CAD, mining & 12D geometry	.str (Surpac binary/text strings), .dtm (Surpac binary/text DTM meshes), .dxf (polyface meshes), 12D culvert export text	.gpkg (3D lines & polygon meshes), .parquet (geometry tables), DataFrames	get_combined_df_from_files(), dtm_str_converter_to_gpkg.py, extract_dxf_polyface.py, polyface_parquet_to_gpkg.py	Translates mining CAD and 12D export formats into modern GIS/Parquet geometries.
Hydrologic & model results	TUFLOW 1D/2D CSVs (_POMM, _PO, _Cmx, _Nmx, _Chan, _ccA, _RLL_Qmx, _Q, _H, etc.), .tlf logs, RORB .out files	.xlsx (multi-sheet styled workbooks), .parquet, .csv, DataFrames	load_tuflow_data(), parse_batch_output(), ExcelExporter, save_to_excel()	Ingests simulation outputs into consolidated tabular datasets, summary reports, and Excel exports.

Excel and QGIS resources

The resource submodules contain project templates and application assets:

• excel-resources/workbooks/: Excel templates for hydrology, frequency and culvert workflows.
• qgis-resources/processing-models/tuflow/: QGIS models for common TUFLOW input layers.
• qgis-resources/processing-models/tuflow/supporting-workbooks/: formula-driven supporting workbooks used by relevant models.
• qgis-resources/styles/: QML styles, QPT layouts and supporting spatial assets.
• qgis-resources/scripts/: QGIS Python console and PyQGIS utilities.

Except for the TUFLOW QML files staged into wheel builds, treat these resources as project templates rather than files installed with the Python package. The parent repository pins each submodule to a reviewed commit.

Repository automation

• Pull request template records summaries, validation and review checklists.
• Code-review instructions describe repository-specific review expectations.
• Development guide is the canonical architecture and validation reference.
• MCP setup documents focused inspection tools and staged CLI workflow discovery for AI clients.