Skip to content

Cached benchmark file I/O - #1700

Draft
cmdupuis3 wants to merge 7 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/bench_io
Draft

Cached benchmark file I/O#1700
cmdupuis3 wants to merge 7 commits into
UXARRAY:mainfrom
cmdupuis3:cmd/bench_io

Conversation

@cmdupuis3

@cmdupuis3 cmdupuis3 commented Aug 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Closes #1698

Overview

This PR is a large-scale overhaul of the benchmarks to support single-loads of files per benchmark suite.

Currently, the ASV benchmark suite runs a new process per benchmark, requiring full-scale loads of files for every single benchmark. There are a limited number of files that are actually used to benchmark, so we end up loading many files dozens of times, running up wallclock time on the benchmark suite as a whole.

Larger benchmark suites will benefit more from cached file I/O, while individual benchmark runs won't see much improvement.

Simultaneously, this reorganization of file I/O will help to expand the scope of what most non-I/O benchmarks are able to test, since some of them were artificially restricted to smaller grids.

General CI runs should see a wallclock improvement for whole benchmark suites including CI. CI benchmark runs are showing improvements of about 19 minutes so far (38min -> 19min). Runs on HPC can expect larger improvements.

Other general speed improvements:

  1. Test process forking from warm-cache process
  2. Async dispatch of tests on file arrival, so large files don't stall tests of smaller files.
  3. Sample generation on parent process for reuse in forked processes; sacrifices a bit of robustness for significant speed gains in geometry routines.

PR Checklist

General

  • An issue is created and linked
  • Added appropriate labels (if your uxarray repo permissions allow it)
  • Filled out Overview and Expected Usage (if applicable) sections

Testing & Benchmarking

  • Adequate tests are created if there is new functionality
  • Tests are not too basic (such as simply calling a function and nothing else)
  • Tests cover all major paths in your new functions
  • If this PR could affect performance, ran ASV benchmarks and confirmed they show expected behavior (add a new benchmark if necessary)

Documentation

  • Docstrings have been added to all new functions
  • Docstrings have been updated with any function changes
  • Internal (private) function names start with an underscore (_)

AI Disclosure

AI Usage: Claude Opus 5

  • I take responsibility for all AI-generated content in my PR.
  • I have tested all AI-generated content in my PR.

@cmdupuis3 cmdupuis3 added scalability Related to scalability & performance efforts benchmarking Related to benchmarks, memory usage, and/or time profiling labels Aug 21, 2026
@cmdupuis3 cmdupuis3 self-assigned this Aug 21, 2026
@cmdupuis3 cmdupuis3 added the run-benchmark Run ASV benchmark workflow label Aug 21, 2026
@github-actions

github-actions Bot commented Aug 21, 2026 •
edited
Loading

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have stayed the same:

Change Before [9d5bcdb] After [71604e1] Ratio Benchmark (Parameter)
232±0.5ms 243±6ms 1.05 bench_connectivity.Connectivity.time_edge_face('120km')
13.1±0.06ms 13.7±0.4ms 1.04 bench_connectivity.Connectivity.time_edge_face('480km')
231±0.1ms 239±8ms 1.03 bench_connectivity.Connectivity.time_edge_node('120km')
12.3±0.03ms 12.8±0.4ms 1.03 bench_connectivity.Connectivity.time_edge_node('480km')
232±0.9ms 239±7ms 1.03 bench_connectivity.Connectivity.time_face_edge('120km')
12.7±0.03ms 12.9±0.4ms 1.02 bench_connectivity.Connectivity.time_face_edge('480km')
833±6ms 831±5ms 1.00 bench_connectivity.Connectivity.time_face_face('120km')
50.1±0.3ms 50.3±0.3ms 1.00 bench_connectivity.Connectivity.time_face_face('480km')
45.0±0.4μs 45.3±0.3μs 1.01 bench_connectivity.Connectivity.time_face_node('120km')
44.6±0.5μs 45.8±0.5μs 1.03 bench_connectivity.Connectivity.time_face_node('480km')
342±9μs 346±6μs 1.01 bench_connectivity.Connectivity.time_n_nodes_per_face('120km')
264±3μs 262±2μs 0.99 bench_connectivity.Connectivity.time_n_nodes_per_face('480km')
232±0.7ms 241±7ms 1.04 bench_connectivity.Connectivity.time_node_edge('120km')
12.7±0.03ms 13.1±0.4ms 1.03 bench_connectivity.Connectivity.time_node_edge('480km')
78.1±0.7ms 75.4±1ms 0.97 bench_connectivity.Connectivity.time_node_face('120km')
4.80±0.07ms 4.78±0.04ms 1.00 bench_connectivity.Connectivity.time_node_face('480km')
7.98±0.03ms 7.90±0.05ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
2.46±0.03ms 2.48±0.02ms 1.01 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
10.3±10s 12.0±10ms ~0.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
1.43±0.03ms 1.42±0.02ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
57.3k 57.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
12.3k 12.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
123k 123k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
128 128 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.27M 1.27M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
50.1k 50.1k 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
1.48M 1.48M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
712 712 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.97M 1.97M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
1.96M 1.96M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
2.12M 2.12M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
35.5k 35.5k 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
335M 334M 1.00 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
366M 364M 0.99 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
336M 336M 1.00 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
335M 336M 1.00 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.11±0.02μs 1.10±0.03μs 0.99 geometry_kernels.AccucrossKernels.time_accucross
2.51±0.04μs 2.51±0.04μs 1.00 geometry_kernels.AccucrossKernels.time_accucross_pair
380±100ns 381±5ns 1.00 geometry_kernels.EFTPrimitives.time_acc_sqrt_re
396±10ns 391±10ns 0.99 geometry_kernels.EFTPrimitives.time_diff_of_products
361±20ns 370±20ns 1.02 geometry_kernels.EFTPrimitives.time_two_prod
346±9ns 376±30ns 1.09 geometry_kernels.EFTPrimitives.time_two_sum
1.38±0.02μs 1.42±0.05μs 1.03 geometry_kernels.GCAConstLatIntersection.time_accux_constlat_kernel
1.03±0.05μs 1.05±0.04μs 1.02 geometry_kernels.GCAConstLatIntersection.time_gca_const_lat_intersection
1.79±0.02μs 1.84±0.03μs 1.02 geometry_kernels.GCAConstLatIntersection.time_try_gca_const_lat_intersection
1.59±0.04μs 1.56±0.04μs 0.98 geometry_kernels.GCAGCAIntersection.time_accux_gca_kernel
1.28±0.01μs 1.31±0.05μs 1.02 geometry_kernels.GCAGCAIntersection.time_gca_gca_intersection
2.01±0.02μs 2.03±0.02μs 1.01 geometry_kernels.GCAGCAIntersection.time_try_gca_gca_intersection
38.0±0.3μs 37.4±0.3μs 0.98 geometry_kernels.OrientPredicates.time_on_minor_arc
37.9±0.2μs 38.2±0.5μs 1.01 geometry_kernels.OrientPredicates.time_orient3d_on_sphere
3.01±0.01ms 3.03±0.01ms 1.01 geometry_samebody.SameBodyConstLat.time_accux_dispatch
1.32±0.01ms 1.32±0ms 1.00 geometry_samebody.SameBodyConstLat.time_accux_kernel
1.78±0.01ms 1.78±0.01ms 1.00 geometry_samebody.SameBodyConstLat.time_fp64_dispatch
149±0.3μs 149±0.3μs 1.00 geometry_samebody.SameBodyConstLat.time_fp64_kernel
32.6±0.06ms 32.9±0.06ms 1.01 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_dispatch
10.7±0.06ms 10.6±0.09ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_kernel
27.0±0.04ms 26.9±0.07ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_dispatch
4.44±0.08ms 4.33±0ms 0.98 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_kernel
786±6ms 790±10ms 1.01 import.Imports.timeraw_import_uxarray
300M 293M 0.98 import.Imports.track_peakmem_import_uxarray
2.07±0.05ms 2.05±0.01ms 0.99 mpas_ocean.CheckNorm.time_check_norm('120km')
1.73±0.02ms 1.62±0.01ms 0.94 mpas_ocean.CheckNorm.time_check_norm('480km')
727±3ms 713±1ms 0.98 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
45.7±0.4ms 44.4±0.5ms 0.97 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
578±10μs 580±5μs 1.00 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
514±7μs 527±6μs 1.02 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
4.99±0.02ms 4.95±0.02ms 0.99 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
3.48±0.06ms 3.43±0.01ms 0.99 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
105±0.3ms 105±0.5ms 1.00 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
10.5±0.09ms 9.75±0.3ms 0.93 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
19.9±0.03ms 20.0±0.04ms 1.00 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
1.01±0.01ms 1.02±0.01ms 1.01 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
11.0±0.09ms 11.0±0.05ms 1.00 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
641±10μs 641±10μs 1.00 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
529±8ms 528±7ms 1.00 mpas_ocean.CrossSections.time_const_lat('120km', 1)
273±4ms 268±2ms 0.98 mpas_ocean.CrossSections.time_const_lat('120km', 2)
139±0.5ms 139±3ms 1.00 mpas_ocean.CrossSections.time_const_lat('120km', 4)
479±2ms 475±9ms 0.99 mpas_ocean.CrossSections.time_const_lat('480km', 1)
242±3ms 245±2ms 1.01 mpas_ocean.CrossSections.time_const_lat('480km', 2)
124±0.4ms 123±1ms 0.99 mpas_ocean.CrossSections.time_const_lat('480km', 4)
354M 355M 1.00 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 1)
354M 354M 1.00 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 2)
354M 354M 1.00 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 4)
338M 337M 1.00 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 1)
338M 337M 1.00 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 2)
338M 337M 1.00 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 4)
23.2±0.08ms 22.1±0.3ms 0.95 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
2.47±0.04ms 2.65±1ms 1.08 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
59.1±0.5ms 60.2±2ms 1.02 mpas_ocean.FaceAreas.time_face_areas('120km')
4.94±5s 6.95±6ms ~0.00 mpas_ocean.FaceAreas.time_face_areas('480km')
229k 229k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('120km')
14.3k 14.3k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('480km')
2.13M 2.13M 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('120km')
823k 823k 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('480km')
848±9ms 839±1ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', False)
54.2±0.6ms 53.0±0.5ms 0.98 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', True)
72.0±0.3ms 72.7±0.6ms 1.01 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', False)
4.75±0.3ms 4.81±0.1ms 1.01 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', True)
165±0.7ms 165±0.7ms 1.00 mpas_ocean.Gradient.time_gradient('120km')
11.1±0.03ms 11.1±0.03ms 1.00 mpas_ocean.Gradient.time_gradient('480km')
457k 457k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('120km')
28.7k 28.7k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('480km')
4.86M 4.86M 1.00 mpas_ocean.Gradient.track_peakmem_gradient('120km')
313k 313k 1.00 mpas_ocean.Gradient.track_peakmem_gradient('480km')
350M 350M 1.00 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('120km')
334M 330M 0.99 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('480km')
267±5μs 259±7μs 0.97 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
132±2μs 127±1μs 0.97 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
190±2μs 190±2μs 1.00 mpas_ocean.Integrate.time_integrate('120km')
172±1μs 175±10μs 1.02 mpas_ocean.Integrate.time_integrate('480km')
18.4M 18.4M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('120km')
1.2M 1.2M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('480km')
188±2ms 186±3ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
187±1ms 187±4ms 1.00 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
189±1ms 193±6ms 1.02 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
13.2±0.07ms 13.5±0.4ms 1.02 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
13.0±0.1ms 13.3±0.3ms 1.02 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
13.0±0.1ms 13.4±0.3ms 1.03 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
329±7μs 364±20μs ~1.11 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
332±10μs 329±6μs 0.99 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
312±3μs 329±30μs 1.05 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
311±10μs 305±5μs 0.98 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
123±0.4ms 126±2ms 1.02 mpas_ocean.RemapDownsample.time_bilinear_remapping
17.1±0.3ms 17.1±0.2ms 1.00 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
15.5±0.2ms 15.4±0.2ms 1.00 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping
1.10±0.01s 1.10±0.01s 1.01 mpas_ocean.RemapUpsample.time_bilinear_remapping
25.9±0.1ms 25.5±0.04ms 0.99 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
11.2±0.4ms 10.6±0.1ms 0.95 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
7.98±0.1ms 8.04±0.06ms 1.01 mpas_ocean.ZonalAverage.time_zonal_average('120km')
3.72±0.07ms 3.60±0.05ms 0.97 mpas_ocean.ZonalAverage.time_zonal_average('480km')
356M 356M 1.00 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('120km')
340M 339M 1.00 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('480km')
6.20±0.05ms 6.17±0.04ms 1.00 quad_hexagon.QuadHexagon.time_open_dataset
5.28±0.1ms 5.27±0.05ms 1.00 quad_hexagon.QuadHexagon.time_open_grid
408 408 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_dataset
392 392 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_grid
73.6k 73.6k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_dataset
72.7k 72.7k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_grid

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@cmdupuis3 cmdupuis3

Labels

benchmarking Related to benchmarks, memory usage, and/or time profiling run-benchmark Run ASV benchmark workflow scalability Related to scalability & performance efforts

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Need a holistic approach to file I/O for benchmarks

1 participant

@cmdupuis3