GFDS: CUDA Fortran GPU acceleration and a proposed experimental branch

[mxd45za]

GFDS: An Experimental CUDA Fortran Implementation of FDS — Seeking Feedback

Dear FDS Development Team,

Thank you for developing and maintaining FDS. I have been working on GFDS, an independent, experimental CUDA Fortran GPU implementation based on FDS, and would appreciate your feedback on the approach and possible ways to collaborate.

Implementation and Scope

GFDS focuses on GPU-resident computation within the main time-stepping loop, keeping the primary simulation data in device memory. The overall workflow is:

FDS input and CPU initialization
    -> H2D transfer
    -> GPU-resident time-stepping loop
       (D2H transfers and standard FDS output/dump operations as needed)
    -> finalization

The original CPU execution path is retained, and both single-GPU and MPI-based multi-GPU execution are available. The current GPU implementation covers selected VLES calculations, combustion/species coupling, single-band gray-gas radiation, and non-pyrolyzing one-dimensional solid heat conduction. Pyrolysis, HVAC, and CC_IBM are not yet supported in the GPU path.

Preliminary Performance

In my preliminary tests with simple_test-type fire cases of approximately 10 million cells, I observed a 60–80× speedup on a single NVIDIA H20 compared with FDS running on one CPU core of an AMD EPYC 9K84 96-Core Processor.

A four-GPU configuration achieved 80% strong-scaling efficiency in the tested setup. Scaling depends on mesh decomposition and communication overhead, including NVLink/PCIe connectivity.

Separate tests on an NVIDIA GeForce RTX 4060 Laptop GPU showed approximately 30× acceleration. Single-precision execution provided a further speedup approaching 2× in selected tests, with similar overall trends in the quantities examined. These preliminary observations are specific to the tested configurations and do not constitute a comprehensive numerical accuracy assessment.

Verification and Codebase

I have CPU/GPU comparison cases and supporting reports available for review. The README distinguishes cases with saved CPU/GPU comparison results from engineering inputs retained for development and further verification. The available results cover selected input configurations, not all combinations of FDS models and inputs.

Development started from the June 2025 FDS codebase, and not all subsequent upstream changes have been incorporated. The current implementation builds and runs on Linux using nvfortran, MPI, and OpenMP, with changes to some source files and the CMake configuration. I have also incorporated selected fixes for compatibility with more recent Smokeview versions.

Feedback and Possible Collaboration

Would the team be open to an initial technical discussion about this work? I would welcome feedback on the implementation and on any additional comparison tests you would consider useful.

My preference is to keep this work in a separate experimental branch while development and verification continue, rather than seek mainline integration at this stage. If, after review, you consider an experimental branch in the official repository appropriate, I would be glad to discuss that possibility. Continuing in a separately maintained GFDS repository would also be an option.

Since the work has not yet been published and the source code is currently private, would there be a suitable contact for an initial non-public exchange?

Thank you for your time and consideration.

Best regards,
Xiao Xinming
Sun Yat-sen university
email adress : xiaoxm8@mail2.sysu.edu.cn

Attachment
Simple validation results.pdf
README(1)(1).md

[marcosvanella]

Hi Xiao, your GPU implementation work seems very interesting. We can discuss on the implementation you have in this thread, but note we already have a research branch in firemodels (FireX). All experimental work is being added in there, including linking to GPU enabled libraries.
I'm curious, do you have any technical documentation on the Cuda Fortran implementation you have done?
Also, if you give us access to your repository we can do a performance comparison here using all NVIDIA CPU nodes (Grace-Grace) vs a CPU-GPU nodes (Grace-Hopper).
Best Regards,
Marcos

[marcosvanella]

Thanks. If it is convenient on your side and you have two GPU, it might also be interesting to try the same 72-mesh case on two GPUs. Also I curious about how many the multi-GPU scaling efficiency, since with many MPI processes the MESH EXCHANGE overhead might become significant. XiaoXinming

Hi Xiao, question for you: If I have a node with 72 CPU cores and 2 gpus and want 72 MPI processes (one mesh per process), 36 assigned to each gpu, how do I submit the run? Will it distribute the MPI processes equally per GPU?

[mxd45za]

Hi Marcos,

Yes, provided all 72 processes can see the same two GPUs. The current assignment is implemented in Source/main.f90:

GPU_DEVICE_ID = MOD(GPU_LOCAL_RANK,GPU_DEVICE_COUNT)
CUDA_STAT = CUDASetDevice(GPU_DEVICE_ID)

GPU_LOCAL_RANK is the rank within the node, and GPU_DEVICE_COUNT is the number of GPUs visible to that process.

With two GPUs, even-numbered ranks use GPU 0 and odd-numbered ranks use GPU 1, so each GPU gets 36 processes. Each process still keeps the meshes assigned to it.

This is a simple round-robin assignment. It balances the number of processes across the two GPUs, but it does not consider mesh adjacency or computational load.

I have also invited both @cxp484 and @ericvmueller to the repository.

Best regards,

Xiaoxinming

[ericvmueller]

I got the invite and am eager to run some tests. Thank you for sharing with us!

[cxp484]

Hi Marcos,

Yes, provided all 72 processes can see the same two GPUs. The current assignment is implemented in Source/main.f90:

GPU_DEVICE_ID = MOD(GPU_LOCAL_RANK,GPU_DEVICE_COUNT)
CUDA_STAT = CUDASetDevice(GPU_DEVICE_ID)

GPU_LOCAL_RANK is the rank within the node, and GPU_DEVICE_COUNT is the number of GPUs visible to that process.

With two GPUs, even-numbered ranks use GPU 0 and odd-numbered ranks use GPU 1, so each GPU gets 36 processes. Each process still keeps the meshes assigned to it.

This is a simple round-robin assignment. It balances the number of processes across the two GPUs, but it does not consider mesh adjacency or computational load.

I have also invited both @cxp484 and @ericvmueller to the repository.

Best regards,

Xiaoxinming

Thanks a lot, Xiao. Nice work! I got the invite and will try to test it out and help Marcos along the way as needed.

[mxd45za]

Thanks Eric and Chris. I appreciate your help and look forward to your test results.

[mxd45za]

@marcosvanella @cxp484 @ericvmueller

Hi, I made a small update to make it easier to test different MPI process counts without editing the input file.

The original MPI assignment rules in the FDS version used by GFDS have two restrictions:

1. If MPI_PROCESS is not specified, running fewer MPI processes than meshes causes an error, except for a single-process run.

2. If MPI_PROCESS is specified, launching fewer processes than required by those assignments also requires editing the input file, again except for a single-process run.

This is inconvenient for GPU testing because GPU nodes usually have far fewer GPUs than CPU cores.

In some of my H20 tests, two MPI processes on one GPU were faster than one, mainly because DUMP took less time. But using too many processes can also increase GPU scheduling, MPI communication, synchronization, and MESH_EXCHANGE time.

Meshes within the same process exchange data through device-memory copies. Communication between processes goes through MPI. With CUDA-aware MPI, device buffers may be transferred directly without staging through host memory.

The updated rules are:

1. Automatic mesh assignment:
   If no mesh specifies MPI_PROCESS, meshes are divided into contiguous groups in input order. Any remainder is assigned to the first few ranks.

   Example: 10 meshes with 3 processes:
   rank 0 -> meshes 1-4
   rank 1 -> meshes 5-7
   rank 2 -> meshes 8-10

2. Reassignment when fewer processes are launched:
   If any specified MPI_PROCESS is outside the launched process range, the original assignments are replaced with the automatic distribution above.
   Otherwise, the existing manual assignments are kept.

3. Contiguous MPI-to-GPU assignment:
   When node-local processes see the same GPU list, processes are assigned to GPUs in contiguous groups. Scheduler-provided per-process GPU visibility is still respected.

The changes are in:

• read.f90: mesh-to-MPI assignment
• g_wall.f90: removed the HT1D diagnostic messages
• main.f90: MPI-to-GPU binding changed to contiguous groups

This is still a static, count-based assignment. It is mainly intended to make it easier to choose a more suitable number of MPI processes for GPU execution.

Running 144 MPI processes on one GPU may introduce considerable overhead. With this update, you can test fewer MPI processes than meshes simply by changing mpirun -np, while keeping the same input file.

Best regards,

Xiaoxinming