DOCA GPUNetIO Open Source

This repository provides an open-source version of the DOCA GPUNetIO and DOCA Verbs libraries. The features included here are limited to enabling GPUDirect Async Kernel-Initiated (GDAKI) network communication technology over RDMA protocols (InfiniBand and RoCE) using a DOCA-like API in an open-source environment.

Open vs Full

The table below highlights the key differences between this DOCA GPUNetIO open-source project and the full DOCA GPUNetIO SDK:

Item	DOCA Full SDK	DOCA Open Source
Verbs CPU control path	Closed-source shared library	Open-source C++ files
GPUNetIO CPU control path	Closed-source shared library	Open-source C++ files
GPUNetIO GPU data path for RDMA Verbs one-sided	Yes	Yes
GPUNetIO GPU data path for RDMA Verbs two-sided	Yes	No
GPUNetIO GPU data path for Ethernet	Yes	No
GPUNetIO GPU data path for DMA	Yes	No

The Full SDK is more comprehensive and includes additional features that are not part of this open-source release. It is important to note, however, that the CUDA header files for the GPUNetIO Verbs data path are identical between the open-source and full versions.

Goals

The overarching goal of DOCA GPUNetIO (both Open Source and Full) is to consolidate multiple GDAKI implementations into a unified driver and library with consistent host- and device-side interfaces. This common foundation can be shared across current and future consumers of GDAKI technology such as NVSHMEM, NCCL or UCX. This approach promotes knowledge sharing while reducing the engineering effort required for long-term maintenance.

Core Features

CPU control path:

• Interfaces to create and manage completion queues (CQs) and queue pairs (QPs) in CPU/GPU memory.
• Support for connecting QPs over Reliable Connection (RC) transport.
• Move CQ/QP resources between CPU and GPU memory.
• Compatibility with standard verbs resources (MRs, PDs, context, device attributes, etc.).
• Possibility to use DOCA SDK restricted features via internal dlopen-based dynamic linking.

GPU data path:

• Device-side APIs to post direct work requests (WRs) and poll completion responses (CQEs).
• Directly ring NIC doorbells from the GPU (update registers).

For a deep dive into features, see the official DOCA GPUNetIO documentation and DOCA Verbs documentation.

Usage

To enable GDAKI technology with the DOCA API, an application must be divided into two phases. A CPU control path phase, which initializes devices, allocates memory, and performs other setup tasks. A GPU data path phase, where a CUDA kernel is launched and GPUNetIO CUDA functions are used within it.

Control Path Workflow

1. Open an RDMA device context: ibv_open_device.
2. Allocate a PD: ibv_alloc_pd.
3. Register memory regions: ibv_reg_mr.
4. Create a GPUNetIO handler: doca_gpu_create.
5. Create CQ and QP using doca_verbs_* functions.
6. Connect QPs with remote peers using doca_verbs_qp_modify.
7. Export QPs and CQs to GPU memory using: doca_gpu_verbs_export_cq and doca_gpu_verbs_export_qp

Data Path Workflow

1. Launch a GPU kernel
2. Post work requests using:

• High-level API in CUDA header files doca_gpunetio_dev_verbs_onesided.cuh and doca_gpunetio_dev_verbs_counter.cuh starting with doca_gpu_dev_verbs_*
• Low-level API (advanced users) in CUDA header files like doca_gpunetio_dev_verbs_qp.cuh and doca_gpunetio_dev_verbs_cq.cuh like doca_gpu_dev_verbs_wqe_prepare_*, doca_gpu_dev_verbs_submit

3. Poll completions with: doca_gpu_dev_verbs_poll_cq_*

Mixing high- and low-level APIs is not recommended.

CPU-assisted GDAKI

Some systems do not support direct NIC doorbell ringing from GPU SMs. In this case, a CUDA kernel can post WQEs and poll CQEs in GPU memory, but it cannot update the network card registers. In such scenarios, DOCA GPUNetIO GDAKI can still be used by enabling CPU-assisted mode: the GPU notifies a CPU thread, which rings the NIC doorbell on its behalf. This mode provides a reliable fallback (with lower performance) and requires a CPU thread to periodically call doca_gpu_verbs_cpu_proxy_progress().

Build

To build the host-side library libdoca_gpunetio.so:

cd doca-gpunetio
make -j

The output is a lib directory containing the shared library.

It is also possible to specify CUDA arch at build time, as well as install library and examples in a specific directory. An example to build and install library and examples for Hopper GPU with sm_90 on a given prefix:

make install install_examples PREFIX=/path/to/directory/install CUDA_ARCH=90

Enable logs

Logs are managed by macro DOCA_LOG, relying on syslog with different log levels:

0. EMERG
1. ALERT
2. CRIT
3. ERR
4. WARNING
5. NOTICE
6. INFO
7. DEBUG

By default, the EMERG level (0) is set. To print the DOCA_LOG with higher level, please set the DOCA_GPUNETIO_LOG environment variable to the right level number.

Enable SDK mode

To access DOCA SDK restricted functionality from GPUNetIO open source, CPU functions now use internal dlopen-based dynamic linking. If the environment variable DOCA_SDK_LIB_PATH is set to a valid DOCA SDK library installation directory (typically /opt/mellanox/doca/libs/x86_64-linux-gnu for x86 systems), GPUNetIO open dynamically loads the DOCA SDK functions and uses them instead of the standalone open-source implementation.

An example command line to enable the SDK mode:

$ DOCA_GPUNETIO_LOG=6 DOCA_SDK_LIB_PATH=/opt/mellanox/doca/lib/x86_64-linux-gnu ./gpunetio_verbs_write_lat -d mlx5_0 -g 8a:00.0
Wed Apr  1 10:42:00 2026 [INFO] [examples/verbs_common.cpp]: 255: create_verbs_resources(): Setting GPU device 0 at 8a:00.0
Wed Apr  1 10:42:00 2026 [WARNING] [src/doca_gpunetio_sdk_wrapper.cpp]: 202: doca_gpu_sdk_wrapper_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64-linux-gnu. DOCA SDK is in use
Wed Apr  1 10:42:00 2026 [INFO] [src/doca_gpunetio.cpp]: 120: doca_gpu_create(): Use DOCA GPUNetIO SDK
Wed Apr  1 10:42:00 2026 [WARNING] [src/doca_verbs_dev_sdk_wrapper.cpp]: 164: doca_verbs_sdk_wrapper_dev_open_from_pd(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64-linux-gnu. DOCA SDK is in use
Wed Apr  1 10:42:00 2026 [INFO] [src/doca_verbs_dev.cpp]: 83: doca_verbs_dev_open(): Use DOCA Verbs Dev SDK
Wed Apr  1 10:42:00 2026 [WARNING] [src/doca_verbs_qp_sdk_wrapper.cpp]: 1430: doca_verbs_sdk_wrapper_ah_attr_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64-linux-gnu. DOCA SDK is in use
Wed Apr  1 10:42:00 2026 [INFO] [src/doca_verbs_qp.cpp]: 2902: doca_verbs_ah_attr_create(): Use DOCA Verbs AH Attr SDK
Wed Apr  1 10:42:00 2026 [WARNING] [src/doca_verbs_cq_sdk_wrapper.cpp]: 246: doca_verbs_sdk_wrapper_cq_attr_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64-linux-gnu. DOCA SDK is in use
Wed Apr  1 10:42:00 2026 [INFO] [src/doca_verbs_cq.cpp]: 327: doca_verbs_cq_attr_create(): Use DOCA Verbs CQ Attr SDK
Wed Apr  1 10:42:00 2026 [WARNING] [src/doca_verbs_umem_sdk_wrapper.cpp]: 204: doca_verbs_sdk_wrapper_umem_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64-linux-gnu. DOCA SDK is in use
Wed Apr  1 10:42:00 2026 [INFO] [src/doca_verbs_umem.cpp]: 150: doca_verbs_umem_create(): Use DOCA Verbs UMEM SDK
Wed Apr  1 10:42:00 2026 [WARNING] [src/doca_verbs_cq_sdk_wrapper.cpp]: 449: doca_verbs_sdk_wrapper_cq_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64-linux-gnu. DOCA SDK is in use
Wed Apr  1 10:42:00 2026 [INFO] [src/doca_verbs_cq.cpp]: 589: doca_verbs_cq_create(): Use DOCA Verbs CQ SDK
Wed Apr  1 10:42:00 2026 [WARNING] [src/doca_verbs_uar_sdk_wrapper.cpp]: 160: doca_verbs_sdk_wrapper_uar_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64-linux-gnu. DOCA SDK is in use
Wed Apr  1 10:42:00 2026 [INFO] [src/doca_verbs_uar.cpp]: 164: doca_verbs_uar_create(): Use DOCA Verbs UAR SDK
Wed Apr  1 10:42:00 2026 [WARNING] [src/doca_verbs_qp_sdk_wrapper.cpp]: 574: doca_verbs_sdk_wrapper_qp_init_attr_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64-linux-gnu. DOCA SDK is in use
....

Prints like Use DOCA GPUNetIO SDK, Use DOCA Verbs Dev SDK, etc.. indicate the path specified in DOCA_SDK_LIB_PATH correcly points to a DOCA SDK library installation directory.

Conversely, if something is wrong with the path specified in DOCA_SDK_LIB_PATH, the output changes:

$ DOCA_GPUNETIO_LOG=6 DOCA_SDK_LIB_PATH=/opt/mellanox/doca/lib/x86_64 ./gpunetio_verbs_write_lat -d mlx5_0 -g 8a:00.0
Wed Apr  1 10:41:52 2026 [INFO] [examples/verbs_common.cpp]: 255: create_verbs_resources(): Setting GPU device 0 at 8a:00.0
Wed Apr  1 10:41:52 2026 [ERR] [src/doca_gpunetio_sdk_wrapper.cpp]: 110: doca_gpunetio_sdk_wrapper_init(): Failed to find libdoca_common.so library /opt/mellanox/doca/lib/x86_64/libdoca_common.so (DOCA_SDK_LIB_PATH=/opt/mellanox/doca/lib/x86_64)
Wed Apr  1 10:41:52 2026 [WARNING] [src/doca_gpunetio_sdk_wrapper.cpp]: 193: doca_gpu_sdk_wrapper_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64, but DOCA SDK libraries not found. DOCA SDK is not in use
Wed Apr  1 10:41:52 2026 [INFO] [src/doca_gpunetio.cpp]: 131: doca_gpu_create(): Use DOCA GPUNetIO open
Wed Apr  1 10:41:52 2026 [ERR] [src/doca_verbs_dev_sdk_wrapper.cpp]: 87: doca_verbs_sdk_wrapper_init(): Failed to find libdoca_common.so library /opt/mellanox/doca/lib/x86_64/libdoca_common.so (DOCA_SDK_LIB_PATH=/opt/mellanox/doca/lib/x86_64)
Wed Apr  1 10:41:52 2026 [WARNING] [src/doca_verbs_dev_sdk_wrapper.cpp]: 154: doca_verbs_sdk_wrapper_dev_open_from_pd(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64, but DOCA SDK libraries not found. DOCA SDK is not in use
Wed Apr  1 10:41:52 2026 [INFO] [src/doca_verbs_dev.cpp]: 94: doca_verbs_dev_open(): Use DOCA Verbs Dev open
Wed Apr  1 10:41:52 2026 [INFO] [src/doca_verbs_dev.cpp]: 106: doca_verbs_dev_open(): doca_verbs_dev_open=0x564b462a7e00 was created
Wed Apr  1 10:41:52 2026 [ERR] [src/doca_verbs_qp_sdk_wrapper.cpp]: 295: doca_verbs_sdk_wrapper_init(): Failed to find libdoca_common.so library /opt/mellanox/doca/lib/x86_64/libdoca_common.so (DOCA_SDK_LIB_PATH=/opt/mellanox/doca/lib/x86_64)
Wed Apr  1 10:41:52 2026 [WARNING] [src/doca_verbs_qp_sdk_wrapper.cpp]: 1404: doca_verbs_sdk_wrapper_ah_attr_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64, but DOCA SDK libraries not found. DOCA SDK is not in use
Wed Apr  1 10:41:52 2026 [INFO] [src/doca_verbs_qp.cpp]: 2913: doca_verbs_ah_attr_create(): Use DOCA Verbs AH Attr open
Wed Apr  1 10:41:52 2026 [INFO] [src/doca_verbs_qp.cpp]: 2919: doca_verbs_ah_attr_create(): doca_verbs_verbs_ah_open=0x564b462a7e20 was created
Wed Apr  1 10:41:52 2026 [ERR] [src/doca_verbs_cq_sdk_wrapper.cpp]: 133: doca_verbs_sdk_wrapper_init(): Failed to find libdoca_common.so library /opt/mellanox/doca/lib/x86_64/libdoca_common.so (DOCA_SDK_LIB_PATH=/opt/mellanox/doca/lib/x86_64)
Wed Apr  1 10:41:52 2026 [WARNING] [src/doca_verbs_cq_sdk_wrapper.cpp]: 237: doca_verbs_sdk_wrapper_cq_attr_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64, but DOCA SDK libraries not found. DOCA SDK is not in use
Wed Apr  1 10:41:52 2026 [INFO] [src/doca_verbs_cq.cpp]: 338: doca_verbs_cq_attr_create(): Use DOCA Verbs CQ Attr open
Wed Apr  1 10:41:52 2026 [INFO] [src/doca_verbs_cq.cpp]: 344: doca_verbs_cq_attr_create(): doca_verbs_cq_attr_open=0x564b462a7eb0 was created
Wed Apr  1 10:41:52 2026 [ERR] [src/doca_verbs_umem_sdk_wrapper.cpp]: 106: doca_verbs_sdk_wrapper_init(): Failed to find libdoca_gpunetio.so library /opt/mellanox/doca/lib/x86_64/libdoca_gpunetio.so (DOCA_SDK_LIB_PATH=/opt/mellanox/doca/lib/x86_64)
Wed Apr  1 10:41:52 2026 [WARNING] [src/doca_verbs_umem_sdk_wrapper.cpp]: 186: doca_verbs_sdk_wrapper_umem_create(): Env var DOCA_SDK_LIB_PATH set to /opt/mellanox/doca/lib/x86_64, but DOCA SDK libraries not found. DOCA SDK is not in use
Wed Apr  1 10:41:52 2026 [INFO] [src/doca_verbs_umem.cpp]: 161: doca_verbs_umem_create(): Use DOCA Verbs UMEM open
...

Examples

Six examples are included to demonstrate usage and measure performance. Make sure to build libdoca_gpunetio.so before compiling examples.

All examples require both a client and a server running on network-connected machines. Client and server must run the same example with the same number of iterations (-i): they check it when they connect and stop on a mismatch. With the AUTO NIC handler, an example falls back to CPU proxy, and starts the CPU proxy thread, when the GPU can't ring the NIC doorbell. NIC handler 10 (GPU SM no DBR) rings the doorbell from the GPU without updating the doorbell record. It uses the NIC hardware no-DBR mode when available (ConnectX-8), otherwise the software-emulated one, which needs GDRCopy and runs a CPU thread that updates the doorbell record. In gpunetio_verbs_write_lat, -r 1 (hardware) or -r 2 (software-emulated) selects the mode; BlueFlame (-p 6) can't use -r 2, as it writes the doorbell record from the GPU. NIC handler 17 (CPU proxy free flow): the GPU marks each WQE ready and a CPU thread rings the doorbell. Data Direct (-z 1) registers the GPU buffers through the NIC Data Direct device. Each side only changes its own registrations, so pass it to both client and server. It needs CUDA 12.8 or newer and a Data Direct-capable libmlx5, both detected at build time by scripts/configure, and a NIC that exposes a Data Direct device. GPU timers can be enabled per operation by setting #define KERNEL_DEBUG_TIMES 1 (useful for debugging, not recommended for performance testing).

Additional samples are available in the NVIDIA DOCA Full Samples repository.

The following command lines assume samples are running on systems where GPU is at PCIe address 8A:00.0 and NIC interface is mlx5_0.

Example 1: gpunetio_verbs_put_bw

This example is a GDAKI perftest ib_write_bw-like benchmark where client launches a CUDA kernel to execute the high-level doca_gpu_dev_verbs_put operation. Server doesn't launch any CUDA kernel: once the client reports that all its runs have completed, server validates the data received from client and exits. Server fails without validating if the client closes the connection first, for example on an error, or if ctrl+c stops the wait.

Run (server):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_put_bw -g 8A:00.0 -d mlx5_0

Run (client):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_put_bw -g 8A:00.0 -d mlx5_0 -c 192.168.1.64

Modes:

• CUDA Thread execution scope (default).
• CUDA Warp execution scope: add -e 1.
• NIC handler type: add -p <nic_handler value> where 0: AUTO (default), 1: CPU Proxy, 2: GPU SM DB, 10: GPU SM no DBR, 17: CPU Proxy free flow.
• Data Direct: add -z 1. The server buffers are then in GPU memory too.

Validation success message (server):

Validation successful! Data received correctly from client.

Example 2: gpunetio_verbs_write_lat

This example is a GDAKI perftest ib_write_lat-like benchmark where Client and server both launch CUDA kernels using low-level APIs.

Run (server):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_write_lat -g 8A:00.0 -d mlx5_0 -p 2

Run (client):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_write_lat -g 8A:00.0 -d mlx5_0 -p 2 -c <server_ip_address>

Modes:

• NIC handler type: add -p <nic_handler value> where 0: AUTO (default), 1: CPU Proxy, 2: GPU SM DB, 6: GPU SM BlueFlame, 10: GPU SM no DBR, 17: CPU Proxy free flow.
• Data Direct: add -z 1.

Example 3: gpunetio_verbs_write_bw

This example is a GDAKI perftest ib_write_bw-like benchmark where client launches a CUDA kernel using low-level APIs. Server doesn't launch any CUDA kernel: once the client reports that all its runs have completed, server validates the data received from client and exits. Server fails without validating if the client closes the connection first, for example on an error, or if ctrl+c stops the wait. The kernel posts all the writes before polling the last completion, so the number of iterations (-i) can't exceed the send queue size, 2048.

Run (server):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_write_bw -g 8A:00.0 -d mlx5_0

Run (client):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_write_bw -g 8A:00.0 -d mlx5_0 -c <server_ip_address>

Modes:

• NIC handler type: add -p <nic_handler value> where 0: AUTO (default), 1: CPU Proxy, 2: GPU SM DB, 10: GPU SM no DBR, 17: CPU Proxy free flow.
• Data Direct: add -z 1. The server buffers are then in GPU memory too.

GPU SM BlueFlame is not supported in this example.

Validation success message (server):

Validation successful! Data received correctly from client.

Example 4: gpunetio_verbs_get_bw

This example is a GDAKI perftest ib_read_bw-like benchmark where client launches a CUDA kernel to execute the high-level doca_gpu_dev_verbs_get operation, reading the server buffers into its GPU memory. Server doesn't launch any CUDA kernel: it keeps its buffers registered until the client reports that all its runs have completed and that the data it read is correct. Server fails if the client closes the connection first, for example on an error or a validation failure, or if ctrl+c stops the wait. On GPUs older than Hopper, the kernel waits for the gets with doca_gpu_dev_verbs_get_wait and DOCA_GPUNETIO_VERBS_MCST_ENABLED: a Dump WQE then makes the data the NIC wrote into GPU memory visible to the GPU.

Run (server):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_get_bw -g 8A:00.0 -d mlx5_0

Run (client):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_get_bw -g 8A:00.0 -d mlx5_0 -c <server_ip_address>

Modes:

• CUDA Thread execution scope (default).
• CUDA Warp execution scope: add -e 1.
• NIC handler type: add -p <nic_handler value> where 0: AUTO (default), 1: CPU Proxy, 2: GPU SM DB, 10: GPU SM no DBR, 17: CPU Proxy free flow.
• Data Direct: add -z 1. The server buffers are then in GPU memory too.

Validation success message (client):

Validation successful! Data read correctly from server

Example 5: gpunetio_verbs_put_signal_bw

This example is a GDAKI perftest ib_write_bw-like benchmark where client launches a CUDA kernel to execute the high-level doca_gpu_dev_verbs_put_signal operation: each CUDA thread writes its own chunk of the server buffer, then adds 1 to its own server signal with an atomic fetch and add, which returns the previous signal value to the client. With -w 1 each CUDA thread executes instead the high-level doca_gpu_dev_verbs_put_immediate operation: a single RDMA Write with Immediate, whose immediate carries the thread's own signal id and the value 1 to add to it. Server creates a rank-local signal map with doca_gpu_verbs_signal_map_create() and registers one signal per CUDA thread and message size with doca_gpu_verbs_signal_register_range(), so that the signal id of a CUDA thread is (message size index * CUDA threads) + CUDA thread index. A service, given the map with doca_gpu_verbs_service_set_signal_map() and the QP with doca_gpu_verbs_service_monitor_qp(), drains the QP receive queue, adds the value of each immediate to the GPU memory slot of its signal id, and returns the receive credits to the NIC. There is no remote signal address, no memory key, and no previous value returned to the client. Server doesn't launch the benchmark kernel: once the client reports that all its runs have completed, server validates the data received from client, logs the sum of its signal values for each message size and exits. With -w 1 that report only means the server NIC acknowledged the client's writes, and the service may still be adding their values: the server first waits, up to 10 seconds, until every signal reaches 2 × (iterations / CUDA threads), counting the short ones with a small CUDA kernel because the host can't copy registered signals, and then checks that each signal holds exactly that value. Server fails without validating if the client closes the connection first, for example on an error, or if ctrl+c stops the wait.

Run (server):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_put_signal_bw -g 8A:00.0 -d mlx5_0

Run (client):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_put_signal_bw -g 8A:00.0 -d mlx5_0 -c <server_ip_address>

Modes:

• CUDA Thread execution scope (default).
• CUDA Warp execution scope: add -e 1. Each warp then posts the puts of its threads together with a single fetch and add, on the signal of its lane 0.
• NIC handler type: add -p <nic_handler value> where 0: AUTO (default), 1: CPU Proxy, 2: GPU SM DB, 10: GPU SM no DBR, 17: CPU Proxy free flow.
• RDMA Write with Immediate and signal map: add -w 1, on both peers. The server QP is then created with a receive queue, and its signals are the slots of its signal map, in GPU memory even without Data Direct. Requires GDRCopy, which maps the receive doorbell record and the signal slots for the CPU. Thread scope only: the immediate of a single RDMA Write carries a single signal id, so -e 1 is rejected.
• Data Direct: add -z 1. The server buffers and signals are then in GPU memory too.

Validation success message (server):

Validation successful! Data received correctly from client

Example 6: gpunetio_verbs_put_signal_lat

This example is a GDAKI perftest ib_write_lat-like benchmark where client and server both launch a single CUDA thread kernel. Each message is a high-level doca_gpu_dev_verbs_put_signal operation, and each side polls its own signal, which the other side increments after writing the message, to know when the message has arrived. With -w 1 each message is instead a high-level doca_gpu_dev_verbs_put_immediate operation: a single RDMA Write with Immediate, whose immediate carries the signal id of the remote peer and the value 1 to add to it. Both peers create a rank-local signal map with doca_gpu_verbs_signal_map_create() and register one signal per message size with doca_gpu_verbs_signal_register(), so that the signal id of a message size is its index. As the map is rank-local, both peers use the same ids, and each one signals the id of the message size it is sending. A service, given the map with doca_gpu_verbs_service_set_signal_map() and the QP with doca_gpu_verbs_service_monitor_qp(), drains the QP receive queue and adds the value of each immediate to the GPU memory slot its kernel polls.

Run (server):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_put_signal_lat -g 8A:00.0 -d mlx5_0

Run (client):

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/path/to/doca-gpunetio/lib DOCA_GPUNETIO_LOG=6 ./gpunetio_verbs_put_signal_lat -g 8A:00.0 -d mlx5_0 -c <server_ip_address>

Modes:

• NIC handler type: add -p <nic_handler value> where 0: AUTO (default), 1: CPU Proxy, 2: GPU SM DB, 10: GPU SM no DBR, 17: CPU Proxy free flow.
• RDMA Write with Immediate and signal map: add -w 1, on both peers. Both QPs are then created with a receive queue, and the signal each side polls is a slot of its own signal map. Requires GDRCopy, which maps the receive doorbell record and the signal slots for the CPU. The service thread also progresses the QP for the CPU proxy NIC handlers, so the sample doesn't start its own CPU proxy thread in this mode.
• Data Direct: add -z 1.

GPU SM BlueFlame is not supported in this example.

Acknowledgments

If you use this software in your work, please cite the official DOCA GPUNetIO documentation.

Contributing

This project is developed internally and released as open source. We currently do not accept external contributions.

Troubleshooting & Feedback

We appreciate community discussion and feedback in support of DOCA GPUNetIO Open users and developers. We ask that users:

• Review the DOCA SDK Programming Guide for system configuration, technology explaination, API, etc...
• Ask questions on the NVIDIA DOCA Support Forum.
• Report issues on the GitHub Issues board.

License

See the LICENSE.txt file.