Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,8 @@ if (BYCORF_ENABLE_RDMA)
find_package(PkgConfig REQUIRED)
pkg_check_modules(BYCORF_IBVERBS REQUIRED IMPORTED_TARGET libibverbs)
pkg_check_modules(BYCORF_RDMACM REQUIRED IMPORTED_TARGET librdmacm)
target_sources(bycorf PRIVATE src/io/rdma_memory.cpp)
target_sources(bycorf PRIVATE src/io/rdma_memory.cpp src/rdma/rc.cpp
src/rdma/rc_engine.cpp)
target_link_libraries(bycorf PUBLIC
PkgConfig::BYCORF_IBVERBS PkgConfig::BYCORF_RDMACM)
endif()
Expand Down Expand Up @@ -473,6 +474,7 @@ if (BYCORF_BUILD_TESTS)
tests/cross_core_drain_test.cpp
tests/worker_scheduler_test.cpp
tests/received_buffer_queue_test.cpp
tests/local_shared_ptr_test.cpp
tests/backend_selection_test.cpp)
target_link_libraries(bycorf_unit_tests PRIVATE bycorf::core GTest::gtest_main)
bycorf_discover_tests(bycorf_unit_tests 30 "runtime;unit")
Expand All @@ -487,6 +489,25 @@ if (BYCORF_BUILD_TESTS)
endforeach()
bycorf_discover_tests(bycorf_rdma_memory_tests 30 "runtime;rdma;unit")

add_executable(bycorf_rdma_rc_fake_tests tests/rdma_rc_test.cpp tests/rdma_fake.cpp)
target_compile_definitions(bycorf_rdma_rc_fake_tests PRIVATE BYCORF_RDMA_FAKE=1)
target_link_libraries(bycorf_rdma_rc_fake_tests PRIVATE bycorf::core GTest::gtest_main)
foreach(symbol IN ITEMS rdma_create_event_channel rdma_destroy_event_channel
rdma_create_id rdma_destroy_id rdma_bind_addr rdma_listen rdma_resolve_addr
rdma_resolve_route rdma_connect rdma_accept rdma_establish rdma_disconnect
rdma_reject rdma_get_cm_event rdma_ack_cm_event rdma_init_qp_attr
ibv_get_device_name ibv_open_device ibv_close_device ibv_query_device
ibv_alloc_pd ibv_dealloc_pd ibv_create_comp_channel ibv_destroy_comp_channel
ibv_create_cq ibv_destroy_cq ibv_create_qp ibv_modify_qp ibv_destroy_qp
ibv_get_cq_event ibv_ack_cq_events ibv_get_async_event ibv_ack_async_event)
target_link_options(bycorf_rdma_rc_fake_tests PRIVATE "LINKER:--wrap=${symbol}")
endforeach()
bycorf_discover_tests(bycorf_rdma_rc_fake_tests 30 "runtime;rdma;stub")

add_executable(bycorf_rdma_rc_tests tests/rdma_rc_test.cpp)
target_link_libraries(bycorf_rdma_rc_tests PRIVATE bycorf::core GTest::gtest_main)
bycorf_discover_tests(bycorf_rdma_rc_tests 30 "runtime;rdma;integration")

add_executable(bycorf_rdma_registration_tests tests/rdma_registration_test.cpp)
target_link_libraries(bycorf_rdma_registration_tests PRIVATE bycorf::core GTest::gtest_main)
bycorf_discover_tests(bycorf_rdma_registration_tests 30 "runtime;rdma;integration")
Expand Down
40 changes: 32 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ connections, and asynchronous I/O; applications provide services above

- [Name](#name)
- [Build and test](#build-and-test)
- [Optional RDMA registration](#optional-rdma-registration)
- [Optional RDMA connections and registration](#optional-rdma-connections-and-registration)
- [Run the examples](#run-the-examples)
- [Use Bycorf in a CMake project](#use-bycorf-in-a-cmake-project)
- [Development](#development)
Expand Down Expand Up @@ -179,18 +179,18 @@ Follow the [DPDK/SPDK build and test guide](docs/dpdk-prototype.md) for its
additional dependencies, virtual-device checks, physical-device configuration,
and experimental compatibility limits.

## Optional RDMA registration
## Optional RDMA connections and registration

`BYCORF_ENABLE_RDMA` enables the private worker facade and explicit memory
registration API. It defaults to off and is independent of
`BYCORF_ENABLE_RDMA` enables the private worker facade, RC connection API and
explicit memory registration. It defaults to off and is independent of
`BYCORF_KERNEL_BYPASS`; enabling it requires libibverbs and librdmacm development
packages, without DPDK/SPDK. On Ubuntu:

```bash
sudo apt-get install --no-install-recommends libibverbs-dev librdmacm-dev
cmake -S . -B build-rdma -G Ninja -DCMAKE_BUILD_TYPE=Debug -DBYCORF_ENABLE_RDMA=ON
cmake --build build-rdma --parallel
ctest --test-dir build-rdma -R '^bycorf_RdmaMemory\.' --output-on-failure
ctest --test-dir build-rdma -R '^bycorf_(RdmaMemory|RcLoopFake)\.' --output-on-failure
```

The device-free `RdmaMemory` suite injects verbs failures and verifies lazy
Expand All @@ -206,9 +206,21 @@ BYCORF_RDMA_TEST_DEVICE=rxe0 ctest --test-dir build-rdma \

This opens the selected device, registers and deregisters memory, creates real
QPs sharing a PD, and checks permissions, bounds, and mismatched PD rejection.
It requires access to the device's uverbs node and sufficient memlock. It does
not test a connected data-transfer or immediate-data protocol. No device is
opened by unused workers or test discovery.
It requires access to the device's uverbs node and sufficient memlock. For RC
connection tests on an existing Soft-RoCE interface, supply its local IP:

```bash
BYCORF_RDMA_TEST_IP=192.0.2.1 ctest --test-dir build-rdma \
-R '^bycorf_RcLoop\.' --output-on-failure
```

These exercise idle-worker connection events, private data, distinct worker
contexts, shared CQ/PD ownership, refused connections and close/accept races.
The device-free `RcLoopFake` cases additionally inject notification races,
provider capacity rounding, budget exhaustion, deadlines, CQ/device failures,
and late events during cancellation and shutdown. They use real io_uring with
link-time CM/verbs stubs. Data-transfer and immediate-data protocols remain
outside these tests. No device is opened by unused workers or test discovery.

Use `Worker::OpenRdmaDevice(name)` or `AcquireRdmaDomain(owning_context)` on the
owner worker, then `RegisterRdmaMemory(domain, backing_span, explicit_access)`.
Expand All @@ -219,6 +231,18 @@ backing memory and must stop remote access before releasing registrations and
freeing it. See [RDMA ownership](docs/architecture/rdma.md) for remote descriptors,
permissions, CM context ownership, and the independent storage-buffer contract.

The RC API is in `bycorf/rdma/rc.h`. On the owning worker, call
`RcListener::Listen(worker, numeric_ip, port)` and await `listener.Accept()`, or
await `RcConnection::Connect(worker, numeric_ip, port)`. Pass `RcOptions` to set
QP/CQ capacities and the handshake timeout, and an optional byte span for CM
private data. Await `connection.Close()` or `listener.Close()` to join resource
retirement. Destruction only requests close. Use
`connection.protection_domain()` when registering memory for that connection.
An accepted connection stays on the listener's worker; moving a C++ handle
does not move worker ownership. Private data is limited to 56 bytes; CM may
report a padded peer-data length. The first connection fixes that worker/device
CQ capacity, and subsequent connections must request the same capacity.

## Run the examples

The default build enables `BYCORF_BUILD_EXAMPLES`. If it was disabled, set
Expand Down
7 changes: 4 additions & 3 deletions docs/architecture/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,9 @@ and asynchronous I/O. Abseil supplies status types and spdlog supplies logging.
| Module | Responsibility | Document |
|---|---|---|
| Runtime | Worker lifecycle, coroutine scheduling, sleep coordination, cross-worker and foreign ingress | [Runtime](runtime.md) |
| I/O | Worker-owned shared io_uring, network/storage adapters, optional RDMA registration and SPDK storage, shared DPDK environment | [I/O backends](io.md), [RDMA ownership](rdma.md) |
| I/O | Worker-owned shared io_uring, network/storage adapters, optional SPDK storage, shared DPDK environment | [I/O backends](io.md) |
| Networking | Listeners, streams, services, TLS, optional DPDK/FreeBSD IPv4 TCP | [Networking](networking.md) |
| RDMA | Worker-owned RC connections, CM/QP lifecycle, shared CQ notifications and memory registration | [RDMA](rdma.md) |
| RPC | Framed request/reply protocol above streams | [RPC](rpc.md) |

Each connection and its coroutines have one worker owner. Key ownership and
Expand All @@ -38,8 +39,8 @@ worker before that coroutine uses its stream again.
Startup selection independently activates network and storage backends and
remains fixed for the process lifetime. Both default to io_uring even in a
bypass-capable build.
`BYCORF_ENABLE_RDMA` independently enables the RDMA facade and registration
layer; resources are acquired only on explicit use.
`BYCORF_ENABLE_RDMA` independently enables the RDMA facade, RC connections and
registration layer; resources are acquired only on explicit use.
The [prototype runbook](../dpdk-prototype.md) describes the DPDK configuration.

Architecture documents describe the current core model and stable tradeoffs.
Expand Down
11 changes: 6 additions & 5 deletions docs/architecture/io.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,12 @@ Network adapters supply notification preparation, idle checks,
deadlines, and cleanup to the [worker sleep protocol](runtime.md#sleep-and-wakeups).
The worker enters the ring's blocking wait; adapters never block independently.

The ring outlives all facades. RDMA drops its weak resource cache on shutdown;
registration and operation handles retain their PD/context independently. The
worker initializes the ring first, stops storage and networking in that order,
then shuts the ring down. Storage dispatch remains
valid through shutdown completion draining; SPDK callbacks use the owning
The ring outlives all facades. RDMA stops admission before ring shutdown but
retains channel polls until the ring has quiesced; it then retires remaining
RDMA events and connection resources. Exported registration handles retain
their PD/context independently. The worker initializes the ring first, stops
storage and networking in that order, then shuts the ring down. Storage dispatch
remains valid through shutdown completion draining; SPDK callbacks use the owning
worker's TLS to reach its storage adapter. Kernel registrations remain live
until ring shutdown quiesces kernel access.

Expand Down
103 changes: 98 additions & 5 deletions docs/architecture/rdma.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,16 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# RDMA registration and ownership
# RDMA connections, notifications and memory ownership

`BYCORF_ENABLE_RDMA` adds an independent I/O capability backed by libibverbs and
librdmacm. It defaults to off and does not select or require DPDK/SPDK. The
private `RdmaBackend<Worker>` facade reaches the worker's existing io_uring
through `KernelBackend()` for channel readiness and cancellation. It creates no
ring, polling thread, thread pool, or process-wide blocking lock. The worker
continues to own shared-ring submission and completion dispatch.
continues to own shared-ring submission, completion dispatch and sleeping.
`bycorf::rdma::RcListener` and `RcConnection` provide numeric-address RC
Listen/Accept/Connect/Close and CM private data independently of TCP streams.

Construction does not enumerate or open devices. Explicit `OpenRdmaDevice`
selects a named verbs device; `AcquireRdmaDomain` accepts an owning reference to
Expand All @@ -37,8 +39,97 @@ independent of worker shutdown. Partial initialization unwinds in reverse order.
PD and retain the domain until QP destruction. A registered MR can be used by
multiple QPs in the same PD; it belongs to neither a particular QP nor an IMM
(immediate-data) operation. Operation leases check both the QP's context and PD.
Connection management, QP posting and completion protocols are outside this
registration layer.
RC connections expose their actual protection domain for registration. A caller
must use that domain rather than assume a separately opened device handle has
the same PD. Data-posting and application completion protocols are outside the
connection API.

## RC ownership and connection flow

Each worker lazily owns one nonblocking CM channel and its ID registry. Each
listener and every connection accepted by it stay on that worker; a connection
returned by Accept survives listener close. Connect and accepted handshakes
have one absolute monotonic deadline covering address resolution, route
resolution and establishment. Submission backpressure does not restart it.
Arguments are copied before returning a lazy task. Private data is limited to
56 bytes in both directions; peer data is copied before acknowledging the CM
event, with the length reported by CM (which can include provider padding).
Listen binds immediately; Accept, Connect and Close are awaitable operations.

Librdmacm shares its routing contexts process-wide. RC therefore opens a private
verbs context per worker on the device selected by CM, and uses CM's external-QP
mode. CM negotiates the path/QP number; the worker creates only RC QPs in its own
PD and applies CM-provided INIT/RTR/RTS attributes, then completes the active
handshake with `rdma_establish`. It does not replace CM's context or attach its
QP to the CM ID. QP, PD, MR, CQ and the verbs asynchronous event stream belong to
the same private context, with exactly one worker reading that event stream.
Other workers neither consume its events nor migrate its connections.

Each private context shares one completion channel and fixed-capacity CQ.
Admission reserves the provider's actual send plus receive WR capacities for
each QP, including capacities rounded up by the provider. All send WRs are
signalled. A QP remains unconnected and has no posted work until its actual
budget fits; excess capacity fails admission instead of growing the CQ.
Reservations cover error/flush completions as well as successful completions,
and are returned only after QP destruction and CQ draining.

## Notification and failure flow

The CM channel, completion channel and private context's async-event fd are
nonblocking, with persistent one-shot readiness polls in the shared io_uring.
A CQ notification is requested before checking the CQ, including initial
activation. Each delivered CQ event is acknowledged, notification is requested
again, and the CQ is drained before rearming the fd poll. An event may remain
after its CQE was already consumed; it still needs acknowledgement and rearming.
Notifications stay enabled during busy and idle periods. This makes pending
work either visible in the CQ recheck or capable of waking an idle worker;
there is no separate polling thread or second ring.

CM events are copied and acknowledged before dispatch can destroy an ID.
Failures retain the CM event name/raw status or the failing verbs operation.
QP errors terminate the affected connection; shared CQ errors, device fatal
errors and device removal terminate every connection on that device context.
Port errors terminate connections using the affected port. Event objects are
acknowledged before this fanout can destroy their referenced resources. A
failed CM channel stops all endpoints on its worker and rejects new admission.

## Close and worker shutdown

Public handles are move-only and used on their owner worker. Engine, device,
listener and connection states use non-atomic strong/weak ownership. Every
handle and associated coroutine frame is accessed, moved and destroyed on its
creating thread, including any handles retained after worker shutdown; they
cannot be handed to another thread for cleanup. Exported PD/MR/context handles
retain their separate shared ownership contract. Destruction requests close;
awaiting Close joins retirement. Closing stops admission,
disconnects the CM association and puts the external QP into ERR. The
connection retains its timer tag until the timer's terminal completion, even
when cancellation has already been accepted. It leaves the ID registry before
resource destruction and releases its CQ budget only after the QP is destroyed
and the CQ drained. The shared channel polls reference the worker's event
engine/device state, never a listener or connection object.

Listener close terminates queued and handshaking children and wakes Accept;
already returned connections remain independent. One Accept and one Close
waiter per handle are supported. Waiters detach when their coroutine frames
are reclaimed. Failed device state is released only after its connections and
both fd polls have retired. A cancellation request or the cancel request's own
CQE is not a retirement boundary for the original fd poll.

Worker shutdown first stops RDMA admission and requests cancellation while
retaining every poll tag and descriptor. It then quiesces the shared ring.
Only after ring shutdown confirms that no kernel operation can refer to those
tags does RDMA retire requests that did not receive callbacks, acknowledge
remaining events, destroy QPs/CM IDs, and release channels/CQs. Coroutine frames
are reclaimed afterward. Exported registration/PD handles retain their own
context independently. `Server` quiesces the ring and reclaims coroutine frames
on the worker thread. Direct `Runtime` integrations must likewise call worker
`Shutdown()` and `DestroyDetachedTasks()` on that thread before it exits, and
release any application-owned RC handles and Tasks there. Unexpected destruction
failures fail closed rather than
release resources that may still be referenced by the provider.

## Registered memory

`RegisteredMemory` registers caller-owned backing memory with explicit access
permissions. Copies, registered slices, and operation leases retain the same
Expand Down Expand Up @@ -67,4 +158,6 @@ access and lifetime requirements.

Sources: `include/bycorf/io/rdma_backend.h`,
`include/bycorf/io/rdma_memory.h`, `src/io/rdma_memory.cpp`,
`include/bycorf/runtime/worker.h`, `src/runtime/worker.cpp`.
`include/bycorf/rdma/rc.h`, `src/rdma/rc.cpp`, `src/rdma/rc_engine.cpp`,
`src/rdma/rc_internal.h`, `include/bycorf/runtime/worker.h`,
`src/runtime/worker.cpp`, `src/io/io_uring_backend.cpp`.
13 changes: 10 additions & 3 deletions docs/architecture/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,11 @@ notifications, rechecks I/O, mailbox and scheduler work, completes the mailbox
wake-sequence handshake, and submits operations and peer wakes. After a final
idle check it waits on the shared io_uring. Runnable work, a stop request,
unsubmitted operations, or storage requiring polling prevent sleep. Network
deadlines and enabled connection-idle checks bound the wait.
deadlines and enabled connection-idle checks bound the wait. Optional RDMA CM,
CQ-notification and device-async polls are persistent wake sources on this same
ring; RDMA connection deadlines are absolute ring timeouts. CQ notifications
are always armed and rechecked, so RDMA needs no busy polling or additional
sleep-preparation phase.

Cross-worker submissions enter destination mailboxes and resume coroutines on
their original workers. The mailbox handshake makes racing publication either
Expand All @@ -78,8 +82,11 @@ an empty selection means every worker. Prepare sees the full runtime count,
while Run and FinalizeWorker execute only on selected workers. During shutdown,
workers first leave their loops, then quiesce I/O and reclaim coroutine frames
on their owning native threads. Service finalizers run after all worker frames
are reclaimed. A worker that fails initialization contributes to both teardown
barriers so initialized peers can finish. Temporary I/O submission shortages
are reclaimed. RDMA uses a two-phase shutdown: stop admission before the ring
is quiesced, then release retained polls/events and connection resources after
kernel access has ended. This also covers accepted operations whose callbacks
are suppressed by ring shutdown. A worker that fails initialization contributes
to both teardown barriers so initialized peers can finish. Temporary I/O submission shortages
are retried by the worker loop. A permanent submission error, notification
preparation or cleanup failure, or shared-ring wait failure stops peer workers
before entering those barriers and yields a nonzero exit. Services outlive
Expand Down
Loading