Skip to content

feat(rdma): add RC connections and notification-driven event loop - #40

Merged
thweetkomputer merged 5 commits into
eloqdata:mainfrom
thweetkomputer:feat/rdma-rc-event-loop
Nov 17, 2026
Merged

thweetkomputer merged 5 commits into
eloqdata:mainfrom
thweetkomputer:feat/rdma-rc-event-loop

Conversation

@thweetkomputer

@thweetkomputer thweetkomputer commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Add worker-owned RC listeners and connections so an idle worker can accept and establish RDMA connections through its existing io_uring. The optional RDMA build now supports Listen/Accept/Connect/Close, connection deadlines, and CM private data.

Implementation

  • Lazily open a private verbs context on the CM-selected device for each worker. Use CM external-QP mode to keep QP/PD/MR/CQ ownership consistent and route asynchronous device events to the owning worker.
  • Share a completion channel and fixed-capacity CQ per worker/device context. Reserve each QP's actual send and receive capacities before admission.
  • Integrate nonblocking CM, completion-channel, and device-event descriptors with the existing ring. Request CQ notifications before rechecking completions and acknowledge/rearm each delivered event.
  • Preserve concrete connection errors and terminate affected connections on device or shared-CQ failure.
  • Retain timer/poll tags and event owners until terminal completion or ring quiescence. Closing a listener preserves connections already returned by Accept.
  • Keep unaccepted connections in a flat hash map and use a borrowing inline ring queue for Accept readiness. Transfer the owning reference on Accept and clear readiness before releasing children on close.
  • Document resource ownership, notification flow, failure handling, and shutdown.

Ownership

RC Engine, Device, Listener and Connection state uses non-atomic LocalSharedPtr/LocalWeakPtr. Strong references still retain states across suspended coroutines; weak back-references avoid cycles and detect owner expiration. Handle access, movement and destruction, including coroutine-frame destruction after shutdown, stay on the creating thread. Debug builds check affinity.

Bycorf had no local reference-counting utility; the implementation follows Lavik's LocalSharedPtr thread contract and adds weak observation and incomplete-type support without a dependency on Lavik. The audit retains std::shared_ptr for cross-thread timer cancellation, TLS context sharing/state handoff, runtime startup coordination, and exported PD/MR/verbs-context ownership.

Validation

Latest ownership revision on ARM64:

  • GCC 13 Debug, RDMA disabled: ctest --test-dir /home/ubuntu/.cache/bycorf-localrefs/default --output-on-failure — 118/118 passed.
  • Clang 18 Debug, RDMA enabled: full suite passed with 149 executed tests and 7 device-dependent tests skipped. A subsequent lazy-Accept lifetime regression was added and the RC suite rerun: 24/24 passed.
  • /home/ubuntu/.cache/bycorf-localrefs/rdma/bycorf_rdma_rc_fake_tests — 24/24 passed.
  • ASan/UBSan: /home/ubuntu/.cache/bycorf-localrefs/asan/bycorf_unit_tests --gtest_filter='LocalSharedPtr*' — 7/7 passed; /home/ubuntu/.cache/bycorf-localrefs/asan/bycorf_rdma_rc_fake_tests — 24/24 passed.
  • clang-format 23.1.1 and whitespace checks passed.

The tests cover strong/weak lifetime, self-weak release during destruction, incomplete types, construction failure, foreign-thread misuse assertions, real worker-thread shutdown, lazy Accept after handle destruction, CM/CQ failure fanout and cancellation retirement. Earlier revisions passed 6 Soft-RoCE RC tests and the registration integration test; the current host has no Soft-RoCE device, so physical-provider checks were not rerun for this revision.

Scope and limitations

Numeric IPv4/IPv6 addresses only; CM private data is limited to 56 bytes. Data-posting APIs, automatic CQ growth, and application buffer pools are outside this change. RDMA remains disabled by default and independent of DPDK/SPDK. Real-device validation used Soft-RoCE; physical RNIC providers have not been tested.

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9b4905e5-87fb-43d8-8bc7-363964cdebd2
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@thweetkomputer
thweetkomputer merged commit 67a591e into eloqdata:main Nov 17, 2026
10 checks passed
@thweetkomputer
thweetkomputer deleted the feat/rdma-rc-event-loop branch November 17, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant