Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/architecture/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ and asynchronous I/O. Abseil supplies status types and spdlog supplies logging.
| Module | Responsibility | Document |
|---|---|---|
| Runtime | Worker lifecycle, coroutine scheduling, cross-worker and foreign ingress | [Runtime](runtime.md) |
| I/O | io_uring, optional SPDK storage, shared DPDK environment | [I/O backends](io.md) |
| I/O | Worker-owned shared io_uring, network/storage adapters, optional SPDK storage, shared DPDK environment | [I/O backends](io.md) |
| Networking | Listeners, streams, services, TLS, optional DPDK/FreeBSD IPv4 TCP | [Networking](networking.md) |
| RPC | Framed request/reply protocol above streams | [RPC](rpc.md) |

Expand Down
28 changes: 22 additions & 6 deletions docs/architecture/io.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,27 @@ allocation or runtime startup fails. This preserves DMA allocator identity,
registered-resource ownership and native TCP state. Unsupported capabilities
fail explicitly rather than falling back.

`NetBackend` owns the worker's single `IoUringBackend`. Its optional DPDK adapter
borrows this object; kernel sockets, storage, timers and MSG_RING use the same
submission/completion queues. Network-specific operations dispatch to the
selected backend. The worker drives completion polling and batching, and does
not create a second ring or worker pool for either accelerator.
`Worker` owns one `IoUringBackend`, shared by its private CRTP facades
`NetBackend<Worker>` and `StorageBackend<Worker>`. The facades obtain the ring
through the owner contract rather than receiving it on each submission or
storing a separate pointer. Kernel sockets, storage, timers and MSG_RING use the same
submission/completion queues. `NetBackend` selects kernel networking or its
optional DPDK adapter; `StorageBackend` selects kernel storage or its embedded
SPDK backend. SPDK default construction acquires no device resources; only the
selected backend is initialized. Storage dispatch uses its worker-local
initialized state, which remains active through shutdown completion draining.
Storage operations require a successfully initialized worker and stop before
its backend shutdown completes. The worker forwards storage operations without
selecting SPDK itself. Network event-loop polling and submission advance the
shared ring; separate storage polling advances only SPDK. Pending polled storage prevents
parking, while kernel storage uses the shared ring's wake/completion path.

The worker initializes the ring before either adapter and stops storage, then
networking, before shutting down the ring. Kernel storage registrations live
until ring shutdown quiesces kernel access. The ring is the first base subobject, so both
facades are destroyed before it; neither creates a second ring or worker pool. SPDK callbacks
resolve their backend through the storage adapter using the owning worker's TLS,
including shutdown drains.

Each io_uring backend owns a FIFO for accepted operations awaiting submission
space. Successful admission does not mean the kernel has received the request.
Expand Down Expand Up @@ -117,6 +133,6 @@ supports one runtime per process. Network I/O and descriptors stop on worker
shutdown, but the complete BSD kernel subsystem teardown is not implemented.

Sources: `include/bycorf/io/backend_options.h`, `src/io/backend_options.cpp`,
`include/bycorf/io/net_backend.h`, `src/io/io_uring_backend.cpp`,
`include/bycorf/io/net_backend.h`, `include/bycorf/io/storage_backend.h`, `src/io/io_uring_backend.cpp`,
`src/io/storage.cpp`, `src/io/spdk_storage.cpp`, `src/io/dpdk_environment.cpp`,
`cmake/Dpdk.cmake`, `cmake/Freebsd.cmake`, `cmake/build_freebsd.py`.
6 changes: 4 additions & 2 deletions docs/architecture/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,10 @@ inherited mask. Every thread completes affinity setup before any
worker main function runs. An affinity or thread-launch failure prevents the
worker main functions from entering partial service initialization.

A `Worker` owns its coroutine queues, connections, I/O backend, completion
handling, and optional storage poller. The scheduler bounds foreground and
A `Worker` owns its coroutine queues, connections, shared kernel I/O backend,
network and storage facades, and completion handling. The facades are private
CRTP bases sharing the worker's kernel backend; storage-specific polling is
encapsulated by `StorageBackend`. The scheduler bounds foreground and
background coroutine work and polls external completions between rounds.
Workers submit queued foreground I/O and peer wakes before a maintenance slice,
then submit any work queued by that slice before advancing to the next round.
Expand Down
2 changes: 1 addition & 1 deletion include/bycorf/io/completion.h
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ class IoCompletion {
std::coroutine_handle<> awaiting_{};

private:
friend class SpdkStorageBackend;
friend class SpdkBackend;

// Status-only storage completions need the expected successful byte count
// (zero for flush). Keep it with the caller-owned tag so submitted commands
Expand Down
3 changes: 2 additions & 1 deletion include/bycorf/io/dpdk_backend.h
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ class DpdkBackend {
// Release peers waiting for worker 0 if any worker fails during startup.
static void AbortStartup(const absl::Status& reason);
static void StopRuntime();
absl::Status Init(const IoBackendOptions&, Worker*, int wake_fd);
// The worker initializes the borrowed ring before attaching networking.
absl::Status Init(const IoBackendOptions&, Worker*);
void Shutdown();
bool Poll();
bool Wait(int timeout_ms);
Expand Down
125 changes: 50 additions & 75 deletions include/bycorf/io/net_backend.h
Original file line number Diff line number Diff line change
Expand Up @@ -25,94 +25,106 @@
#endif

namespace bycorf {
// One kernel ring per worker, shared by kernel sockets, storage, timers and
// wakeups. An optional DPDK adapter borrows that ring. Backend selection stays
// fixed after Init, so calls use a predictable branch without virtual dispatch.
// CRTP network facade. Owner derives non-virtually from this specialization and
// grants it access to KernelBackend(), which returns the owner's shared ring.
// Downcasting computes a fixed address adjustment; no owner/ring pointer is
// stored here. Owner must keep the ring alive through adapter destruction.
template <typename Owner>
class NetBackend {
public:
absl::Status Init(const IoBackendOptions& options, Worker* worker,
int wake_fd) {
// Initialize networking after the owner initializes the shared ring. DPDK
// borrows that ring for its adapter lifetime.
absl::Status Init(const IoBackendOptions& options, Worker* worker) {
#if BYCORF_KERNEL_BYPASS
if (DpdkNetworkEnabled()) {
dpdk_.emplace(kernel_);
return dpdk_->Init(options, worker, wake_fd);
dpdk_.emplace(Kernel());
return dpdk_->Init(options, worker);
}
#endif
return kernel_.Init(options, worker, wake_fd);
(void)options;
(void)worker;
return absl::OkStatus();
}
// Release DPDK peers waiting for startup if shared-ring initialization fails.
void AbortStartup(const absl::Status& status) {
#if BYCORF_KERNEL_BYPASS
if (DpdkNetworkEnabled()) DpdkBackend::AbortStartup(status);
#else
(void)status;
#endif
}
// Stop networking while the borrowed ring is still alive.
void Shutdown() {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) {
dpdk_->Shutdown();
return;
}
#endif
kernel_.Shutdown();
}
int WakeHandle() const noexcept { return kernel_.WakeHandle(); }
absl::Status SubmitSend(const RegisteredFile& file,
std::span<const std::byte> buffer,
IoCompletion* tag) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->SubmitSend(file, buffer, tag);
#endif
return kernel_.SubmitSend(file, buffer, tag);
return Kernel().SubmitSend(file, buffer, tag);
}
absl::Status SubmitSendMsg(const RegisteredFile& file, const msghdr* message,
IoCompletion* tag) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->SubmitSendMsg(file, message, tag);
#endif
return kernel_.SubmitSendMsg(file, message, tag);
return Kernel().SubmitSendMsg(file, message, tag);
}
absl::Status SubmitAcceptMultishot(int listen_fd, IoCompletion* tag) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->SubmitAcceptMultishot(listen_fd, tag);
#endif
return kernel_.SubmitAcceptMultishot(listen_fd, tag);
return Kernel().SubmitAcceptMultishot(listen_fd, tag);
}
absl::Status SubmitConnect(int fd, const sockaddr* address,
socklen_t address_length, IoCompletion* tag) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->SubmitConnect(fd, address, address_length, tag);
#endif
return kernel_.SubmitConnect(fd, address, address_length, tag);
return Kernel().SubmitConnect(fd, address, address_length, tag);
}
absl::Status SubmitCancel(IoCompletion* target) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->SubmitCancel(target);
#endif
return kernel_.SubmitCancel(target);
return Kernel().SubmitCancel(target);
}
// Cancel kernel socket operations that have not acquired their fd reference
// yet. Native DPDK handles must be excluded by the caller.
void CancelUnsubmittedForFd(int fd) noexcept {
kernel_.CancelUnsubmittedForFd(fd);
Kernel().CancelUnsubmittedForFd(fd);
}
absl::Status StartRecvMultishot(Connection* connection) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->StartRecvMultishot(connection);
#endif
return kernel_.StartRecvMultishot(connection);
return Kernel().StartRecvMultishot(connection);
}
// Null tag requests cancellation without a separate completion consumer.
absl::Status SubmitCancelRecv(Connection* connection, IoCompletion* tag) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->SubmitCancelRecv(connection, tag);
#endif
return kernel_.SubmitCancelRecv(connection, tag);
return Kernel().SubmitCancelRecv(connection, tag);
}
absl::Status StartPeerDisconnectPoll(Connection* connection) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->StartPeerDisconnectPoll(connection);
#endif
return kernel_.StartPeerDisconnectPoll(connection);
return Kernel().StartPeerDisconnectPoll(connection);
}
absl::Status CancelPeerDisconnectPoll(Connection* connection) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->CancelPeerDisconnectPoll(connection);
#endif
return kernel_.CancelPeerDisconnectPoll(connection);
return Kernel().CancelPeerDisconnectPoll(connection);
}
std::span<const std::byte> ViewRecvBuffer(const Connection* connection,
std::uint16_t buffer_id,
Expand All @@ -122,84 +134,47 @@ class NetBackend {
if (dpdk_)
return dpdk_->ViewRecvBuffer(connection, buffer_id, offset, length);
#endif
return kernel_.ViewRecvBuffer(connection, buffer_id, offset, length);
return Kernel().ViewRecvBuffer(connection, buffer_id, offset, length);
}
void ReleaseRecvBuffer(Connection* connection, std::uint16_t buffer_id) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->ReleaseRecvBuffer(connection, buffer_id);
#endif
return kernel_.ReleaseRecvBuffer(connection, buffer_id);
}
absl::Status RegisterFixedFiles(unsigned count) {
return kernel_.RegisterFixedFiles(count);
}
absl::Status RegisterBuffers(std::span<const iovec> buffers) {
return kernel_.RegisterBuffers(buffers);
}
void UnregisterStorageResources() {
return kernel_.UnregisterStorageResources();
}
absl::Status SubmitOpenDirect(std::string_view path, int flags, mode_t mode,
FixedFile file, IoCompletion* tag) {
return kernel_.SubmitOpenDirect(path, flags, mode, file, tag);
}
absl::Status SubmitCloseDirect(FixedFile file, IoCompletion* tag) {
return kernel_.SubmitCloseDirect(file, tag);
}
absl::Status SubmitReadFixed(FixedFile file, FixedBuffer buffer,
std::uint64_t offset, IoCompletion* tag) {
return kernel_.SubmitReadFixed(file, buffer, offset, tag);
}
absl::Status SubmitRead(FixedFile file, std::span<std::byte> buffer,
std::uint64_t offset, IoCompletion* tag) {
return kernel_.SubmitRead(file, buffer, offset, tag);
}
absl::Status SubmitWrite(FixedFile file, std::span<const std::byte> buffer,
std::uint64_t offset, IoCompletion* tag) {
return kernel_.SubmitWrite(file, buffer, offset, tag);
}
absl::Status SubmitWriteFixed(FixedFile file, FixedBuffer buffer,
std::uint64_t offset, IoCompletion* tag) {
return kernel_.SubmitWriteFixed(file, buffer, offset, tag);
}
absl::Status SubmitFdatasync(FixedFile file, IoCompletion* tag) {
return kernel_.SubmitFdatasync(file, tag);
}
absl::Status SubmitTimeout(const __kernel_timespec& timeout,
IoCompletion* tag) {
return kernel_.SubmitTimeout(timeout, tag);
}
absl::Status SubmitPoll(int fd, unsigned events, IoCompletion* tag) {
return kernel_.SubmitPoll(fd, events, tag);
return Kernel().ReleaseRecvBuffer(connection, buffer_id);
}
absl::Status Submit() {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->Submit();
#endif
return kernel_.Submit();
return Kernel().Submit();
}
bool Poll() {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->Poll();
#endif
return kernel_.Poll();
return Kernel().Poll();
}
// The shared kernel ring can retain work even when DPDK serves networking.
bool HasPending() const noexcept { return kernel_.HasPending(); }
bool Wait(int timeout_ms) {
#if BYCORF_KERNEL_BYPASS
if (dpdk_) return dpdk_->Wait(timeout_ms);
#endif
return kernel_.Wait(timeout_ms);
return Kernel().Wait(timeout_ms);
}
void WakeRemote(int peer_ring_fd) noexcept {
return kernel_.WakeRemote(peer_ring_fd);
}
void WakeSelf() noexcept { return kernel_.WakeSelf(); }

private:
// Declaration order ensures the borrowing adapter dies before its ring.
IoUringBackend kernel_;
// Only the declared owner may construct the facade; a standalone base
// would make the CRTP downcast invalid. Never access Owner in base teardown.
friend Owner;
NetBackend() = default;
~NetBackend() = default;

IoUringBackend& Kernel() noexcept {
return static_cast<Owner&>(*this).KernelBackend();
}
const IoUringBackend& Kernel() const noexcept {
return static_cast<const Owner&>(*this).KernelBackend();
}

#if BYCORF_KERNEL_BYPASS
// Inline the small adapter: packet/stream submissions already follow its
// implementation pointer, so a separate heap adapter adds a dependent load
Expand Down
15 changes: 9 additions & 6 deletions include/bycorf/io/spdk_storage.h
Original file line number Diff line number Diff line change
Expand Up @@ -105,16 +105,19 @@ void FreeStorageBuffer(void* buffer, std::size_t alignment) noexcept;
// capacity. The backend owns waiting request metadata; each tag represents one
// unfinished request. Submit methods never dispatch completions inline. Poll
// drives both device completions and deferred submissions.
class SpdkStorageBackend {
class SpdkBackend {
public:
SpdkStorageBackend() = default;
SpdkStorageBackend(const SpdkStorageBackend&) = delete;
SpdkStorageBackend& operator=(const SpdkStorageBackend&) = delete;
~SpdkStorageBackend();
SpdkBackend() = default;
SpdkBackend(const SpdkBackend&) = delete;
SpdkBackend& operator=(const SpdkBackend&) = delete;
~SpdkBackend();

// Initialize only the storage backend embedded in worker.
absl::Status Init(Worker* worker);
void Shutdown();
// Owner-thread state: true after successful Init until Shutdown finishes
// draining, including callbacks that submit more work during that drain.
bool IsInitialized() const noexcept { return worker_ != nullptr; }

absl::Status RegisterFixedFiles(unsigned count);
absl::Status RegisterBuffers(std::span<const iovec> buffers);
Expand Down Expand Up @@ -182,6 +185,7 @@ class SpdkStorageBackend {
OpenFile* Lookup(FixedFile file);
static void CompleteAsync(void* context, const spdk_nvme_cpl* completion);

// Binding is also the initialized state; retain it through shutdown drains.
Worker* worker_ = nullptr;
std::vector<OpenFile> files_;
std::vector<ControllerChannel> channels_;
Expand All @@ -190,7 +194,6 @@ class SpdkStorageBackend {
// Completion callbacks may open/close files and invalidate channel indices.
std::size_t channel_generation_ = 0;
std::size_t outstanding_ = 0;
bool initialized_ = false;
};

#endif // BYCORF_KERNEL_BYPASS
Expand Down
Loading