Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -491,7 +491,8 @@ if (BYCORF_BUILD_TESTS)
tests/connection_storage_test.cpp
tests/http_service_test.cpp
tests/tcp_buffered_read_test.cpp
tests/io_uring_backpressure_test.cpp)
tests/io_uring_backpressure_test.cpp
tests/worker_wait_test.cpp)
target_link_libraries(bycorf_runtime_tests PRIVATE bycorf::core GTest::gtest_main)
bycorf_discover_tests(bycorf_runtime_tests 90 "runtime;net;io_uring")

Expand Down Expand Up @@ -525,7 +526,7 @@ if (BYCORF_BUILD_TESTS)
${BYCORF_DPDK_COMPILE_FLAGS})
foreach(symbol IN ITEMS rte_eth_dev_configure rte_eth_dev_info_get
rte_eth_dev_rx_intr_ctl_q_get_fd rte_eth_dev_rx_intr_enable
rte_eth_dev_rx_intr_disable)
rte_eth_dev_rx_intr_disable bycorf_bsd_deadline_ns)
target_link_options(bycorf_dpdk_adaptive_tests PRIVATE
"LINKER:--wrap=${symbol}")
endforeach()
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ results.
|---|---|
| `InlineRingQueue`, `ReceivedBufferQueue` | Queue types and inline capacities, moves, FIFO order, wrapping, growth, and shrinking |
| `CrossCoreDrain`, `CrossCoreDrainState` | Shared sender quotas across message kinds, overflow fairness, bitmap wrapping, and concurrent publication without lost work |
| `WorkerWait` | Publication between idle rechecks and blocking, MSG_RING/eventfd wakes, stop races, and runnable work preventing sleep |
| `WorkerScheduler` | Foreground/background FIFO admission, deferred turns, and reentrant local/remote work across queue relocation |
| `TaskTransfer` | Bounded native stack for sequential and nested awaits, asynchronous completion, cancellation ownership and reentrant callbacks |
| `RuntimeStartup`, `TcpServiceStop`, `Timer`, `Connect` | Early shutdown, timers and cancellation, loopback TCP, refused connections, deadlines, address validation, and retirement of losing deadlines |
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ and asynchronous I/O. Abseil supplies status types and spdlog supplies logging.

| Module | Responsibility | Document |
|---|---|---|
| Runtime | Worker lifecycle, coroutine scheduling, cross-worker and foreign ingress | [Runtime](runtime.md) |
| Runtime | Worker lifecycle, coroutine scheduling, sleep coordination, cross-worker and foreign ingress | [Runtime](runtime.md) |
| I/O | Worker-owned shared io_uring, network/storage adapters, optional SPDK storage, shared DPDK environment | [I/O backends](io.md) |
| Networking | Listeners, streams, services, TLS, optional DPDK/FreeBSD IPv4 TCP | [Networking](networking.md) |
| RPC | Framed request/reply protocol above streams | [RPC](rpc.md) |
Expand Down
66 changes: 30 additions & 36 deletions docs/architecture/io.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ limitations under the License.

# I/O backends

Every worker retains one io_uring instance for ordinary file I/O, kernel
sockets, completion delivery, timers, and cross-worker MSG_RING wakeups.
Registered storage buffers and network receive buffers have separate
lifecycles. The default network backend uses Linux TCP and provided-buffer
multishot receives, with a per-connection one-shot fallback.
`Worker` owns one `IoUringBackend` for kernel sockets, file I/O, completion
delivery, timers, and MSG_RING/eventfd wakeups. Its `NetBackend<Worker>` and
`StorageBackend<Worker>` facades share that ring. Network receive buffers and
registered storage buffers have separate lifecycles. Linux TCP uses
provided-buffer multishot receives with a per-connection one-shot fallback.

`BYCORF_KERNEL_BYPASS` is the single build option for DPDK networking and SPDK
storage. It defaults to off; enabling it compiles and links both capabilities.
Expand All @@ -32,36 +32,30 @@ allocation or runtime startup fails. This preserves DMA allocator identity,
registered-resource ownership and native TCP state. Unsupported capabilities
fail explicitly rather than falling back.

`Worker` owns one `IoUringBackend`, shared by its private CRTP facades
`NetBackend<Worker>` and `StorageBackend<Worker>`. The facades obtain the ring
through the owner contract rather than receiving it on each submission or
storing a separate pointer. Kernel sockets, storage, timers and MSG_RING use the same
submission/completion queues. `NetBackend` selects kernel networking or its
optional DPDK adapter; `StorageBackend` selects kernel storage or its embedded
SPDK backend. SPDK default construction acquires no device resources; only the
selected backend is initialized. Storage dispatch uses its worker-local
initialized state, which remains active through shutdown completion draining.
Storage operations require a successfully initialized worker and stop before
its backend shutdown completes. The worker forwards storage operations without
selecting SPDK itself. Network event-loop polling and submission advance the
shared ring; separate storage polling advances only SPDK. Pending polled storage prevents
parking, while kernel storage uses the shared ring's wake/completion path.

The worker initializes the ring before either adapter and stops storage, then
networking, before shutting down the ring. Kernel storage registrations live
until ring shutdown quiesces kernel access. The ring is the first base subobject, so both
facades are destroyed before it; neither creates a second ring or worker pool. SPDK callbacks
resolve their backend through the storage adapter using the owning worker's TLS,
including shutdown drains.
`NetBackend` selects kernel networking or the DPDK adapter; `StorageBackend`
selects kernel storage or SPDK. Only selected adapters acquire resources.
The worker explicitly polls the shared ring, native network work, and polled
storage. It flushes native network output before submitting the shared ring.
Network polling and flushing advance only DPDK work; storage polling advances
only SPDK. Kernel TCP and storage progress through the worker's ring calls.
Network adapters supply notification preparation, idle checks,
deadlines, and cleanup to the [worker sleep protocol](runtime.md#sleep-and-wakeups).
The worker enters the ring's blocking wait; adapters never block independently.

The ring outlives both adapters. The worker initializes it first, stops storage
and networking in that order, then shuts it down. Storage dispatch remains
valid through shutdown completion draining; SPDK callbacks use the owning
worker's TLS to reach its storage adapter. Kernel registrations remain live
until ring shutdown quiesces kernel access.

Each io_uring backend owns a FIFO for accepted operations awaiting submission
space. Successful admission does not mean the kernel has received the request.
The backend retains its prepared descriptor; the caller retains the completion
tag, referenced buffers and parameter storage, and any registered resources
until the operation's terminal completion. Submission and completion polling
advance the queue on its owner worker, without dispatching completions inside
an operation's submission call. The event loop advances queued work before
parking, including when no earlier request can produce a completion.
an operation's submission call. Unsubmitted operations veto worker sleep even
when no earlier request can produce a completion.

Cancellation preserves this ownership boundary: a queued operation retires
through deferred completion, while an issued operation remains live until its
Expand All @@ -82,14 +76,14 @@ unfinished storage operation has a distinct tag, which retains its expected
successful result; the caller keeps the tag and buffers alive until completion.
The completion callback may reuse or destroy the tag. Polling and shutdown
drains retain the owning worker's thread-local context for completion dispatch.
Deferred commands participate in outstanding-I/O accounting, preventing
premature sleep, close or shutdown. Submission errors that cannot make progress
through a completion are reported to the caller.
The io_uring selection uses ordinary aligned buffers and kernel file/block I/O.
SPDK completion polling is inactive when storage uses io_uring; in-flight SPDK
storage prevents sleeping before its completions can be polled. As with the
NVMe FLUSH command, SPDK durability barriers cover previously completed writes;
callers await data writes before requesting a barrier. Controllers that report
Accepted and deferred SPDK commands require polling and veto worker sleep
until they retire; they also participate in close and shutdown accounting.
Submission errors that cannot make progress through a completion are reported
to the caller. Kernel storage uses ordinary aligned buffers and the shared
ring's completion path without SPDK polling.

As with the NVMe FLUSH command, SPDK durability barriers cover previously
completed writes; callers await data writes before requesting a barrier. Controllers that report
no volatile write cache already make completed writes durable, so their no-op
FLUSH completes through the worker's deferred completion queue.

Expand Down
28 changes: 14 additions & 14 deletions docs/architecture/networking.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,15 +157,16 @@ are bounded: overflow and partial device TX free unaccepted packets, leaving
native TCP responsible for retransmission. This is a functional integration
boundary, not a zero-copy or tuned packet allocator.

## Polling and sleep
## DPDK notifications

Poll mode keeps workers running without arming RX interrupts. Adaptive mode
can wait on the same io_uring used for worker messages. A queue owner enables
its RX interrupt, registers the notification descriptor with io_uring, and
rechecks packet and software queues before sleeping. Packet publication either
prevents sleep or sends a MSG_RING wake to the parked destination. The next
BSD timer deadline bounds the wait; an arriving packet does not need to wait
for a timer tick. RX interrupts are disabled on returning to polling.
DPDK participates in the [worker sleep protocol](runtime.md#sleep-and-wakeups).
Explicit poll mode vetoes sleep without arming interrupts. Adaptive preparation
marks the software lane parked and, for a queue owner, enables RX interrupts
and registers the notification descriptor with the shared io_uring. Packet
publication either appears in the RX/TX queue recheck or sends MSG_RING to the
parked destination. The FreeBSD timer deadline bounds the worker's wait.
Cleanup clears the lane's parked state and disables any interrupt enabled by
that attempt, including on early exit or partial preparation failure.

A TAP virtual device provides a readable packet descriptor instead of a
physical interrupt eventfd. The notification code preserves that distinction:
Expand All @@ -175,12 +176,11 @@ after a timer or mailbox wake does not indicate lost interrupt support. If the
PMD cannot provide the required notification capability on every RX queue,
adaptive startup fails before services start. Configuration, notification
registration, and interrupt-arm failures never silently select polling; the
error identifies an explicit `BYCORF_DPDK_MODE=poll` override. A notification
failure during a wait terminates the worker loop and stops peer workers before
teardown barriers, producing a nonzero server exit. Virtual-device tests cover
this wake protocol but do not validate physical NIC interrupt behavior or predict kernel-bypass throughput. The TAP
PMD additionally uses Linux realtime signals for its internal RX trigger in
both modes; this is separate from Bycorf's worker sleep protocol.
error identifies an explicit `BYCORF_DPDK_MODE=poll` override. Notification
completion and interrupt-cleanup failures use the runtime's worker-failure
shutdown path. The TAP PMD uses Linux realtime signals for its internal RX
trigger in both modes. Virtual-device tests cover the notification protocol, but do not
validate physical NIC interrupt behavior or predict kernel-bypass throughput.

Sources: `include/bycorf/net/connection.h`, `src/net/tcp_service.cpp`, `src/net/tcp_listener.cpp`,
`src/net/tcp_stream.cpp`, `src/net/socket_ops.cpp`, `src/io/dpdk_backend.cpp`,
Expand Down
70 changes: 46 additions & 24 deletions docs/architecture/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,16 @@ inherited mask. Every thread completes affinity setup before any
worker main function runs. An affinity or thread-launch failure prevents the
worker main functions from entering partial service initialization.

A `Worker` owns its coroutine queues, connections, shared kernel I/O backend,
network and storage facades, and completion handling. The facades are private
CRTP bases sharing the worker's kernel backend; storage-specific polling is
encapsulated by `StorageBackend`. The scheduler bounds foreground and
background coroutine work and polls external completions between rounds.
Workers submit queued foreground I/O and peer wakes before a maintenance slice,
then submit any work queued by that slice before advancing to the next round.
Coroutines cooperate: a long non-suspending handler can delay all work on its
worker. Worker-local synchronization and cross-worker synchronization have
separate ownership contracts.
A `Worker` owns its coroutine queues, connections, completion handling, and
[shared I/O backend and adapters](io.md). Each round polls completions and
runs bounded foreground and background coroutine work. The worker polls the
shared ring before native network work and polled storage. Before maintenance
and at the end of the round, it queues peer wakes, flushes native network
output, and submits the shared ring. Idle rechecks use the same ordering;
facades do not poll or submit the ring on the worker's behalf. Coroutines
cooperate: a non-suspending handler can
delay all work on its worker. Worker-local and cross-worker synchronization
have separate ownership contracts.

A worker resumes a ready coroutine directly. `Task` enters its child and
returns to its parent continuation by symmetric transfer, within the same
Expand All @@ -46,12 +46,32 @@ the selected toolchain and build configuration. Awaited child frames remain
owned by the awaiting `Task` object, including when shutdown destroys a
suspended root and its child chain.

Cross-worker submissions enter destination mailboxes and resume waiting
coroutines on their original workers. Workers use io_uring MSG_RING to wake a
peer; foreign ingress and process control use the pre-created eventfds.
`ForeignExecutor` is non-owning and is valid only while its runtime accepts
work. Stop closes ingress and wakes workers through descriptors that remain
alive until after their threads have joined.
## Sleep and wakeups

The worker alone decides when its thread can block. It prepares source
notifications, rechecks I/O, mailbox and scheduler work, completes the mailbox
wake-sequence handshake, and submits operations and peer wakes. After a final
idle check it waits on the shared io_uring. Runnable work, a stop request,
unsubmitted operations, or storage requiring polling prevent sleep. Network
deadlines and enabled connection-idle checks bound the wait.

Cross-worker submissions enter destination mailboxes and resume coroutines on
their original workers. The mailbox handshake makes racing publication either
prevent sleep or leave a persistent wake: workers use MSG_RING, while foreign
ingress and process control use pre-created eventfds. Consuming a completion
can make work runnable or observe a stop request, so rechecking must account
for both before blocking.

Every wait attempt clears its mailbox parked state and ends source notification
preparation, including attempts abandoned before blocking. Source cleanup
releases per-attempt state; outstanding notification polls retain their tags
and descriptors through completion or ring shutdown.

`ForeignExecutor` is non-owning and valid while its runtime accepts work. Stop
closes ingress and wakes workers through descriptors that remain alive until
after their threads have joined.

## Shutdown and integration

`Server` starts each registered service on its selected global worker IDs;
an empty selection means every worker. Prepare sees the full runtime count,
Expand All @@ -60,23 +80,25 @@ workers first leave their loops, then quiesce I/O and reclaim coroutine frames
on their owning native threads. Service finalizers run after all worker frames
are reclaimed. A worker that fails initialization contributes to both teardown
barriers so initialized peers can finish. Temporary I/O submission shortages
are retried by the worker loop. A permanent submission error or unexpected
backend wait failure stops peer workers before entering those barriers and
yields a nonzero exit. Services outlive the server.
are retried by the worker loop. A permanent submission error, notification
preparation or cleanup failure, or shared-ring wait failure stops peer workers
before entering those barriers and yields a nonzero exit. Services outlive
the server.

With DPDK networking, runtime preparation configures the shared port before
launch. Worker 0 initializes the native FreeBSD kernel services; remaining
workers attach separate VNETs in serialized startup. The existing worker loop
workers attach separate VNETs in serialized startup. The worker loop
runs packet input, TCP timers, and deferred BSD tasks. EAL may have control or
interrupt helper threads; it does not launch Bycorf network polling workers.
Worker-affine BSD sockets close and EAL thread registrations detach before
join; the runtime then releases the port and software packet queues.

Sources: `src/runtime/runtime.cpp`, `src/runtime/worker.cpp`,
`include/bycorf/runtime/task.h`, `include/bycorf/runtime/cross_core.h`, `src/runtime/foreign_executor.cpp`,
`src/net/server.cpp`.

Active registered streams are counted across all workers and runtimes in the
process. The atomic snapshot includes outgoing control connections and is
updated only at connection open/close, so monitoring can read it without
scheduling work on a control worker.

Sources: `src/runtime/runtime.cpp`, `src/runtime/worker.cpp`,
`src/runtime/worker_wait.h`, `include/bycorf/runtime/task.h`,
`include/bycorf/runtime/cross_core.h`, `src/runtime/foreign_executor.cpp`,
`src/net/server.cpp`.
18 changes: 16 additions & 2 deletions include/bycorf/io/dpdk_backend.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

#include <memory>

#include "absl/status/statusor.h"
#include "bycorf/io/io_uring_backend.h"

namespace bycorf {
Expand Down Expand Up @@ -53,9 +54,22 @@ class DpdkBackend {
// The worker initializes the borrowed ring before attaching networking.
absl::Status Init(const IoBackendOptions&, Worker*);
void Shutdown();
// Drive native packet/TCP work only; the worker separately polls io_uring.
bool Poll();
bool Wait(int timeout_ms);
absl::Status Submit();
// Owner-thread sleep protocol. Every PrepareWait call, including a partial
// failure or a polling veto, must be paired with FinishWait before retrying.
// True allows the worker to recheck all sources; false requires polling.
absl::StatusOr<bool> PrepareWait();
// Call after Poll: checks residual software work and notification health.
bool CanWait() const;
// Bound the caller's timeout by the current BSD deadline (-1 is infinite).
int WaitTimeout(int timeout_ms) const;
// Release this attempt's parked state and RX interrupt, even if no wait ran.
// Outstanding notification polls retain their tag/fd until completion.
// waited records entry to the shared ring wait for diagnostic accounting.
absl::Status FinishWait(bool waited);
// Flush native TX work without submitting the borrowed kernel ring.
void Flush();
absl::Status SubmitSend(const RegisteredFile&, std::span<const std::byte>,
IoCompletion*);
absl::Status SubmitConnect(int fd, const sockaddr*, socklen_t, IoCompletion*);
Expand Down
8 changes: 6 additions & 2 deletions include/bycorf/io/io_uring_backend.h
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,12 @@ class IoUringBackend {

// Event loop.
absl::Status Submit();
bool Poll(); // reap completions, dispatch, then re-arm recvs
bool Wait(int timeout_ms); // block for a completion, dispatch, drain
bool Poll(); // reap completions, dispatch, then re-arm recvs
// The worker coordinates source notification arming and idle rechecks before
// calling this sole blocking primitive. Pending submissions veto blocking;
// an existing CQE returns immediately. Dispatch can enqueue runnable work
// or consume a stop notification, so the caller must re-enter scheduling.
bool Wait(int timeout_ms);
// Software work or SQEs still requiring an enter must prevent worker sleep.
bool HasPending() const noexcept;

Expand Down
Loading