From f050116d2d1447eca06105ab0e80328a7228aec6 Mon Sep 17 00:00:00 2001 From: JackDoan Date: Wed, 26 Aug 2026 14:20:06 -0500 Subject: [PATCH 1/2] Document offload and CPU pinning config --- docs/config/listen.mdx | 33 ++++++++++++++++++++ docs/config/routines.mdx | 4 +++ docs/config/tun.mdx | 66 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 103 insertions(+) diff --git a/docs/config/listen.mdx b/docs/config/listen.mdx index c1d2b425..d4d6fafc 100644 --- a/docs/config/listen.mdx +++ b/docs/config/listen.mdx @@ -37,6 +37,10 @@ nodes. Using 0 on lighthouses and relay hosts will likely lead to connectivity i Sets the max number of packets to pull from the kernel for each syscall (under systems that support `recvmmsg`). +When [`listen.udp_offloads`](#listenudp_offloads) has enabled GRO (Linux kernel 5.10+), each receive slot is sized for a +full 64KiB coalesced superpacket, so the receive scratch memory is `batch × 64KiB` per listening socket (about 4MiB per +routine at the default of 64). Lower this value to trade peak per-syscall throughput for memory on constrained hosts. + ## listen.read_buffer, listen.write_buffer Configure socket buffers for the udp side (outside), leave unset to use the system defaults. Values will be doubled by @@ -91,3 +95,32 @@ ip rule add not from all fwmark 4242 lookup 4242 ip rule add from all lookup main suppress_prefixlength 0 ip route add default dev nebula1 via table 4242 ``` + +## listen.rebind_on_network_change + +Default: True +Added in v1.11.0 + +This option is only supported on macOS and is not reloadable. + +macOS scopes the UDP socket to the interface it was created on, so moving between networks (wifi to wired, office to +home) leaves Nebula sending out an interface that no longer has a route. When enabled, Nebula watches the routing socket +and rebinds the listener once the change settles, so devices moving between networks recover without waiting for dead +tunnel detection. iOS does not use this option, the host app drives the same rebind itself. + +## listen.udp_offloads + +Default: False +Added in v1.12.0 + +This option is only supported on Linux and is not reloadable. + +When enabled, Nebula attempts to enable UDP Generic Segmentation Offload (GSO, Linux kernel 4.18+) and Generic Receive +Offload (GRO, Linux kernel 5.10+) on its UDP socket(s). With GSO, batches of packets bound for the same host are handed +to the kernel as a single "superpacket" along with the size to split it back into individual datagrams, reducing +per-packet syscall and network stack overhead on the send path. With GRO, the kernel coalesces bursts of datagrams +arriving from the same host into a single large read. Support for each feature is probed when the socket is created, and +Nebula falls back to standard sends and receives on kernels that lack it. + +Enabling GRO increases the fixed memory used by the receive path, see [`listen.batch`](#listenbatch) for details. For +the biggest throughput gains, pair this option with [`tun.use_offloads`](/docs/config/tun/#tunuse_offloads). diff --git a/docs/config/routines.mdx b/docs/config/routines.mdx index 66c1c912..8628388c 100644 --- a/docs/config/routines.mdx +++ b/docs/config/routines.mdx @@ -17,3 +17,7 @@ on the tun device and `SO_REUSEPORT` on the UDP socket to allow multiple queues. The _maximum_ recommended setting is half of the available CPU cores. It's recommended to set this to a lower value still, to avoid resource starvation. + +When set above `1`, the reader threads are pinned to CPUs by default; see +[`tun.pin_threads`](/docs/config/tun/#tunpin_threads) and [`tun.cpu_affinity`](/docs/config/tun/#tuncpu_affinity) to +control this behavior. diff --git a/docs/config/tun.mdx b/docs/config/tun.mdx index 38d7f655..b7d0bf53 100644 --- a/docs/config/tun.mdx +++ b/docs/config/tun.mdx @@ -153,3 +153,69 @@ This option is only supported on Linux. Buffer size for reading route updates. `0` means the system default buffer size is used (`/proc/sys/net/core/rmem_default`). If using massive route updates, for example BGP, you may need to increase this value to avoid packet loss. `SO_RCVBUFFORCE` is used to avoid having to raise the system wide max. + +## tun.use_offloads + +Default: False +Added in v1.12.0 + +This option is only supported on Linux and is not reloadable. + +When enabled, Nebula attempts to enable segmentation offloads on the tun device. The host network stack +can then hand Nebula TCP and UDP payloads much larger than the MTU in a single read, and accept large coalesced packets +on write, substantially reducing per-packet overhead for high-throughput flows. Nebula negotiates the best supported +mode at startup (TSO with USO, then TSO alone on kernels older than 6.2, then no offloads) and logs the result in the +`TUN offload status` message. + +For the biggest throughput gains, pair this option with +[`listen.udp_offloads`](/docs/config/listen/#listenudp_offloads). + +## tun.pin_threads + +Default: True +Added in v1.12.0 + +This option is only supported on Linux and is not reloadable. It only takes effect when +[`routines`](/docs/config/routines/) is greater than `1`. + +When enabled, each tun reader/encrypt OS thread is pinned to a single CPU. This keeps every thread's batched sends +flowing through one XPS-selected NIC transmit ring, so packets within a flow stay ordered on the wire instead of being +sprayed across multiple transmit rings and reordered. + +Unless [`tun.cpu_affinity`](#tuncpu_affinity) overrides it, Nebula picks the CPUs to pin to automatically: it prefers +performance cores on heterogeneous CPUs (ARM big.LITTLE, Intel P/E hybrids, AMD compact cores), keeps all readers on one +NUMA node and on distinct physical cores when the topology allows (SMT siblings last), leaves CPU 0's physical core as a +last resort, and rotates its starting point per instance so multiple Nebula instances on the same host don't stack their +readers onto the same cores. + +## tun.cpu_affinity + +Added in v1.12.0 + +This option is only supported on Linux and is not reloadable. It only applies while [`tun.pin_threads`](#tunpin_threads) +is `true`. + +Overrides which CPUs the tun reader threads pin to: a list of CPU IDs, one per routine (see +[`routines`](/docs/config/routines/)). A list shorter than `routines` is cycled across the queues; extra entries are ignored. +IDs must be within the process's allowed CPU set, so the list respects `taskset` and cgroup cpusets. +If any entry is not an integer or not in the allowed set, the whole override is ignored and Nebula falls +back to the automatic selection described under [`tun.pin_threads`](#tunpin_threads). + +```yml +tun: + cpu_affinity: + - 2 + - 4 +``` + +## tun.pin_threads_key + +Default: pid +Added in v1.12.0 + +This option is only supported on Linux and is not reloadable. It only applies while [`tun.pin_threads`](#tunpin_threads) +is `true` and no [`tun.cpu_affinity`](#tuncpu_affinity) override is set. + +Controls how the automatic CPU selector varies its starting point between Nebula instances. Valid values are `pid` and +`port`. Use `port` if you want Nebula to choose the same cores every run (keyed by the bound UDP port), which is useful +for reproducible benchmarking. From 1a15505a5012f944f854f9ae49252e98c02b4739 Mon Sep 17 00:00:00 2001 From: JackDoan Date: Wed, 26 Aug 2026 14:48:58 -0500 Subject: [PATCH 2/2] Fix prettier wrapping in tun.mdx Co-Authored-By: Claude Fable 5 --- docs/config/tun.mdx | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/config/tun.mdx b/docs/config/tun.mdx index b7d0bf53..01103134 100644 --- a/docs/config/tun.mdx +++ b/docs/config/tun.mdx @@ -161,10 +161,10 @@ value to avoid packet loss. `SO_RCVBUFFORCE` is used to avoid having to raise th This option is only supported on Linux and is not reloadable. -When enabled, Nebula attempts to enable segmentation offloads on the tun device. The host network stack -can then hand Nebula TCP and UDP payloads much larger than the MTU in a single read, and accept large coalesced packets -on write, substantially reducing per-packet overhead for high-throughput flows. Nebula negotiates the best supported -mode at startup (TSO with USO, then TSO alone on kernels older than 6.2, then no offloads) and logs the result in the +When enabled, Nebula attempts to enable segmentation offloads on the tun device. The host network stack can then hand +Nebula TCP and UDP payloads much larger than the MTU in a single read, and accept large coalesced packets on write, +substantially reducing per-packet overhead for high-throughput flows. Nebula negotiates the best supported mode at +startup (TSO with USO, then TSO alone on kernels older than 6.2, then no offloads) and logs the result in the `TUN offload status` message. For the biggest throughput gains, pair this option with @@ -196,10 +196,10 @@ This option is only supported on Linux and is not reloadable. It only applies wh is `true`. Overrides which CPUs the tun reader threads pin to: a list of CPU IDs, one per routine (see -[`routines`](/docs/config/routines/)). A list shorter than `routines` is cycled across the queues; extra entries are ignored. -IDs must be within the process's allowed CPU set, so the list respects `taskset` and cgroup cpusets. -If any entry is not an integer or not in the allowed set, the whole override is ignored and Nebula falls -back to the automatic selection described under [`tun.pin_threads`](#tunpin_threads). +[`routines`](/docs/config/routines/)). A list shorter than `routines` is cycled across the queues; extra entries are +ignored. IDs must be within the process's allowed CPU set, so the list respects `taskset` and cgroup cpusets. If any +entry is not an integer or not in the allowed set, the whole override is ignored and Nebula falls back to the automatic +selection described under [`tun.pin_threads`](#tunpin_threads). ```yml tun: