Document offload and CPU pinning config

docs/config/listen.mdx

@@ -37,6 +37,10 @@ nodes. Using 0 on lighthouses and relay hosts will likely lead to connectivity i
 
 Sets the max number of packets to pull from the kernel for each syscall (under systems that support `recvmmsg`).
 
+When [`listen.udp_offloads`](#listenudp_offloads) has enabled GRO (Linux kernel 5.10+), each receive slot is sized for a
+full 64KiB coalesced superpacket, so the receive scratch memory is `batch × 64KiB` per listening socket (about 4MiB per
+routine at the default of 64). Lower this value to trade peak per-syscall throughput for memory on constrained hosts.
+
 ## listen.read_buffer, listen.write_buffer
 
 Configure socket buffers for the udp side (outside), leave unset to use the system defaults. Values will be doubled by
@@ -91,3 +95,32 @@ ip rule add not from all fwmark 4242 lookup 4242
 ip rule add from all lookup main suppress_prefixlength 0
 ip route add default dev nebula1 via <nebula_unsafe_route_g> table 4242
 ```
+
+## listen.rebind_on_network_change
+
+<Pill className="mb-24">Default: True</Pill>
+<Pill className="mb-24">Added in v1.11.0</Pill>
+
+This option is only supported on macOS and is not reloadable.
+
+macOS scopes the UDP socket to the interface it was created on, so moving between networks (wifi to wired, office to
+home) leaves Nebula sending out an interface that no longer has a route. When enabled, Nebula watches the routing socket
+and rebinds the listener once the change settles, so devices moving between networks recover without waiting for dead
+tunnel detection. iOS does not use this option, the host app drives the same rebind itself.
+
+## listen.udp_offloads
+
+<Pill className="mb-24">Default: False</Pill>
+<Pill className="mb-24">Added in v1.12.0</Pill>
+
+This option is only supported on Linux and is not reloadable.
+
+When enabled, Nebula attempts to enable UDP Generic Segmentation Offload (GSO, Linux kernel 4.18+) and Generic Receive
+Offload (GRO, Linux kernel 5.10+) on its UDP socket(s). With GSO, batches of packets bound for the same host are handed
+to the kernel as a single "superpacket" along with the size to split it back into individual datagrams, reducing
+per-packet syscall and network stack overhead on the send path. With GRO, the kernel coalesces bursts of datagrams
+arriving from the same host into a single large read. Support for each feature is probed when the socket is created, and
+Nebula falls back to standard sends and receives on kernels that lack it.
+
+Enabling GRO increases the fixed memory used by the receive path, see [`listen.batch`](#listenbatch) for details. For
+the biggest throughput gains, pair this option with [`tun.use_offloads`](/docs/config/tun/#tunuse_offloads).

docs/config/routines.mdx

@@ -17,3 +17,7 @@ on the tun device and `SO_REUSEPORT` on the UDP socket to allow multiple queues.
 
 The _maximum_ recommended setting is half of the available CPU cores. It's recommended to set this to a lower value
 still, to avoid resource starvation.
+
+When set above `1`, the reader threads are pinned to CPUs by default; see
+[`tun.pin_threads`](/docs/config/tun/#tunpin_threads) and [`tun.cpu_affinity`](/docs/config/tun/#tuncpu_affinity) to
+control this behavior.

docs/config/tun.mdx

@@ -153,3 +153,69 @@ This option is only supported on Linux.
 Buffer size for reading route updates. `0` means the system default buffer size is used
 (`/proc/sys/net/core/rmem_default`). If using massive route updates, for example BGP, you may need to increase this
 value to avoid packet loss. `SO_RCVBUFFORCE` is used to avoid having to raise the system wide max.
+
+## tun.use_offloads
+
+<Pill className="mb-24">Default: False</Pill>
+<Pill className="mb-24">Added in v1.12.0</Pill>
+
+This option is only supported on Linux and is not reloadable.
+
+When enabled, Nebula attempts to enable segmentation offloads on the tun device. The host network stack can then hand
+Nebula TCP and UDP payloads much larger than the MTU in a single read, and accept large coalesced packets on write,
+substantially reducing per-packet overhead for high-throughput flows. Nebula negotiates the best supported mode at
+startup (TSO with USO, then TSO alone on kernels older than 6.2, then no offloads) and logs the result in the
+`TUN offload status` message.
+
+For the biggest throughput gains, pair this option with
+[`listen.udp_offloads`](/docs/config/listen/#listenudp_offloads).
+
+## tun.pin_threads
+
+<Pill className="mb-24">Default: True</Pill>
+<Pill className="mb-24">Added in v1.12.0</Pill>
+
+This option is only supported on Linux and is not reloadable. It only takes effect when
+[`routines`](/docs/config/routines/) is greater than `1`.
+
+When enabled, each tun reader/encrypt OS thread is pinned to a single CPU. This keeps every thread's batched sends
+flowing through one XPS-selected NIC transmit ring, so packets within a flow stay ordered on the wire instead of being
+sprayed across multiple transmit rings and reordered.
+
+Unless [`tun.cpu_affinity`](#tuncpu_affinity) overrides it, Nebula picks the CPUs to pin to automatically: it prefers
+performance cores on heterogeneous CPUs (ARM big.LITTLE, Intel P/E hybrids, AMD compact cores), keeps all readers on one
+NUMA node and on distinct physical cores when the topology allows (SMT siblings last), leaves CPU 0's physical core as a
+last resort, and rotates its starting point per instance so multiple Nebula instances on the same host don't stack their
+readers onto the same cores.
+
+## tun.cpu_affinity
+
+<Pill className="mb-24">Added in v1.12.0</Pill>
+
+This option is only supported on Linux and is not reloadable. It only applies while [`tun.pin_threads`](#tunpin_threads)
+is `true`.
+
+Overrides which CPUs the tun reader threads pin to: a list of CPU IDs, one per routine (see
+[`routines`](/docs/config/routines/)). A list shorter than `routines` is cycled across the queues; extra entries are
+ignored. IDs must be within the process's allowed CPU set, so the list respects `taskset` and cgroup cpusets. If any
+entry is not an integer or not in the allowed set, the whole override is ignored and Nebula falls back to the automatic
+selection described under [`tun.pin_threads`](#tunpin_threads).
+
+```yml
+tun:
+  cpu_affinity:
+    - 2
+    - 4
+```
+
+## tun.pin_threads_key
+
+<Pill className="mb-24">Default: pid</Pill>
+<Pill className="mb-24">Added in v1.12.0</Pill>
+
+This option is only supported on Linux and is not reloadable. It only applies while [`tun.pin_threads`](#tunpin_threads)
+is `true` and no [`tun.cpu_affinity`](#tuncpu_affinity) override is set.
+
+Controls how the automatic CPU selector varies its starting point between Nebula instances. Valid values are `pid` and
+`port`. Use `port` if you want Nebula to choose the same cores every run (keyed by the bound UDP port), which is useful
+for reproducible benchmarking.