diff --git a/README.md b/README.md index b0e05806f..4f2effb96 100644 --- a/README.md +++ b/README.md @@ -421,9 +421,9 @@ For Prometheus exposition and debug endpoints, use [`middleware/metrics`](middle | `Detach` (async keep-alive) | yes | yes | no | | Connection hijack | yes | yes | yes | -**CPU pinning** (epoll and io_uring): loop `i` pins its thread to the `i`-th CPU the thread that starts the server is allowed to use (`taskset`, a cgroup cpuset, `sched_setaffinity`), never to a CPU outside that set; on a host with several NUMA nodes the CPUs are taken from the nodes the set covers in turn, so every allowed CPU holds a loop before any holds a second. On a host whose allowed CPUs differ in capacity (arm64 big.LITTLE, read from `/sys/devices/system/cpu/cpuN/cpu_capacity`, or the CPU parts in `/proc/cpuinfo` when the kernel exports none), the engines start at most one loop per CPU with at least half the largest capacity (`min(GOMAXPROCS, non-little CPUs)`: 8 loops on a 12-CPU host with 4 little CPUs, fewer than `GOMAXPROCS`) and pin every one of them: none runs on a little CPU and none is left unpinned (celeris#1036; a loop that floated drifted onto the little CPUs and a loop pinned to one stalled its connections). A `Config.Workers` above that count is honoured: the extra loops wrap round-robin over the non-little CPUs, each pinned, so the CPUs that get a second loop (4 of 8 for `Workers: 12`) take about twice the connections of the others; on io_uring the count is still capped by `RLIMIT_MEMLOCK`. Consequences of the smaller count: `Metrics.Workers` and `NumWorkers()` read the loops started (on the adaptive engine, the sum over its two engines), driver pools sized from `NumWorkers()` default smaller (a PostgreSQL pool of `4*NumWorkers`: 32 on that host, not 48), and `Config.MaxConns` is per loop, so the server-wide limit shrinks with it. Measured on the one big.LITTLE host tried (arm64, 8 big and 4 little CPUs, the NIC interrupt pinned to a big CPU; two dispatches, 35 s windows with 5 repetitions per cell, and 90 s windows with 4 repetitions at io_uring with 512 connections), against celeris v1.5.11 (all 12 loops pinned, 4 of them on the little CPUs): requests per second were 1.43 times at io_uring with 256 connections, 1.09 times at adaptive with 128 connections, 1.51 times at io_uring with 512 connections and 1.04 times for epoll with 4 KiB POST bodies, and 0.95 times for epoll with 512 connections (the one cell below v1.5.11). The behaviour before this change, with 4 floating loops, served epoll with 4 KiB POST bodies at 0.47 times v1.5.11 in an earlier run. The tail costs something: for epoll with 4 KiB POST bodies the median p99.99 was 99 ms against 48 ms for v1.5.11, and at io_uring with 512 connections a 90 s run had a median of 61 requests of 1 s or more and its longest took 1.0 to 2.5 s, against 664 and 12 to 21 s for v1.5.11, so stalls of 1 to 2.5 s remain there. A `Config.Workers` above the non-little CPU count is wrapped over the pinned big CPUs; it ran on hardware once, in an earlier run (35 s windows, 3 repetitions), and showed a worse tail than the default at io_uring with 512 connections (0.10% to 0.20% of requests at about 208 ms, maxima of 2.1 to 2.2 s in two of three cells). The engine logs one line saying how many loops it started and why; a pin the kernel refuses is logged once. With `Workers` unset, and a default count of at least one loop for each non-little CPU (8 of 8 on that host; a lower `GOMAXPROCS`, such as the default under a Kubernetes CPU limit, plans as above, so 7 default loops are 0, 1, 6, 7, 8, 9 and 10, more than the 6 loops of 8), the default plan also leaves the loops off the non-little CPUs that service the busy interrupt vectors of the NIC port that received the traffic (celeris#1036 item 1). At start the engine takes the physical ports behind the listen address (through bonds and VLANs); if one of them took at least 60% of the packets received since boot (`/sys/class/net//statistics/rx_packets`, and at least a million) it leaves without a loop the non-little CPUs that service its vectors with at least 1000 interrupts since boot (`/proc/interrupts`, `/proc/irq/N/effective_affinity_list`): CPUs 6 and 8 on that host, so the loops sit on 0, 1, 7, 9, 10 and 11. It frees never more than a quarter of the non-little CPUs (the loops never fall under `max(min(2, n), ceil(3n/4))` for `n` non-little CPUs) and never raises the count. A CPU that also services a busy vector of another port (or one shared with another port) keeps its loop, and at least one busy vector of a port with at least 10% of the packets pooled over the listener's ports (a choice, not measured; msr1's ports carry about 27% and 73%) must stay on a CPU that keeps its loop: with a single port, or with a data NIC beside a management NIC that carries less, the plan is the plan without the rule. The rule also does nothing when a file cannot be read, when a port has a million packets and no busy vector, when the busy vectors sit on more CPUs than the quarter allows (a NIC spread over its queues), when the ports share the packets about evenly, or when a busy vector of another port is serviced by a CPU that runs no loop (the msr1 boot placement, IRQ 167 on little CPU 4: that plan is not measured). A host whose CPUs are alike, and any `Config.Workers` you set, read no NIC file. The Info line of the plan names the CPUs left free, the port and its share; the Debug line names the candidates it passed over or why the rule stepped aside. Validated on one cluster host (msr1, arm64 big.LITTLE; probatorium runs 39355553598, 35 s, and 39367858847, 90 s, same boot): the rule planned exactly the loops 0, 1, 7, 9, 10 and 11 by itself and matched the hand-pinned set within 1% of requests per second on io256, ad128, ep512 and post4k, and at io512 reached 495k against 500k requests per second with a maximum of 34 ms and no request of 1 s or more (v1.5.11: 343k, maximum 14.1 s, 664 requests of 1 s or more; the 8 loops of this paragraph alone: 518k, maximum 1.7 s, 61 requests of 1 s or more). Against v1.5.11 post4k p99.99 was 3.4 ms against 48 ms (v1.5.11) and 99 ms (the 8 loops alone), with throughput within 1% of the hand-pinned set; io256 gained 35%, ad128 5% and io512 45%. Against the 8 loops alone the rule has lower tails at io512 (maximum 34 ms against 1.7 s) and post4k, 495k against 518k requests per second at io512, and its io256 p99.99 is the FAIL cell below. The cost: ep512 throughput is about 0.845 times that of v1.5.11 (15% lower; the 8 loops alone: 0.948 times). The pre-registered test of the rule came out FAIL by its letter, only because the io256 p99.99 of the rule over the hand-pinned set was 5.59 times at the medians (overlapping ranges, 3.2 to 32.8 ms and 3.6 to 28.3 ms). A re-run of that cell on one commit (probatorium run 39396052154: the rule against the same six loops forced by a knob, 8 repetitions per arm) had the rule plan the same six loops by itself, with 0.995 times the requests per second, a maximum-latency ratio of 1.02 and a p99.99 median of 16.8 ms against 6.0 ms (2.83 times; 6 of 8 against 3 of 8 repetitions over 10 ms); the one-sided exact Mann-Whitney test does not reject (p=0.080) and the registered outcome is NOISE-ONLY, but at 8 repetitions its power is low and a modest p99.99 tail cost at io256 in the 10 to 30 ms range is not excluded. Limits: the rule is fitted to that one host after its data was seen (the 60% and the quarter are choices), and its counts are since boot, so a server started at boot steps aside and a wrapped 32-bit interrupt counter can hide a vector (IRQ 135 on msr1 is at about 4.05e9 of 4.29e9); a LAG rehash that moves the traffic to the other port leaves the plan on the old CPUs until the next start; a wildcard listener pools every physical port that is up, related or not (the 10% share above governs a data NIC beside a management NIC); on NICs whose ports share all their vectors (mlx4) the rule never acts; io_uring's `RLIMIT_MEMLOCK` cap keeps the first loops of the plan in CPU order and can drop the CPU the rule meant to keep (a memlock of 4 gives 0, 1, 7 and 9); and the adaptive engine builds its io_uring standby when it first switches, from the counts of that moment, so its loop set can differ from the primary's. A handler that runs inline runs on a pinned thread and a process it starts inherits the pin: start it from a goroutine of its own, or mark the route `.Async()` (see `Config.AsyncHandlers`). Setting `AsyncHandlers` alone is not enough: a route that only inherits it runs inline until it is promoted. +**CPU pinning** (epoll and io_uring): loop `i` pins its thread to the `i`-th CPU the thread that starts the server is allowed to use (`taskset`, a cgroup cpuset, `sched_setaffinity`), never to a CPU outside that set; on a host with several NUMA nodes the CPUs are taken from the nodes the set covers in turn, so every allowed CPU holds a loop before any holds a second. With `Workers` unset on a host whose allowed CPUs are alike, the engines start `GOMAXPROCS - max(1, GOMAXPROCS/8)` loops (28 of 32, 14 of 16, 7 of 8, 3 of 4; no reserve for a `GOMAXPROCS` of 2 or less): a loop can hold its thread and its P while it blocks in the kernel, so the Ps without a loop are left to the goroutines that are not loops (async and WebSocket handlers, the HTTP/2 stream pool, the garbage collector's workers) (celeris#756, #757). The mechanism is inferred from traces, not established; the reserve is provisional and its effect on latency is still being measured. To opt out, set `Workers: runtime.GOMAXPROCS(0)`: on such a host it starts one loop per P, as before (on a big.LITTLE host the default is not reduced, and a `Workers` above the non-little CPUs wraps loops over them, see below); a `Workers` you set is never reduced. `Metrics.Workers`, `NumWorkers()` and the server-wide `Config.MaxConns` limit follow the loops started, as on big.LITTLE below. On a host whose allowed CPUs differ in capacity (arm64 big.LITTLE, read from `/sys/devices/system/cpu/cpuN/cpu_capacity`, or the CPU parts in `/proc/cpuinfo` when the kernel exports none), the engines start at most one loop per CPU with at least half the largest capacity (`min(GOMAXPROCS, non-little CPUs)`: 8 loops on a 12-CPU host with 4 little CPUs, fewer than `GOMAXPROCS`) and pin every one of them: none runs on a little CPU and none is left unpinned (celeris#1036; a loop that floated drifted onto the little CPUs and a loop pinned to one stalled its connections). A `Config.Workers` above that count is honoured: the extra loops wrap round-robin over the non-little CPUs, each pinned, so the CPUs that get a second loop (4 of 8 for `Workers: 12`) take about twice the connections of the others; on io_uring the count is still capped by `RLIMIT_MEMLOCK`. Consequences of the smaller count: `Metrics.Workers` and `NumWorkers()` read the loops started (on the adaptive engine, the sum over its two engines), driver pools sized from `NumWorkers()` default smaller (a PostgreSQL pool of `4*NumWorkers`: 32 on that host, not 48), and `Config.MaxConns` is per loop, so the server-wide limit shrinks with it. Measured on the one big.LITTLE host tried (arm64, 8 big and 4 little CPUs, the NIC interrupt pinned to a big CPU; two dispatches, 35 s windows with 5 repetitions per cell, and 90 s windows with 4 repetitions at io_uring with 512 connections), against celeris v1.5.11 (all 12 loops pinned, 4 of them on the little CPUs): requests per second were 1.43 times at io_uring with 256 connections, 1.09 times at adaptive with 128 connections, 1.51 times at io_uring with 512 connections and 1.04 times for epoll with 4 KiB POST bodies, and 0.95 times for epoll with 512 connections (the one cell below v1.5.11). The behaviour before this change, with 4 floating loops, served epoll with 4 KiB POST bodies at 0.47 times v1.5.11 in an earlier run. The tail costs something: for epoll with 4 KiB POST bodies the median p99.99 was 99 ms against 48 ms for v1.5.11, and at io_uring with 512 connections a 90 s run had a median of 61 requests of 1 s or more and its longest took 1.0 to 2.5 s, against 664 and 12 to 21 s for v1.5.11, so stalls of 1 to 2.5 s remain there. A `Config.Workers` above the non-little CPU count is wrapped over the pinned big CPUs; it ran on hardware once, in an earlier run (35 s windows, 3 repetitions), and showed a worse tail than the default at io_uring with 512 connections (0.10% to 0.20% of requests at about 208 ms, maxima of 2.1 to 2.2 s in two of three cells). The engine logs one line saying how many loops it started and why; a pin the kernel refuses is logged once. With `Workers` unset, and a default count of at least one loop for each non-little CPU (8 of 8 on that host; a lower `GOMAXPROCS`, such as the default under a Kubernetes CPU limit, plans as above, so 7 default loops are 0, 1, 6, 7, 8, 9 and 10, more than the 6 loops of 8), the default plan also leaves the loops off the non-little CPUs that service the busy interrupt vectors of the NIC port that received the traffic (celeris#1036 item 1). At start the engine takes the physical ports behind the listen address (through bonds and VLANs); if one of them took at least 60% of the packets received since boot (`/sys/class/net//statistics/rx_packets`, and at least a million) it leaves without a loop the non-little CPUs that service its vectors with at least 1000 interrupts since boot (`/proc/interrupts`, `/proc/irq/N/effective_affinity_list`): CPUs 6 and 8 on that host, so the loops sit on 0, 1, 7, 9, 10 and 11. It frees never more than a quarter of the non-little CPUs (the loops never fall under `max(min(2, n), ceil(3n/4))` for `n` non-little CPUs) and never raises the count. A CPU that also services a busy vector of another port (or one shared with another port) keeps its loop, and at least one busy vector of a port with at least 10% of the packets pooled over the listener's ports (a choice, not measured; msr1's ports carry about 27% and 73%) must stay on a CPU that keeps its loop: with a single port, or with a data NIC beside a management NIC that carries less, the plan is the plan without the rule. The rule also does nothing when a file cannot be read, when a port has a million packets and no busy vector, when the busy vectors sit on more CPUs than the quarter allows (a NIC spread over its queues), when the ports share the packets about evenly, or when a busy vector of another port is serviced by a CPU that runs no loop (the msr1 boot placement, IRQ 167 on little CPU 4: that plan is not measured). A host whose CPUs are alike, and any `Config.Workers` you set, read no NIC file. The Info line of the plan names the CPUs left free, the port and its share; the Debug line names the candidates it passed over or why the rule stepped aside. Validated on one cluster host (msr1, arm64 big.LITTLE; probatorium runs 39355553598, 35 s, and 39367858847, 90 s, same boot): the rule planned exactly the loops 0, 1, 7, 9, 10 and 11 by itself and matched the hand-pinned set within 1% of requests per second on io256, ad128, ep512 and post4k, and at io512 reached 495k against 500k requests per second with a maximum of 34 ms and no request of 1 s or more (v1.5.11: 343k, maximum 14.1 s, 664 requests of 1 s or more; the 8 loops of this paragraph alone: 518k, maximum 1.7 s, 61 requests of 1 s or more). Against v1.5.11 post4k p99.99 was 3.4 ms against 48 ms (v1.5.11) and 99 ms (the 8 loops alone), with throughput within 1% of the hand-pinned set; io256 gained 35%, ad128 5% and io512 45%. Against the 8 loops alone the rule has lower tails at io512 (maximum 34 ms against 1.7 s) and post4k, 495k against 518k requests per second at io512, and its io256 p99.99 is the FAIL cell below. The cost: ep512 throughput is about 0.845 times that of v1.5.11 (15% lower; the 8 loops alone: 0.948 times). The pre-registered test of the rule came out FAIL by its letter, only because the io256 p99.99 of the rule over the hand-pinned set was 5.59 times at the medians (overlapping ranges, 3.2 to 32.8 ms and 3.6 to 28.3 ms). A re-run of that cell on one commit (probatorium run 39396052154: the rule against the same six loops forced by a knob, 8 repetitions per arm) had the rule plan the same six loops by itself, with 0.995 times the requests per second, a maximum-latency ratio of 1.02 and a p99.99 median of 16.8 ms against 6.0 ms (2.83 times; 6 of 8 against 3 of 8 repetitions over 10 ms); the one-sided exact Mann-Whitney test does not reject (p=0.080) and the registered outcome is NOISE-ONLY, but at 8 repetitions its power is low and a modest p99.99 tail cost at io256 in the 10 to 30 ms range is not excluded. Limits: the rule is fitted to that one host after its data was seen (the 60% and the quarter are choices), and its counts are since boot, so a server started at boot steps aside and a wrapped 32-bit interrupt counter can hide a vector (IRQ 135 on msr1 is at about 4.05e9 of 4.29e9); a LAG rehash that moves the traffic to the other port leaves the plan on the old CPUs until the next start; a wildcard listener pools every physical port that is up, related or not (the 10% share above governs a data NIC beside a management NIC); on NICs whose ports share all their vectors (mlx4) the rule never acts; io_uring's `RLIMIT_MEMLOCK` cap keeps the first loops of the plan in CPU order and can drop the CPU the rule meant to keep (a memlock of 4 gives 0, 1, 7 and 9); and the adaptive engine builds its io_uring standby when it first switches, from the counts of that moment, so its loop set can differ from the primary's. A handler that runs inline runs on a pinned thread and a process it starts inherits the pin: start it from a goroutine of its own, or mark the route `.Async()` (see `Config.AsyncHandlers`). Setting `AsyncHandlers` alone is not enough: a route that only inherits it runs inline until it is promoted. -**NIC interrupts on big.LITTLE hosts** (epoll and io_uring, Linux, hosts whose allowed CPUs differ in capacity only; a host whose CPUs are alike reads nothing and is unchanged). Once the engine has started its loops it finds the CPUs that service the interrupt vectors of the network interface the listener is reachable through (celeris#1036) and logs one Warn when a serviced CPU of the allowed set runs no loop and is either a little CPU, or a big CPU while no vector it found has a loop beside it. It only reports: it adds no loop, moves no loop and moves no interrupt, so the loops, `Metrics.Workers` and `NumWorkers()` are the ones described above on every host. The rule behind the Warn is the premise of the rx-port rule above: a loop stays beside a busy vector. On the host measured, CPUs 6 and 8 service the vectors of enp49s0 and run no loop in the default plan, and that plan is the measured one as long as CPU 1 keeps a loop beside IRQ 167; so a big CPU without a loop is not a reason to warn while another vector has a loop beside it, and with `Workers: 4` or a `GOMAXPROCS` of 4 (loops on 0, 1, 6 and 7, CPU 8 without one) nothing is logged. A little CPU runs no loop by design, so a vector serviced by one is always named. The CPUs the rx-port rule left without a loop on purpose are never named: a vector it listed, on a CPU it freed. What the Warn does on that host: +**NIC interrupts on big.LITTLE hosts** (epoll and io_uring, Linux, hosts whose allowed CPUs differ in capacity only; a host whose CPUs are alike reads nothing there). Once the engine has started its loops it finds the CPUs that service the interrupt vectors of the network interface the listener is reachable through (celeris#1036) and logs one Warn when a serviced CPU of the allowed set runs no loop and is either a little CPU, or a big CPU while no vector it found has a loop beside it. It only reports: it adds no loop, moves no loop and moves no interrupt, so the loops, `Metrics.Workers` and `NumWorkers()` are the ones described above on every host. The rule behind the Warn is the premise of the rx-port rule above: a loop stays beside a busy vector. On the host measured, CPUs 6 and 8 service the vectors of enp49s0 and run no loop in the default plan, and that plan is the measured one as long as CPU 1 keeps a loop beside IRQ 167; so a big CPU without a loop is not a reason to warn while another vector has a loop beside it, and with `Workers: 4` or a `GOMAXPROCS` of 4 (loops on 0, 1, 6 and 7, CPU 8 without one) nothing is logged. A little CPU runs no loop by design, so a vector serviced by one is always named. The CPUs the rx-port rule left without a loop on purpose are never named: a vector it listed, on a CPU it freed. What the Warn does on that host: - IRQ 167 on CPU 1, default `Workers` (the rule frees CPUs 6 and 8, loops on 0, 1, 7, 9, 10 and 11): no Warn. - IRQ 167 on little CPU 4 (the placement after boot; the rule steps aside, eight loops): the Warn names CPU 4 and IRQ 167. This is the case it exists for. If the 32-bit counter of IRQ 167 has wrapped and reads small, the rule still sees the vector busy (1000 interrupts) and steps aside, and the Warn names it whatever its share. diff --git a/config.go b/config.go index e3959425e..98fc4b9d7 100644 --- a/config.go +++ b/config.go @@ -120,7 +120,27 @@ type Config struct { // Engine is the I/O engine (default Adaptive on Linux, Std elsewhere). Engine EngineType - // Workers is the number of I/O worker goroutines (default GOMAXPROCS). + // Workers is the number of I/O worker goroutines (the loops of the native + // engines). Default: GOMAXPROCS, less a reserve on Linux: with Workers unset + // the native engines (epoll, io_uring, adaptive) on a host whose allowed + // CPUs are alike start GOMAXPROCS - max(1, GOMAXPROCS/8) loops (no reserve + // for a GOMAXPROCS of 2 or less): 28 on 32, 14 on 16, 7 on 8, 3 on 4, 2 on + // 3. A loop locks its thread and can block in the kernel while it holds its + // P, so the Ps that run no loop are left to the goroutines that are not + // loops (async and WebSocket handlers, the HTTP/2 stream pool, the garbage + // collector's workers). That mechanism is inferred from traces, not + // established; the reserve is provisional, its effect on latency is still + // being measured, and it may change (celeris#756, #757). A Workers you set + // is honoured as it is, so to opt out of the reserve set Workers to + // runtime.GOMAXPROCS(0): on such a host that starts one loop per P, as + // before the reserve (on a big.LITTLE host the default is not reduced, see + // below, and a Workers set above the non-little CPUs wraps loops over + // them). What follows the loop count follows the loops started: Metrics.Workers, + // the server-wide limit of MaxConns, and a driver pool sized from the + // engine's NumWorkers (a PostgreSQL pool opened with WithEngine, whose + // default of 4*NumWorkers is 56, not 64, on 16 CPUs). A pool sized from + // GOMAXPROCS or the standalone driver loop (Redis commands, Memcached with + // the native engines) is unchanged. // // On Linux the native engines (epoll, io_uring) pin each loop to one CPU // the process is allowed to use. On a host whose allowed CPUs differ in @@ -225,8 +245,9 @@ type Config struct { // host, one observation); an automatic extra loop on the little CPU was // tried and rejected because it raised tail latency on other workloads. // Moving the interrupt is the host operator's call (README, "NIC interrupts - // on big.LITTLE hosts"). Hosts whose CPUs are alike are unchanged and read no - // NIC file (README, "CPU pinning"). + // on big.LITTLE hosts"). Hosts whose CPUs are alike read no + // NIC file; the reserve above is their only difference (README, "CPU + // pinning"). Workers int // WorkloadHint optionally declares the expected steady-state concurrency. diff --git a/internal/adaptive/engine.go b/internal/adaptive/engine.go index 2ec34e766..e694b0b74 100644 --- a/internal/adaptive/engine.go +++ b/internal/adaptive/engine.go @@ -219,10 +219,12 @@ var planLoopCPUs = platform.PlanWorkerCPUsFor // `requested` workers (explicit: Config.Resources.Workers was set by the // user): the length of the platform's CPU plan. On a big.LITTLE host that is // one loop per non-little CPU, fewer than GOMAXPROCS (celeris#1036, #1049); on -// every other host it is `requested`. The plan is made without the listen -// address, so the rx-port rule that may leave a few more CPUs out is not -// applied: the count is the upper bound of what the engines start, and the -// memlock test below never says "viable" for fewer loops than they start. It +// a host of alike CPUs it is `requested` less the default reserve when the +// count was not set by the user (celeris#756, #757), `requested` otherwise. +// The plan is made without the listen address, so the rx-port rule that may +// leave a few more CPUs out is not applied: the count is the upper bound of +// what the engines start, and the memlock test below never says "viable" for +// fewer loops than they start. It // plans on the CPU affinity of the thread that calls New, the sub-engines on // the one that calls Listen (sched_getaffinity is per thread): they differ // only if an application narrows the mask of one thread. diff --git a/internal/adaptive/started_loops_756_linux_test.go b/internal/adaptive/started_loops_756_linux_test.go new file mode 100644 index 000000000..93914c76c --- /dev/null +++ b/internal/adaptive/started_loops_756_linux_test.go @@ -0,0 +1,34 @@ +//go:build linux + +package adaptive + +import ( + "testing" + + "github.com/goceleris/celeris/internal/platform" +) + +// celeris#756, #757: startedLoops, which the adaptive engine uses to size the +// io_uring memlock check, counts the loops the engines start: the default +// count on a host of alike CPUs is the request less the reserve of +// max(1, n/8) (none at 2 or less), a user-set count is the request. The real +// planner, no stub; the figures are written out. +func TestStartedLoopsLeaveAReserve756(t *testing.T) { + if platform.PlanWorkerCPUs(12, true).Heterogeneous { + // Not a t.Skip: skip_routing_684_test.go polices every skip of this package. + t.Log("this host's CPUs differ in capacity: the big.LITTLE plan is not the reserve's; nothing to check") + return + } + for _, tc := range []struct { + requested int + explicit bool + want int + }{ + {16, false, 14}, {16, true, 16}, {8, false, 7}, {32, false, 28}, + {4, false, 3}, {3, false, 2}, {2, false, 2}, {1, false, 1}, {5, true, 5}, + } { + if got := startedLoops(tc.requested, tc.explicit); got != tc.want { + t.Errorf("startedLoops(%d, explicit=%v) = %d, want %d", tc.requested, tc.explicit, got, tc.want) + } + } +} diff --git a/internal/engine/epoll/default_loops_756_linux_test.go b/internal/engine/epoll/default_loops_756_linux_test.go new file mode 100644 index 000000000..bb9613a5e --- /dev/null +++ b/internal/engine/epoll/default_loops_756_linux_test.go @@ -0,0 +1,69 @@ +//go:build linux + +package epoll + +import ( + "fmt" + "runtime" + "strings" + "testing" + "time" + + "github.com/goceleris/celeris/internal/platform" + "github.com/goceleris/celeris/internal/platform/pintest" +) + +// celeris#756, #757: an epoll engine whose worker count is not set by the user +// starts GOMAXPROCS - max(1, GOMAXPROCS/8) loops on a host of alike CPUs (none +// for a GOMAXPROCS of 2 or less; Resolve raises a count to two), and a count +// the user sets is started whole. This is the real planner and real engine, no +// stub: Metrics().Workers, NumWorkers(), the loops and the "listening" line +// all read the loops started. The test pins the policy, not any latency. +// +// GOMAXPROCS is set inside the test's own process, so the request is the same +// on any runner (the figures are written out, not computed from the planner). +func TestDefaultLoopsLeaveAReserve756(t *testing.T) { + if !pintest.InOwnProcess(t) { + pintest.RunInOwnProcess(t, 3*time.Minute) + return + } + if p := platform.PlanWorkerCPUs(12, true); p.Heterogeneous { + t.Skip("this host's CPUs differ in capacity: the big.LITTLE plan is not the reserve's") + } + prev := runtime.GOMAXPROCS(0) + defer runtime.GOMAXPROCS(prev) + for _, tc := range []struct { + gomaxprocs int + workers int // Resources.Workers; 0 = the default + want int // loops started + }{ + {16, 0, 14}, // the default: 16 Ps, 14 loops (rc1: 16) + {16, 16, 16}, // Workers = GOMAXPROCS: one loop per P, as before + {16, 5, 5}, // a user-set count is kept whole + {16, 20, 20}, // also above GOMAXPROCS + {8, 0, 7}, + {4, 0, 3}, + {3, 0, 2}, + {2, 0, 2}, // no reserve at two + {1, 0, 2}, // Resolve raises the default to MinWorkers (2); no reserve at two + } { + t.Run(fmt.Sprintf("GOMAXPROCS=%d/Workers=%d", tc.gomaxprocs, tc.workers), func(t *testing.T) { + runtime.GOMAXPROCS(tc.gomaxprocs) + var log lockedBuf + eng, stop := start909(t, tc.workers, &log) + defer stop() + if got := len(eng.loops); got != tc.want { + t.Errorf("%d loops started, want %d", got, tc.want) + } + if got := eng.Metrics().Workers; got != tc.want { + t.Errorf("Metrics().Workers = %d, want %d", got, tc.want) + } + if got := eng.NumWorkers(); got != tc.want { + t.Errorf("NumWorkers() = %d, want %d", got, tc.want) + } + if got := listeningLine1036(t, &log); !strings.Contains(got, fmt.Sprintf(" loops=%d", tc.want)) { + t.Errorf("the listening line does not say loops=%d:\n%s", tc.want, log.String()) + } + }) + } +} diff --git a/internal/engine/epoll/engine.go b/internal/engine/epoll/engine.go index ae03afa1b..b7a0ddbfe 100644 --- a/internal/engine/epoll/engine.go +++ b/internal/engine/epoll/engine.go @@ -168,7 +168,10 @@ func (e *Engine) Listen(ctx context.Context) error { // celeris#909: the CPU of loop i is the i-th member of the process's // allowed set, and on a big.LITTLE host never a little CPU. The plan sets // the loop count (celeris#1036): on a big.LITTLE host it holds one loop - // per non-little CPU, fewer than the default GOMAXPROCS, every one pinned. + // per non-little CPU, fewer than the default GOMAXPROCS, every one pinned; + // with Workers unset on a host of alike CPUs it holds the default less the + // reserve that leaves Ps to the goroutines that are not loops (celeris#756, + // #757). plan := planWorkerCPUs(resolved.Workers, e.cfg.Resources.Workers > 0, e.cfg.Addr) cpus := plan.CPUs resolved.Workers = len(cpus) diff --git a/internal/engine/epoll/pin_linux.go b/internal/engine/epoll/pin_linux.go index 933e735b4..5f70c4ef8 100644 --- a/internal/engine/epoll/pin_linux.go +++ b/internal/engine/epoll/pin_linux.go @@ -75,7 +75,9 @@ func nicIRQReport(log *slog.Logger, plan platform.CPUPlan, addr string) platform } // The plan is also the loop count: the engine starts len(plan.CPUs) of them -// (celeris#1036), which on a big.LITTLE host is fewer than the default. +// (celeris#1036), which on a big.LITTLE host is fewer than the default, and for +// a default count on a host of alike CPUs is the default less the reserve of +// the platform's defaultLoopReserve (celeris#756, #757). // pinFailures collects the loops whose pin failed until the engine logs them // (celeris#909). diff --git a/internal/engine/iouring/engine.go b/internal/engine/iouring/engine.go index 609b50965..bc7b0c21c 100644 --- a/internal/engine/iouring/engine.go +++ b/internal/engine/iouring/engine.go @@ -425,7 +425,9 @@ func (e *Engine) Listen(ctx context.Context) error { // allowed set, and on a big.LITTLE host never a little CPU. The plan sets // the worker count (celeris#1036): on a big.LITTLE host it holds at most // one worker per non-little CPU, fewer than the default GOMAXPROCS, every - // one pinned. + // one pinned; with Workers unset on a host of alike CPUs it holds the + // default less the reserve that leaves Ps to the goroutines that are not + // loops (celeris#756, #757). // // RLIMIT_MEMLOCK bounds the plan, not the request. Each worker locks ~12 MB // for its ring + buffers; the systemd / kernel default of 8 MB only fits @@ -905,7 +907,8 @@ func (e *Engine) NumWorkers() int { // WorkerLoop returns the WorkerLoop for worker n. Out-of-range n is // reduced modulo NumWorkers so callers can hash a connection / FD across // the available pool without first reading NumWorkers — important now -// that workers can be capped below NumCPU by RLIMIT_MEMLOCK. +// that workers can be below NumCPU: capped by RLIMIT_MEMLOCK, or fewer by the +// default reserve (celeris#756, #757). // Negative n is mirrored to the positive side first. Panics only when // the engine has no workers (i.e. Listen has not yet started). func (e *Engine) WorkerLoop(n int) engine.WorkerLoop { diff --git a/internal/engine/iouring/pin_linux.go b/internal/engine/iouring/pin_linux.go index f546ed680..ef4de94c7 100644 --- a/internal/engine/iouring/pin_linux.go +++ b/internal/engine/iouring/pin_linux.go @@ -33,7 +33,9 @@ func nicIRQReport(log *slog.Logger, plan platform.CPUPlan, addr string) platform } // The plan is also the loop count: the engine starts len(plan.CPUs) of them -// (celeris#1036), which on a big.LITTLE host is fewer than the default. +// (celeris#1036), which on a big.LITTLE host is fewer than the default, and for +// a default count on a host of alike CPUs is the default less the reserve of +// the platform's defaultLoopReserve (celeris#756, #757). // planWithinMemlock plans the workers and then holds the plan to what // RLIMIT_MEMLOCK funds. The plan comes first because on a big.LITTLE host it diff --git a/internal/engine/provider.go b/internal/engine/provider.go index d178bd31e..525271a63 100644 --- a/internal/engine/provider.go +++ b/internal/engine/provider.go @@ -33,8 +33,10 @@ var ErrSwitchingNotFrozen = errors.New("celeris/engine: adaptive engine switchin // listening. type EventLoopProvider interface { // NumWorkers returns the number of worker event loops available for - // driver FD registration. This is typically equal to the engine's - // configured worker count (one per CPU core). + // driver FD registration. This is the loops the engine started: the + // configured worker count, or on Linux for a default count on a host of alike + // CPUs GOMAXPROCS less a reserve (celeris#756, #757), and fewer on a + // big.LITTLE host or under an io_uring memlock cap. NumWorkers() int // WorkerLoop returns the WorkerLoop for worker n, where 0 <= n < NumWorkers(). // Calling WorkerLoop with an out-of-range index panics. diff --git a/internal/platform/cpuplan_1036_linux_test.go b/internal/platform/cpuplan_1036_linux_test.go index 038e2f0d8..bd514a3f0 100644 --- a/internal/platform/cpuplan_1036_linux_test.go +++ b/internal/platform/cpuplan_1036_linux_test.go @@ -89,9 +89,15 @@ func TestPlanHeterogeneousNeverUnpinnedNeverLittle1036(t *testing.T) { t.Fatalf("%s: %d of %d entries pinned", tag, p.Pinned(), len(p.CPUs)) } if !het { - // Alike CPUs: one entry per worker, whatever explicit says. - if len(p.CPUs) != workers { - t.Fatalf("%s: %d entries, want %d (a host of alike CPUs keeps every worker)", tag, len(p.CPUs), workers) + // Alike CPUs: one entry per worker for a user-set count; a + // default count leaves the reserve of max(1, n/8) without + // a loop, none for n <= 2 (celeris#756, #757). + wantLoops := workers + if !explicit && workers > 2 { + wantLoops -= max(1, workers/8) + } + if len(p.CPUs) != wantLoops { + t.Fatalf("%s: %d entries, want %d (a host of alike CPUs keeps every worker it is asked for, less the default reserve)", tag, len(p.CPUs), wantLoops) } continue } diff --git a/internal/platform/cpuplan_linux.go b/internal/platform/cpuplan_linux.go index 4a9e066f6..a0707e533 100644 --- a/internal/platform/cpuplan_linux.go +++ b/internal/platform/cpuplan_linux.go @@ -27,8 +27,10 @@ type CPUPlan struct { // itself to. Its length is the loop count: on a host whose allowed CPUs // differ in capacity it can be smaller than the number of workers asked // for, and the engine starts len(CPUs) loops, not the number it asked for - // (celeris#1036). The planner never leaves an entry at -1 ("run - // unpinned"); an engine still honours a -1 it is given, and a pin the + // (celeris#1036); so does a default count on a host of alike CPUs, which + // leaves defaultLoopReserve(Requested) workers without a loop (celeris#756, + // #757); Note stays empty for it. The planner never leaves an entry at -1 + // ("run unpinned"); an engine still honours a -1 it is given, and a pin the // kernel refuses leaves that one loop unpinned and is logged. CPUs []int // Requested is the number of workers the plan was asked for. @@ -135,9 +137,12 @@ func (p CPUPlan) Truncate(n int, why string) CPUPlan { // numWorkers above the number of non-little CPUs is honoured, never floated: // the extra loops wrap round-robin over the non-little CPUs, each pinned (two // loops share a CPU). Where the CPUs are alike the i-th worker gets the i-th -// allowed CPU, wrapping. On a host with several NUMA nodes the CPUs are taken -// from the nodes the allowed set covers in turn, and every allowed CPU gets a -// loop before any gets a second. +// allowed CPU, wrapping, and a default count (not explicit) is +// numWorkers - [defaultLoopReserve](numWorkers) loops, the Ps left over +// serving the goroutines that are not loops (celeris#756, #757); an explicit +// count is planned as asked. On a host with several NUMA nodes the CPUs are +// taken from the nodes the allowed set covers in turn, and every allowed CPU +// gets a loop before any gets a second. // // Any read error means "homogeneous"; the plan then is what a host without // the information gets. @@ -206,6 +211,28 @@ func schedAllowedCPUs() ([]int, error) { return out, nil } +// defaultLoopReserve is how many of the numWorkers workers a DEFAULT loop count +// (Config.Workers not set) leaves without a loop on a host whose allowed CPUs +// are alike: the loops start numWorkers - defaultLoopReserve(numWorkers). A +// loop locks its OS thread and can block in the kernel (epoll_wait, +// io_uring_enter) while it holds its P, so with as many loops as Ps the +// goroutines that are not loops (async and WebSocket handlers, the HTTP/2 stream +// pool, GC workers) may run only on a P the runtime takes back from a blocked +// loop (celeris#756, #757; a mechanism inferred from traces, not established). +// The reserve leaves that many Ps to them. +// +// max(1, n/8), none for n <= 2 (a count that small would drop to one loop or +// none): 14 of 16, chosen by the maintainer after a cluster comparison of 16, +// 14 and 12 loops on one amd64 host. The runs on the final commit that would +// confirm the reserve are still owed, so the value may change; changing it is +// this function's one return statement (celeris#756, #757). +func defaultLoopReserve(n int) int { + if n <= 2 { + return 0 + } + return max(1, n/8) +} + func planWorkerCPUs(numWorkers int, explicit bool, src planSource) CPUPlan { if numWorkers < 0 { numWorkers = 0 @@ -226,7 +253,14 @@ func planWorkerCPUs(numWorkers int, explicit bool, src planSource) CPUPlan { plan := CPUPlan{Allowed: allowed, Requested: numWorkers} little, capNote := littleCPUs(allowed, src) if len(little) == 0 { - plan.CPUs = spread(numWorkers, allowed, src.nodeCPUs) + // Alike CPUs. A default count (Workers not set by the user) leaves + // defaultLoopReserve(numWorkers) of the Ps without a loop; a user-set + // count is planned as asked, and so is the big.LITTLE branch below. + loops := numWorkers + if !explicit { + loops -= defaultLoopReserve(numWorkers) + } + plan.CPUs = spread(loops, allowed, src.nodeCPUs) return plan } diff --git a/internal/platform/cpuplan_linux_test.go b/internal/platform/cpuplan_linux_test.go index 30e08fd8f..ebe9fd804 100644 --- a/internal/platform/cpuplan_linux_test.go +++ b/internal/platform/cpuplan_linux_test.go @@ -129,7 +129,7 @@ func TestPlanCapacityVariationAmongBigCPUsIsNotLittle(t *testing.T) { for c, v := range []string{"1024", "866", "909", "960", "1024", "866", "880", "1000"} { h.capacity(c, v) } - p := planWorkerCPUs(8, false, h.src()) + p := planWorkerCPUs(8, true, h.src()) if p.Heterogeneous { t.Fatalf("capacities 866..1024 planned as heterogeneous: little %v", p.Little) } @@ -163,26 +163,26 @@ func TestPlanMaskedAllowedSetStaysInsideTheMask(t *testing.T) { // CPUs 2-5 (celeris#909: the old code pinned CPUs 0-7). allowed := append(rng(0, 1), rng(6, 11)...) h := msr1(t, allowed) - p := planWorkerCPUs(8, false, h.src()) + p := planWorkerCPUs(8, true, h.src()) eq(t, "cpus", p.CPUs, allowed) if p.Heterogeneous { t.Errorf("heterogeneous = true for an all-A720 mask") } // More loops than members wrap inside the set. - eq(t, "cpus(10)", planWorkerCPUs(10, false, h.src()).CPUs, []int{0, 1, 6, 7, 8, 9, 10, 11, 0, 1}) + eq(t, "cpus(10)", planWorkerCPUs(10, true, h.src()).CPUs, []int{0, 1, 6, 7, 8, 9, 10, 11, 0, 1}) } func TestPlanTasksetOnAHomogeneousHost(t *testing.T) { h := newFakeHost(t, rng(4, 7)) // taskset -c 4-7 on an 8-CPU x86 host, no cpu_capacity files - eq(t, "cpus(4)", planWorkerCPUs(4, false, h.src()).CPUs, []int{4, 5, 6, 7}) - eq(t, "cpus(6)", planWorkerCPUs(6, false, h.src()).CPUs, []int{4, 5, 6, 7, 4, 5}) + eq(t, "cpus(4)", planWorkerCPUs(4, true, h.src()).CPUs, []int{4, 5, 6, 7}) + eq(t, "cpus(6)", planWorkerCPUs(6, true, h.src()).CPUs, []int{4, 5, 6, 7, 4, 5}) h = newFakeHost(t, []int{1, 2, 3}) - eq(t, "cpus(3)", planWorkerCPUs(3, false, h.src()).CPUs, []int{1, 2, 3}) + eq(t, "cpus(3)", planWorkerCPUs(3, true, h.src()).CPUs, []int{1, 2, 3}) } func TestPlanX86LikeHomogeneousIsTheOldRule(t *testing.T) { h := newFakeHost(t, rng(0, 7)) - p := planWorkerCPUs(10, false, h.src()) + p := planWorkerCPUs(10, true, h.src()) eq(t, "cpus", p.CPUs, []int{0, 1, 2, 3, 4, 5, 6, 7, 0, 1}) if p.Heterogeneous || p.Note != "" || len(p.Little) != 0 { t.Errorf("homogeneous host: %+v", p) @@ -191,25 +191,25 @@ func TestPlanX86LikeHomogeneousIsTheOldRule(t *testing.T) { for c := 0; c < 8; c++ { h.capacity(c, "1024") } - eq(t, "equal caps", planWorkerCPUs(10, false, h.src()).CPUs, p.CPUs) + eq(t, "equal caps", planWorkerCPUs(10, true, h.src()).CPUs, p.CPUs) } func TestPlanSingleCPU(t *testing.T) { h := newFakeHost(t, []int{5}) - eq(t, "one worker", planWorkerCPUs(1, false, h.src()).CPUs, []int{5}) - eq(t, "three workers", planWorkerCPUs(3, false, h.src()).CPUs, []int{5, 5, 5}) + eq(t, "one worker", planWorkerCPUs(1, true, h.src()).CPUs, []int{5}) + eq(t, "three workers", planWorkerCPUs(3, true, h.src()).CPUs, []int{5, 5, 5}) } func TestPlanMissingOrBrokenCapacityIsHomogeneous(t *testing.T) { // No cpu_capacity files at all. h := newFakeHost(t, rng(0, 3)) - eq(t, "missing", planWorkerCPUs(4, false, h.src()).CPUs, rng(0, 3)) + eq(t, "missing", planWorkerCPUs(4, true, h.src()).CPUs, rng(0, 3)) // One allowed CPU has no file: not enough to classify, assume homogeneous. h = newFakeHost(t, rng(0, 3)) h.capacity(0, "1024") h.capacity(1, "1024") h.capacity(2, "100") - eq(t, "one file missing", planWorkerCPUs(4, false, h.src()).CPUs, rng(0, 3)) + eq(t, "one file missing", planWorkerCPUs(4, true, h.src()).CPUs, rng(0, 3)) // Garbage, zero and negative values. for _, bad := range []string{"", "abc", "0", "-5"} { h = newFakeHost(t, rng(0, 3)) @@ -286,7 +286,7 @@ func TestPlanAllowedErrorFallsBackToTheCPUCount(t *testing.T) { h := newFakeHost(t, nil) src := h.src() src.allowed = func() ([]int, error) { return nil, errors.New("boom") } - p := planWorkerCPUs(3, false, src) + p := planWorkerCPUs(3, true, src) if len(p.CPUs) != 3 || p.CPUs[0] != 0 || p.Heterogeneous { t.Errorf("fallback plan: %+v", p) } @@ -299,21 +299,21 @@ func TestPlanNUMAIntersectsTheMask(t *testing.T) { // CPU gets a loop before one gets a second, and none leaves the mask. h := newFakeHost(t, []int{2, 3, 4, 5, 6}) h.nodes = [][]int{rng(0, 3), rng(4, 7)} - eq(t, "interleave", planWorkerCPUs(5, false, h.src()).CPUs, []int{2, 4, 3, 5, 6}) - eq(t, "wrap", planWorkerCPUs(7, false, h.src()).CPUs, []int{2, 4, 3, 5, 6, 2, 4}) + eq(t, "interleave", planWorkerCPUs(5, true, h.src()).CPUs, []int{2, 4, 3, 5, 6}) + eq(t, "wrap", planWorkerCPUs(7, true, h.src()).CPUs, []int{2, 4, 3, 5, 6, 2, 4}) // A node the mask excludes entirely contributes nothing: its turns do not // fall back to CPUs the other node already used. h = newFakeHost(t, []int{4, 5, 6, 7}) h.nodes = [][]int{rng(0, 3), rng(4, 7)} - eq(t, "one node of two", planWorkerCPUs(4, false, h.src()).CPUs, []int{4, 5, 6, 7}) + eq(t, "one node of two", planWorkerCPUs(4, true, h.src()).CPUs, []int{4, 5, 6, 7}) // No node list intersects the mask (stale sysfs): plain round-robin on the set. h = newFakeHost(t, []int{8, 9}) h.nodes = [][]int{rng(0, 3), rng(4, 7)} - eq(t, "no intersection", planWorkerCPUs(3, false, h.src()).CPUs, []int{8, 9, 8}) + eq(t, "no intersection", planWorkerCPUs(3, true, h.src()).CPUs, []int{8, 9, 8}) // A member no node lists (stale sysfs) comes after the nodes' CPUs. h = newFakeHost(t, []int{2, 3, 9}) h.nodes = [][]int{rng(0, 3), rng(4, 7)} - eq(t, "unlisted member", planWorkerCPUs(4, false, h.src()).CPUs, []int{2, 3, 9, 2}) + eq(t, "unlisted member", planWorkerCPUs(4, true, h.src()).CPUs, []int{2, 3, 9, 2}) } // celeris#909 (review of #973): a cpuset or taskset that covers one NUMA node, @@ -384,7 +384,7 @@ func TestPlanWorkerCPUsOnTheRealHost(t *testing.T) { if err != nil { t.Fatalf("sched_getaffinity: %v", err) } - p := PlanWorkerCPUs(len(allowed)+3, false) + p := PlanWorkerCPUs(len(allowed)+3, true) if len(p.CPUs) != len(allowed)+3 { t.Fatalf("plan has %d entries", len(p.CPUs)) } @@ -457,12 +457,19 @@ func TestPlanUnrestrictedHomogeneousEqualsTheOldPlan(t *testing.T) { h.nodes = nodes } want := oldDistributeWorkers(workers, ncpu, nodes) - // A homogeneous host ignores whether the count was user-set. + // An explicit count is the old plan. A default count is the old + // plan for the loops left after the reserve (celeris#756, #757; + // the figures are written out here, not taken from the planner). for _, explicit := range []bool{false, true} { + wantN := workers + if !explicit && workers > 2 { + wantN -= max(1, workers/8) + } + wantPlan := want[:wantN] got := planWorkerCPUs(workers, explicit, h.src()).CPUs - if !reflect.DeepEqual(got, want) { + if !reflect.DeepEqual(got, wantPlan) { t.Errorf("%s, %d CPUs, %d workers (explicit=%v): got %v, old plan %v", - name, ncpu, workers, explicit, got, want) + name, ncpu, workers, explicit, got, wantPlan) } } } diff --git a/internal/platform/cpuplan_reserve_756_linux_test.go b/internal/platform/cpuplan_reserve_756_linux_test.go new file mode 100644 index 000000000..0dddc349e --- /dev/null +++ b/internal/platform/cpuplan_reserve_756_linux_test.go @@ -0,0 +1,158 @@ +//go:build linux + +package platform + +import ( + "fmt" + "slices" + "testing" +) + +// celeris#756, #757: a DEFAULT loop count (Config.Workers not set) on a host of +// alike CPUs leaves a reserve of Ps to the goroutines that are not loops: the +// plan holds n - max(1, n/8) loops, none for n <= 2. A user-set count is +// honoured, and so is the big.LITTLE plan. These tests pin the POLICY (the +// loop counts), not any latency: the figures are written out and the tests do +// not call defaultLoopReserve, so they compile and fail by assertion on a tree +// without the reserve. + +// A request for n workers, default, on a host whose n CPUs are alike. +func TestPlanDefaultLoopReserve756(t *testing.T) { + for _, tc := range []struct{ n, want int }{ + {0, 0}, {1, 1}, {2, 2}, // no reserve: a count that small would drop to one loop or none + {3, 2}, {4, 3}, {7, 6}, + {8, 7}, + {9, 8}, {15, 14}, + {16, 14}, + {17, 15}, {24, 21}, {31, 28}, + {32, 28}, + {48, 42}, {64, 56}, + } { + t.Run(fmt.Sprintf("n=%d", tc.n), func(t *testing.T) { + h := newFakeHost(t, rng(0, max(tc.n, 1)-1)) + p := planWorkerCPUs(tc.n, false, h.src()) + if p.Heterogeneous { + t.Fatalf("a host of alike CPUs planned as heterogeneous: %+v", p) + } + if len(p.CPUs) != tc.want { + t.Fatalf("default %d workers: %d loops, want %d (plan %v)", tc.n, len(p.CPUs), tc.want, p.CPUs) + } + if p.Requested != tc.n { + t.Errorf("Requested = %d, want %d (what the plan was asked for)", p.Requested, tc.n) + } + if p.Note != "" { + t.Errorf("Note = %q: a homogeneous plan carries none", p.Note) + } + // Every loop on its own CPU, in order; the CPUs left are the last ones. + for i, c := range p.CPUs { + if c != i { + t.Fatalf("loop %d on CPU %d, want CPU %d (plan %v)", i, c, i, p.CPUs) + } + } + }) + } +} + +// The headline case, alone: 16 identical CPUs, default request 16 -> 14 loops +// (rc1 gave 16). The same request with the count set by the user -> 16. +func TestPlanSixteenAlikeCPUs756(t *testing.T) { + h := newFakeHost(t, rng(0, 15)) + got := planWorkerCPUs(16, false, h.src()).CPUs + eq(t, "default 16 -> 14 loops", got, rng(0, 13)) + got = planWorkerCPUs(16, true, h.src()).CPUs + eq(t, "explicit 16 -> 16 loops", got, rng(0, 15)) +} + +// A user-set count is planned exactly as asked, for every count a host can +// ask: this is the opt-out (Config.Workers = runtime.GOMAXPROCS(0) restores +// one loop per P). +func TestPlanExplicitCountIsHonoured756(t *testing.T) { + h := newFakeHost(t, rng(0, 15)) + for n := 1; n <= 40; n++ { + p := planWorkerCPUs(n, true, h.src()) + if len(p.CPUs) != n { + t.Fatalf("explicit %d workers: %d loops", n, len(p.CPUs)) + } + for i, c := range p.CPUs { + if c != i%16 { + t.Fatalf("explicit %d workers: loop %d on CPU %d, want %d", n, i, c, i%16) + } + } + } +} + +// 1..64 default workers on a host of that many CPUs, against the rule written +// out inline: loops = n - max(1, n/8) for n > 2, n otherwise. +func TestPlanDefaultReserveTable756(t *testing.T) { + for n := 1; n <= 64; n++ { + h := newFakeHost(t, rng(0, n-1)) + want := n + if n > 2 { + want = n - max(1, n/8) + } + if got := len(planWorkerCPUs(n, false, h.src()).CPUs); got != want { + t.Errorf("default %d workers: %d loops, want %d", n, got, want) + } + } +} + +// The reserve counts the workers asked for, not the CPUs allowed: a default +// request of 8 (GOMAXPROCS lowered under a CPU limit) on a 16-CPU host runs 7 +// loops on the first 7 CPUs; a request above the CPUs (GOMAXPROCS set above +// them) wraps. +func TestPlanDefaultReserveCountsWorkersNotCPUs756(t *testing.T) { + h := newFakeHost(t, rng(0, 15)) + eq(t, "default 8 of 16 CPUs", planWorkerCPUs(8, false, h.src()).CPUs, rng(0, 6)) + h = newFakeHost(t, rng(0, 3)) + eq(t, "default 8 of 4 CPUs wraps", planWorkerCPUs(8, false, h.src()).CPUs, []int{0, 1, 2, 3, 0, 1, 2}) +} + +// A read error means "homogeneous" (no allowed mask, no capacity files): the +// default count gets the reserve there too, an explicit one does not. +func TestPlanDefaultReserveWithoutAMask756(t *testing.T) { + h := newFakeHost(t, nil) + src := h.src() + src.allowed = func() ([]int, error) { return nil, fmt.Errorf("boom") } + if p := planWorkerCPUs(16, false, src); len(p.CPUs) != 14 || p.Heterogeneous { + t.Errorf("default 16 without a mask: %d loops (%+v), want 14", len(p.CPUs), p) + } + if p := planWorkerCPUs(16, true, src); len(p.CPUs) != 16 { + t.Errorf("explicit 16 without a mask: %d loops, want 16", len(p.CPUs)) + } +} + +// The big.LITTLE plan (msr1: 12 CPUs, CPUs 2-5 little) is untouched: no reserve +// on top of "one loop per non-little CPU", whatever the default count is, so +// 12 asked -> 8 loops (workers=8 requested=12 pinned=8), and a count below the +// big CPUs is kept whole: 7 asked -> 7 loops (a reserve that leaked in would +// make it 6). +func TestPlanBigLittleIgnoresTheReserve756(t *testing.T) { + h := msr1(t, rng(0, 11)) + for _, tc := range []struct { + n int + explicit bool + want int + }{ + {12, false, 8}, // msr1: workers=8 requested=12 pinned=8 + {8, false, 8}, // exactly the big CPUs: a reserve would make it 7 + {7, false, 7}, // below the big CPUs: kept whole + {4, false, 4}, + {3, false, 3}, + {16, false, 8}, + {12, true, 12}, // a user-set count wraps over the big CPUs + {8, true, 8}, + {16, true, 16}, + } { + p := planWorkerCPUs(tc.n, tc.explicit, h.src()) + if !p.Heterogeneous { + t.Fatalf("msr1-like host planned as homogeneous: %+v", p) + } + if len(p.CPUs) != tc.want || p.Pinned() != tc.want { + t.Errorf("msr1-like, %d workers explicit=%v: %d loops (%d pinned), want %d", + tc.n, tc.explicit, len(p.CPUs), p.Pinned(), tc.want) + } + if tc.n == 12 && !tc.explicit && !slices.Equal(p.CPUs, bigCPUs1036) { + t.Errorf("msr1-like default 12: CPUs %v, want %v", p.CPUs, bigCPUs1036) + } + } +} diff --git a/internal/platform/cpuplan_rxfree_linux_test.go b/internal/platform/cpuplan_rxfree_linux_test.go index 701f40844..189effc59 100644 --- a/internal/platform/cpuplan_rxfree_linux_test.go +++ b/internal/platform/cpuplan_rxfree_linux_test.go @@ -264,7 +264,7 @@ func TestRxFreeHomogeneousHostReadsNothing(t *testing.T) { // No capacity files at all: any read error means homogeneous. cpu := newFakeHost(t, rng(0, 3)) h := msr1RxReal(t, "msr1_rxfree_vector_on_cpu1.txt") - got := planWorkerCPUs(4, false, h.planSrc(cpu, clusterAddr)) + got := planWorkerCPUs(4, true, h.planSrc(cpu, clusterAddr)) eq(t, "no capacity files", got.CPUs, rng(0, 3)) if h.reads() != 0 { t.Errorf("no capacity files: %d reads", h.reads()) diff --git a/internal/resource/preset.go b/internal/resource/preset.go index c333c782a..c54c4b2ca 100644 --- a/internal/resource/preset.go +++ b/internal/resource/preset.go @@ -14,6 +14,9 @@ const ( MaxBufferSize = 262144 ) +// resolveDefaults is the resolved default before any user value. Workers is +// GOMAXPROCS, not the loops the native engines start: on Linux they take a +// default count down by a reserve on a host of alike CPUs (celeris#756, #757). func resolveDefaults() ResolvedResources { numCPU := runtime.GOMAXPROCS(0) if numCPU < 1 { diff --git a/internal/resource/resource.go b/internal/resource/resource.go index ce92c041b..99f50f43b 100644 --- a/internal/resource/resource.go +++ b/internal/resource/resource.go @@ -29,12 +29,16 @@ const ( // Resources allows user overrides of default resource values. // Zero values mean "use engine default". type Resources struct { - // Workers is the number of I/O worker goroutines (0 = GOMAXPROCS). Zero - // is also what tells the native engines the count was not user-set: on a - // host whose CPUs differ in capacity they then start at most one pinned loop - // per non-little CPU, min(GOMAXPROCS, non-little CPUs) (celeris#1036); a - // count set here is honoured, wrapping over those CPUs, every loop pinned - // (io_uring still caps it by RLIMIT_MEMLOCK). + // Workers is the number of I/O worker goroutines (0 = GOMAXPROCS; Resolve + // raises it to MinWorkers). Zero is also what tells the native engines the + // count was not user-set: on a host whose CPUs differ in capacity they then + // start at most one pinned loop per non-little CPU, min(GOMAXPROCS, + // non-little CPUs) (celeris#1036), and on a host whose CPUs are alike (Linux) + // they start the resolved count less a reserve of max(1, count/8) loops + // (none for a count of 2 or less) that leaves Ps to the goroutines that are + // not loops (celeris#756, #757): the loops started are what the engines + // report, not Resolve's figure. A count set here is honoured, wrapping over + // those CPUs, every loop pinned (io_uring still caps it by RLIMIT_MEMLOCK). Workers int // BufferSize is the per-connection I/O buffer size in bytes (0 = engine default). BufferSize int