Repository navigation
feat(engine): default loops leave a reserve of Ps on hosts of alike CPUs #1301
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -120,7 +120,27 @@ type Config struct { | |
| // Engine is the I/O engine (default Adaptive on Linux, Std elsewhere). | ||
| Engine EngineType | ||
|
|
||
| // Workers is the number of I/O worker goroutines (default GOMAXPROCS). | ||
| // Workers is the number of I/O worker goroutines (the loops of the native | ||
| // engines). Default: GOMAXPROCS, less a reserve on Linux: with Workers unset | ||
| // the native engines (epoll, io_uring, adaptive) on a host whose allowed | ||
| // CPUs are alike start GOMAXPROCS - max(1, GOMAXPROCS/8) loops (no reserve | ||
| // for a GOMAXPROCS of 2 or less): 28 on 32, 14 on 16, 7 on 8, 3 on 4, 2 on | ||
| // 3. A loop locks its thread and can block in the kernel while it holds its | ||
| // P, so the Ps that run no loop are left to the goroutines that are not | ||
| // loops (async and WebSocket handlers, the HTTP/2 stream pool, the garbage | ||
| // collector's workers). That mechanism is inferred from traces, not | ||
| // established; the reserve is provisional, its effect on latency is still | ||
| // being measured, and it may change (celeris#756, #757). A Workers you set | ||
| // is honoured as it is, so to opt out of the reserve set Workers to | ||
| // runtime.GOMAXPROCS(0): on such a host that starts one loop per P, as | ||
|
Comment on lines
+134
to
+135
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| // before the reserve (on a big.LITTLE host the default is not reduced, see | ||
| // below, and a Workers set above the non-little CPUs wraps loops over | ||
| // them). What follows the loop count follows the loops started: Metrics.Workers, | ||
| // the server-wide limit of MaxConns, and a driver pool sized from the | ||
| // engine's NumWorkers (a PostgreSQL pool opened with WithEngine, whose | ||
| // default of 4*NumWorkers is 56, not 64, on 16 CPUs). A pool sized from | ||
| // GOMAXPROCS or the standalone driver loop (Redis commands, Memcached with | ||
| // the native engines) is unchanged. | ||
| // | ||
| // On Linux the native engines (epoll, io_uring) pin each loop to one CPU | ||
| // the process is allowed to use. On a host whose allowed CPUs differ in | ||
|
|
@@ -225,8 +245,9 @@ type Config struct { | |
| // host, one observation); an automatic extra loop on the little CPU was | ||
| // tried and rejected because it raised tail latency on other workloads. | ||
| // Moving the interrupt is the host operator's call (README, "NIC interrupts | ||
| // on big.LITTLE hosts"). Hosts whose CPUs are alike are unchanged and read no | ||
| // NIC file (README, "CPU pinning"). | ||
| // on big.LITTLE hosts"). Hosts whose CPUs are alike read no | ||
| // NIC file; the reserve above is their only difference (README, "CPU | ||
| // pinning"). | ||
| Workers int | ||
|
|
||
| // WorkloadHint optionally declares the expected steady-state concurrency. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| //go:build linux | ||
|
|
||
| package adaptive | ||
|
|
||
| import ( | ||
| "testing" | ||
|
|
||
| "github.com/goceleris/celeris/internal/platform" | ||
| ) | ||
|
|
||
| // celeris#756, #757: startedLoops, which the adaptive engine uses to size the | ||
| // io_uring memlock check, counts the loops the engines start: the default | ||
| // count on a host of alike CPUs is the request less the reserve of | ||
| // max(1, n/8) (none at 2 or less), a user-set count is the request. The real | ||
| // planner, no stub; the figures are written out. | ||
| func TestStartedLoopsLeaveAReserve756(t *testing.T) { | ||
| if platform.PlanWorkerCPUs(12, true).Heterogeneous { | ||
| // Not a t.Skip: skip_routing_684_test.go polices every skip of this package. | ||
| t.Log("this host's CPUs differ in capacity: the big.LITTLE plan is not the reserve's; nothing to check") | ||
| return | ||
|
Comment on lines
+19
to
+20
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do not report an unexecuted test as PASS. On a heterogeneous host, this return bypasses every assertion and reports PASS. Avoiding Use a tracked skip with a As per path instructions: “A SKIP is never a PASS.”
|
||
| } | ||
| for _, tc := range []struct { | ||
| requested int | ||
| explicit bool | ||
| want int | ||
| }{ | ||
| {16, false, 14}, {16, true, 16}, {8, false, 7}, {32, false, 28}, | ||
| {4, false, 3}, {3, false, 2}, {2, false, 2}, {1, false, 1}, {5, true, 5}, | ||
| } { | ||
| if got := startedLoops(tc.requested, tc.explicit); got != tc.want { | ||
| t.Errorf("startedLoops(%d, explicit=%v) = %d, want %d", tc.requested, tc.explicit, got, tc.want) | ||
| } | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| //go:build linux | ||
|
|
||
| package epoll | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "runtime" | ||
| "strings" | ||
| "testing" | ||
| "time" | ||
|
|
||
| "github.com/goceleris/celeris/internal/platform" | ||
| "github.com/goceleris/celeris/internal/platform/pintest" | ||
| ) | ||
|
|
||
| // celeris#756, #757: an epoll engine whose worker count is not set by the user | ||
| // starts GOMAXPROCS - max(1, GOMAXPROCS/8) loops on a host of alike CPUs (none | ||
| // for a GOMAXPROCS of 2 or less; Resolve raises a count to two), and a count | ||
| // the user sets is started whole. This is the real planner and real engine, no | ||
| // stub: Metrics().Workers, NumWorkers(), the loops and the "listening" line | ||
| // all read the loops started. The test pins the policy, not any latency. | ||
| // | ||
| // GOMAXPROCS is set inside the test's own process, so the request is the same | ||
| // on any runner (the figures are written out, not computed from the planner). | ||
| func TestDefaultLoopsLeaveAReserve756(t *testing.T) { | ||
| if !pintest.InOwnProcess(t) { | ||
| pintest.RunInOwnProcess(t, 3*time.Minute) | ||
| return | ||
| } | ||
| if p := platform.PlanWorkerCPUs(12, true); p.Heterogeneous { | ||
| t.Skip("this host's CPUs differ in capacity: the big.LITTLE plan is not the reserve's") | ||
| } | ||
| prev := runtime.GOMAXPROCS(0) | ||
| defer runtime.GOMAXPROCS(prev) | ||
| for _, tc := range []struct { | ||
| gomaxprocs int | ||
| workers int // Resources.Workers; 0 = the default | ||
| want int // loops started | ||
| }{ | ||
| {16, 0, 14}, // the default: 16 Ps, 14 loops (rc1: 16) | ||
| {16, 16, 16}, // Workers = GOMAXPROCS: one loop per P, as before | ||
| {16, 5, 5}, // a user-set count is kept whole | ||
| {16, 20, 20}, // also above GOMAXPROCS | ||
| {8, 0, 7}, | ||
| {4, 0, 3}, | ||
| {3, 0, 2}, | ||
| {2, 0, 2}, // no reserve at two | ||
| {1, 0, 2}, // Resolve raises the default to MinWorkers (2); no reserve at two | ||
| } { | ||
| t.Run(fmt.Sprintf("GOMAXPROCS=%d/Workers=%d", tc.gomaxprocs, tc.workers), func(t *testing.T) { | ||
| runtime.GOMAXPROCS(tc.gomaxprocs) | ||
| var log lockedBuf | ||
| eng, stop := start909(t, tc.workers, &log) | ||
| defer stop() | ||
| if got := len(eng.loops); got != tc.want { | ||
| t.Errorf("%d loops started, want %d", got, tc.want) | ||
| } | ||
| if got := eng.Metrics().Workers; got != tc.want { | ||
| t.Errorf("Metrics().Workers = %d, want %d", got, tc.want) | ||
| } | ||
| if got := eng.NumWorkers(); got != tc.want { | ||
| t.Errorf("NumWorkers() = %d, want %d", got, tc.want) | ||
| } | ||
| if got := listeningLine1036(t, &log); !strings.Contains(got, fmt.Sprintf(" loops=%d", tc.want)) { | ||
| t.Errorf("the listening line does not say loops=%d:\n%s", tc.want, log.String()) | ||
| } | ||
| }) | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: goceleris/celeris
Length of output: 12253
Script executed:
Repository: goceleris/celeris
Length of output: 41479
Script executed:
Repository: goceleris/celeris
Length of output: 13837
Script executed:
Repository: goceleris/celeris
Length of output: 15764
Qualify the io_uring loop count.
When
RLIMIT_MEMLOCKcaps io_uring, it can start fewer loops than the formula states. Add this qualification toConfig.Workersand README.md:424. It also applies to adaptive when it starts io_uring through the override.