Skip to content

v1.6.0 polish: adaptive engine #886

Open

Description

@FumingPower3925

Consolidated on 2026-10-02 from the per-PR 'Follow-ups from #N' bundles, by maintainer decision. From now on, review nits are fixed inside the PR before it merges, so no new bundles are filed. Each line names the bundle and item it came from; the bundle's thread keeps the full context. Everything here stays in v1.6.0.

Added 2026-10-03 (from #899, the last pre-rule bundle):

Added 2026-10-09 (#908's lane E7, PR #958):

Added 2026-10-09 (lane C1, ramp-quarantine lift, celeris#708 / PR #961):

Added 2026-10-09 (lane E8, PR #967):

Added 2026-10-10 (lane E11, PR #982 advisor check):

Added 2026-10-10 (lane ADDR, PR #986 review r1):

Added 2026-10-11 (lane DFAIR, PR #993; cross-reference):

Added 2026-10-11 (lane E13, PR #992 review r1):

Added 2026-10-13 (lane ADOPT, PR #1015 review r1):

Added 2026-10-16 (lane PINPLAN, PR #1049, celeris#1036):

Activity

  1. added this to the v1.6.0 milestone on Oct 2, 2026
  2. FumingPower3925 commented on Oct 2, 2026

    @FumingPower3925
    ContributorAuthor

    A lead for this checklist, not an established defect: on adaptive, requests still in flight when other clients disconnect were answered 0.5-1.5 s after the stop instant. Seen in goceleris/probatorium#469's real-refapp check, not investigated.

    • Find out why, on adaptive, the requests in flight when 15 other clients are cut at once take so long to be answered, or rule it out before v1.6.0. With ci: compile magefiles in lint #469, adaptive's median from the stop instant to the window opening is 983 ms at the burst end and 875 ms at the load end. On std, epoll and io_uring it is 484-837 ms. Adaptive is also slower than origin/main's whole wind-down in 32/36 and 35/36 cell pairs.

    The setup.

    • The real observability refapp at celeris fe9264f, one engine per cell, each cell a fresh process.
    • probatorium's real driveTier1 drives it with 50 walkers: 35 Markov, 10 adversarial, 5 h2c.
    • One linux/arm64 container on the maintainer's laptop, inside a laptop-lock slot but not under the timing lock. 6 cells per engine per arm.
    • With ci: compile magefiles in lint #469, each phase ends this way: the 15 adversarial and h2c clients are cut at the stop instant, and the next idle window opens only once the Markov requests already in flight have been answered (bounded by the 5 s request timeout). On origin/main, those Markov requests were cut at the same instant too.

    Stop instant to window opening, ms, 6 cells each (origin/main -> #469):

    engine burst end, median load end, median cell pairs where #469 is slower (burst; load)
    std 713 -> 837 560 -> 598 20/36; 22/36
    epoll 809 -> 807 483 -> 555 20/36; 21/36
    io_uring 782 -> 657 645 -> 484 8/36; 11/36
    adaptive 484 -> 983 387 -> 875 32/36; 35/36
    • Adaptive per cell, with ci: compile magefiles in lint #469: burst end 710, 759, 807, 1159, 1512, 1525; load end 488, 827, 864, 887, 893, 1483.
    • Adaptive per cell, origin/main: burst end 304-1266; load end 277-509.
    • On the other three engines the two arms overlap, because the other slices' wind-down dominates both. On adaptive the in-flight Markov requests are the long pole.

    Not established.

    • Whether adaptive switched engines, or did anything else, at those moments: nothing recorded it.
    • Whether a real client would see this: the measurement is the harness's window opening, not a per-request latency.
    • 6 observations per arm in one container is a lead, not a measurement.

    Rerun (on the maintainer machine, under evidence/lanes-20261002/LP465_466/): scripts/p465real/build.sh, scripts/p465real/run.sh r1 6, then python3 scripts/p465real/phase_end_times.py r1. That regenerates logs/p465real/r1/phase_end_times.txt byte-identical. A search of celeris, probatorium, loadgen and docs found no issue for this.

  3. FumingPower3925 commented on Oct 3, 2026

    @FumingPower3925
    ContributorAuthor

    One item from the round-2 review of #906 (celeris#893, lane F3, correctness lens). Found by reading the code; not measured:

    • With AsyncHandlers: true, a peer that withholds HTTP/2 window can make the adaptive learner promote any adaptive route. handler.go:141-150 (main 1b78612) times an adaptive route's request around c.Next() and calls promoteRouteImmediate when it takes longer than adaptiveBlockingThreshold (2 ms). With fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906, a request that runs on the HTTP/2 worker pool because its connection is over the outbound budget blocks in its response write (internal/conn/response.go sendRest) until the peer grants window, for up to WriteTimeout. That wait is inside the timed c.Next(). So a peer that withholds window can switch an adaptive route to async dispatch for every connection for adaptivePromoteTTL (5 s), and again each time it expires. On main a write never blocks, so flow control could not do this. It costs performance only: the route runs on the slower async path.

      Fix direction: exclude the time a write waits for the peer's window from the learner's sample, or skip the sample for a stream on the pool only for the budget (flagBudgetPool). Either needs a test that drives a waiting write through an adaptive route and checks the route is not promoted.

  4. FumingPower3925 commented on Oct 3, 2026

    @FumingPower3925
    ContributorAuthor

    Two items from the round-2 reviews of #916 (celeris#852, lane SW), on the dispatch of requests no route matches. Lines are at #916's head b4e7900. Neither is a regression against main, where an unmatched request ran no middleware. #916's godoc (Config.AsyncHandlers, Server.Use) and goceleris/docs#90 (configuration.md, middleware.md) now describe the current behaviour. Update them with the fix. Evidence paths are relative to evidence/lanes-20261003/SW/.

    • The unmatched chain has one adaptive key for every unmatched request, so fast unmatched traffic settles it, and after that a blocking unmatched request runs inline.
      • Mechanism: HandleStream times an unmatched request only while unmatchedAdaptiveKey is learning (handler.go:236-243). routeAsync answers unmatchedPromoted() for a path no route matches (router.go:592-595, :624).
      • 256 fast unmatched runs in a row settle the key: scanner 404s, a Use-mounted healthcheck, the swagger page and its assets. A settled key is neither timed nor promoted.
      • So an unmatched request whose chain blocks runs inline on the engine worker every time, until the settle re-opener (every 5 s) happens to time a blocking run. Examples are pprof's 30 s profile and an auth or session middleware that blocks only for some requests.
      • A route has the same property, but it has its own key and an .Async() opt-out. A Use mount has neither.
      • Measured with the correctness reviewer's probe review-correctness-r2/probes/linux/zz_r2_settled_key_linux_test.go: golang:1.27, Workers 2, AsyncHandlers, and a waiting unmatched request that holds for up to 3 s while 16 fresh connections ask GET /ping. I reran it unchanged at b4e7900 (fix-r2/scripts/05-ctr-linux.sh, fix-r2/logs/05-ctr-b4e7900-53dff42/settled-{1,2,3}.log):
        • one blocking unmatched run, then 300 fast 404s: 16/16 pings on epoll and io_uring;
        • 300 fast 404s first: epoll 10, 8 and 9 of 16, io_uring 0, 0 and 0 of 16. The reviewer had 8, 8, 10 and 0, 0, 0 at 79b233e.
      • The reviewer's direction check (review-correctness-r2/logs/ctr3-nosettle.out), on a tree whose key cannot settle: 16/16 in all 12 arm-runs.
      • Fix option (the reviewer's): never settle unmatchedAdaptiveKey, so it stays on the timed path. That costs about two clock reads per unmatched request under AsyncHandlers with global middleware, so measure it.
      • Add with it the guard the reviewer found missing. No test pins the unmatched key's settle and slow-streak path: a mutant that drops recordInlineRun for fast unmatched runs (so the key never settles and the 300 µs-2 ms streak promotion is lost) passes all 37 of the Use-mounted swagger/pprof/debug answer 404 for their own paths: global middleware is skipped on unmatched routes unless a custom NotFound is set #852 tests (review-correctness-r2/logs/916-darwin.out, arm m-settled-never).
    • Under the sync default with .Async() routes, the unmatched chain is never adaptive.
      • Mechanism: buildUnmatchedChains sets router.unmatchedAdaptive only when the server default is async (handler.go:72-75). So RouteAsync answers false for every unmatched path, even after a run blocks, while async dispatch is up (HasAsyncRoutes true).
      • Effect: a blocking global middleware (a Redis-backed ratelimit or session store, an auth upstream) runs inline on the engine worker for every 404 and 405. A route can be marked .Async(); the unmatched chain cannot.
      • Measured at adapter level with the perf/API/security reviewer's probe zz_r2_dispatch_decision_test.go, rerun at b4e7900 (fix-r2/scripts/02-verify-probes-darwin.sh → fix-r2/logs/02-verify/dispatch-h916.log): sync-default+async-routes: HasAsyncRoutes=true RouteAsync(GET /ping)=true RouteAsync(GET /nope) before=false after one 22ms blocking unmatched run=false.
      • Not measured on the wire: the reviewer's container probe (review-perf-api-security-r2/scripts/r2-06-inline-syncdefault-ctr.sh) is written but was not run.
      • Suggested fix: make the unmatched chain adaptive whenever async dispatch exists and the chain runs more than the built-in answer: (defaultAsync || asyncRouteCount > 0) && (…). This changes which requests are timed under the sync default, so measure it.
      • Keep the write conditional. fix(api): the global middleware runs for a request no route matches, so Use-mounted swagger, pprof, debug, healthcheck, metrics and static answer their paths (celeris#852) #916's round 2 writes the flag only when it is true, because Server.AsyncHandlers reads it when no route is async (the data race fixed there; TestAsyncHandlersReadWhileStarting852).
  5. 68 remaining items

  6. FumingPower3925 commented on Nov 1, 2026

    @FumingPower3925
    ContributorAuthor

    Verified on main (2026-11-01) and confirmed by the maintainer; boxes ticked: 1 (the #592 rig now fails on a loop goroutine parked on a mutex instead of the stalled fraction: substitution accepted, #1027), 8 (RequestCount rule after adoption: a non-empty carry counts one, an h2c-upgrade carry none, #1015 then #1127; 16 tests pass across both engines), 18 (ctx-aware lockForShutdown cuts the bind wait, #1019 and #1038; the CI guard runs in the non-required adaptive job). Evidence: evidence/_loop/verify-polish-20261101.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/engineEngine interface or implementation

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions