From b53e1143d670f440d7198c65dc5c442aa9344148 Mon Sep 17 00:00:00 2001 From: Albert Bausili Date: Wed, 4 Nov 2026 02:26:50 +0100 Subject: [PATCH 1/3] docs: bring the pages up to date with celeris v1.6.0 (timeouts, shutdown contract, busy poll, drivers) The v1.6.0 changes that landed after the last docs sweep, each checked against celeris main (42f096e0): TCP_DEFER_ACCEPT and DisableDeferAccept, Config.SocketBusyPoll (off by default) and how the drivers follow it, the epoll 25 ms sweep timer, the monotonic timeout clock, the per-engine Shutdown contract by connection state, Connection: close after Shutdown, the stuck-loop wedge bound, and the redis, postgres and memcached driver timeouts and lifecycle fixes. Also corrects three statements the code does not match: the epoll sweep interval (25 ms, not 0.1 s), the HTTP/2 window wait being an absolute WriteTimeout deadline (it is a minimum rate), and PauseAccept/ResumeAccept working on adaptive (they return ErrAcceptControlNotSupported there). --- src/content/docs/configuration.md | 103 ++++++++++++++++- src/content/docs/core-concepts.md | 39 +++++-- src/content/docs/data-stores.md | 155 ++++++++++++++++++++++++- src/content/docs/engines.md | 49 +++++++- src/content/docs/graceful-shutdown.md | 158 +++++++++++++++++++++++--- src/content/docs/performance.md | 15 +++ src/content/docs/sse.md | 2 +- 7 files changed, 490 insertions(+), 31 deletions(-) diff --git a/src/content/docs/configuration.md b/src/content/docs/configuration.md index f89fc41..4c1c66e 100644 --- a/src/content/docs/configuration.md +++ b/src/content/docs/configuration.md @@ -201,6 +201,65 @@ of 10s defeats slow-loris while still letting legitimate proxies and high-latenc inside their H1 header read loop (`celeris/config.go` (`Config.ReadHeaderTimeout`), `celeris/internal/resource/config.go` (`Config.ReadHeaderTimeout`)). +On a connection that sends nothing, the budget starts when the engine accepts it, and +on `epoll`, `io_uring` and `adaptive` that is later than the connect: +[`TCP_DEFER_ACCEPT`](#connection-socket-and-io) is on by default on their listen +sockets, so the kernel hands the engine a connection when its first byte arrives, or +about a second after its SYN if it stays silent. The bound on a silent connection is +its connect, plus about one second of deferral (measured 1.00 to 1.07 s), plus +`ReadHeaderTimeout`, plus the lateness of the engine's check (about 30 ms on `epoll` +and `io_uring`, measured on docker arm64 with 2 CPUs). `std` counts from the handshake. +With `DisableDeferAccept` the deferral is gone: a silent connection is closed at +`ReadHeaderTimeout` from its connect (a `ReadHeaderTimeout` of one second closed it at +1.00 to 1.03 s). Source: `celeris/config.go` (`Config.ReadHeaderTimeout`, +`Config.DisableDeferAccept`), [celeris#1193](https://github.com/goceleris/celeris/pull/1193). + +### How promptly the native engines enforce a timeout + +`epoll`, `io_uring` and `adaptive` do not arm one timer per connection for +`ReadTimeout`, `IdleTimeout` and `WriteTimeout`; they look at their connections in a +sweep, so each is enforced up to a sweep period after its time. Source: +`celeris/config.go` (`Config.ReadTimeout`). + +- **`epoll`: every 25 ms, whatever the timeouts are.** Each worker arms a 25 ms kernel + timer (a `timerfd`) at start, so a timeout fires within about 30 ms (a 1 s + `IdleTimeout` closed an idle connection at 1.00 to 1.03 s on docker arm64 with 2 + CPUs). Since v1.6.0 that holds with `ReadHeaderTimeout: -1` too. Before, the timer was + armed only when `ReadHeaderTimeout` was set, and without it a quiet loop swept every + 4 to 5 s: a 1 s `IdleTimeout` closed at 4.4 to 5.0 s + ([celeris#1197](https://github.com/goceleris/celeris/pull/1197), the numbers are that + PR's measurement). `adaptive` follows `epoll` while it runs on it. +- **`io_uring`: about 0.8 s apart on an idle worker** (0.83 s measured; about 1 s with + `ReadHeaderTimeout` off), so a timeout of a few seconds or less is approximate by + that much. The HTTP/1 header deadline and the budget of a connection whose protocol + is not decided yet have a timer of their own on `io_uring` and are not late; the + HTTP/2 header-block bound is checked by the sweep and is. + +**The clock is monotonic.** Since v1.6.0 the native engines stamp their timeouts from +a clock that a step of the wall clock does not move +([celeris#1188](https://github.com/goceleris/celeris/pull/1188), `celeris/internal/mono`): +the idle, read, write, header and protocol-detection timeouts, the sweep, the HTTP/1 +header deadline, the WebSocket idle deadline and the HTTP/2 per-stream deadlines, and, +since [celeris#1177](https://github.com/goceleris/celeris/pull/1177), the shutdown +deadlines. An NTP step, a VM clock sync or a clock stepped repeatedly no longer closes +connections that are making progress, and no longer keeps timed-out ones open. Before, +those stamps were wall-clock readings (the PR reports a 32 MiB response cut at 2.7 MB +by repeated 75 s steps). `Context.StartTime()` carries a monotonic reading, +so `time.Since(c.StartTime())` ignores steps; `SetWSIdleDeadline` still takes a +wall-clock Unix time in nanoseconds. Not covered: the `std` engine, which uses +`net/http`'s own timeouts, and application time, which stays wall-clock (the rate +limiter, cache, session and JWT expiry, the stores). + +> **A suspended host.** Linux's monotonic clock (`CLOCK_MONOTONIC`, which Go reads) +> does not advance while the machine is suspended, so a connection that is idle across +> a suspend times out after its full timeout has elapsed *awake*, not at the first +> sweep after the resume (`IdleTimeout` is 600 s by default). The same holds for a +> process restored from a checkpoint (CRIU without a time namespace) onto a host with +> less uptime: its monotonic clock goes backward and its timeouts wait until it catches +> up. The PR says suspend and resume were not verified on real hardware, and that VM live +> migration and snapshot restore typically keep the guest's monotonic clock continuous +> (kvmclock, Firecracker), which it did not verify either. + ### SSE needs no `-1`; other long-lived responses do An SSE stream is a **detached** stream (the middleware calls @@ -213,7 +272,7 @@ end a stream: | Engine | What a detached stream (SSE) sees | | ------ | --------------------------------- | | `std` | `net/http` arms one write deadline for the whole response, and `Detach` clears it for that request. Before celeris#855 it stayed armed and cut the stream at `WriteTimeout`, over HTTP/1.1 (the next `Send` failed) and over h2c (the stream was reset with `INTERNAL_ERROR`). `ReadTimeout` never cut a stream on Go 1.27: `net/http` has stopped it by the time the handler runs. In its place each write of the detached stream (a `Write` or `Flush`) is bounded by `WriteTimeout`: one that has not finished within it fails. For an SSE event, which is small, that means the client has stopped reading; a stream that is idle between events is not cut, and `-1` sets no such bound. | -| `epoll`, `io_uring`, `adaptive` | `ReadTimeout` and `IdleTimeout` are not applied to a detached connection. `WriteTimeout` is, as "no progress for `WriteTimeout`": a detached connection with bytes queued for its peer, which takes none of them for `WriteTimeout`, is closed (the stream's context ends and `OnDisconnect` runs), whatever the middleware's idle deadline or the last read of the connection says. A peer that reads, however slowly (there is no minimum rate), and a connection with nothing queued are not closed; `-1` sets no such bound. The close comes between about 3/4 and 5/4 of `WriteTimeout` after the peer last took a byte (its progress is sampled every quarter of `WriteTimeout`; a send completion restarts the clock at once), plus the engine's sweep interval (up to about 0.1 s on `epoll`, up to 2 s on `io_uring`; a `WriteTimeout` shorter than that can double) ([celeris#888](https://github.com/goceleris/celeris/issues/888) items 38 and 39). | +| `epoll`, `io_uring`, `adaptive` | `ReadTimeout` and `IdleTimeout` are not applied to a detached connection. `WriteTimeout` is, as "no progress for `WriteTimeout`": a detached connection with bytes queued for its peer, which takes none of them for `WriteTimeout`, is closed (the stream's context ends and `OnDisconnect` runs), whatever the middleware's idle deadline or the last read of the connection says. A peer that reads, however slowly (there is no minimum rate), and a connection with nothing queued are not closed; `-1` sets no such bound. The close comes between about 3/4 and 5/4 of `WriteTimeout` after the peer last took a byte (its progress is sampled every quarter of `WriteTimeout`; a send completion restarts the clock at once), plus the engine's sweep interval (25 ms on `epoll`, up to 2 s on `io_uring`; a `WriteTimeout` shorter than that can double) ([celeris#888](https://github.com/goceleris/celeris/issues/888) items 38 and 39). | What bounds a detached stream is what its owner sets: the handler returning, the client closing the connection, a shutdown. A client that stays connected but stops @@ -362,18 +421,57 @@ the floor. | Field | Type | Default | Constraint | Effect | | ------------------ | ------ | -------------- | ----------------------- | ---------------------------------------------- | | `DisableKeepAlive` | `bool` | `false` | — | One request per connection when `true` | +| `DisableDeferAccept` | `bool` | `false` | `epoll` / `io_uring` only | Turns `TCP_DEFER_ACCEPT` off on the listen sockets | | `BufferSize` | `int` | `8192` | `>= 4096`; clamped to `262144` | Per-connection I/O buffer size in bytes | | `SocketRecvBuf` | `int` | OS default (0) | — | `SO_RCVBUF` for accepted connections | | `SocketSendBuf` | `int` | OS default (0) | — | `SO_SNDBUF` for accepted connections | +| `SocketBusyPoll` | `time.Duration` | `0` (off) | `0`, or `1µs` to `2147483647µs`; Linux native engines | `SO_BUSY_POLL` for accepted connections | | `MaxConns` | `int` | unlimited (0) | — | Max simultaneous connections **per worker** | -Sources: `celeris/config.go` (`Config.DisableKeepAlive`, `Config.BufferSize`, `Config.SocketRecvBuf`, `Config.SocketSendBuf`, `Config.MaxConns`), +Sources: `celeris/config.go` (`Config.DisableKeepAlive`, `Config.DisableDeferAccept`, `Config.BufferSize`, `Config.SocketRecvBuf`, `Config.SocketSendBuf`, `Config.SocketBusyPoll`, `Config.MaxConns`), `celeris/internal/resource/resource.go` (`Resources`), `celeris/internal/resource/preset.go` (`MinBufferSize`). - **`DisableKeepAlive`** — when `true`, each request runs on its own connection and the connection closes after the response. Leave `false` (the default) for performance; keep-alive is what makes the engine fast. +- **`DisableDeferAccept`** (new in v1.6.0) — `epoll` and `io_uring` (and `adaptive`, + which runs them) set `TCP_DEFER_ACCEPT` on their listen sockets, and it stays on by + default. While it is on, the kernel keeps a connection that has finished its + handshake but sent no data out of the accept queue: a client that sends its request + at once is accepted with the request already queued, which saves the engine a wakeup + per new connection, and a client that stays silent is accepted about one second after + its SYN, and is then subject to `ReadHeaderTimeout` like any other (see + [`ReadHeaderTimeout`](#readheadertimeout-is-your-slow-loris-defence)). Setting + `DisableDeferAccept: true` gives up that saved wakeup, a cost on connection churn + only: keep-alive traffic, where one accept serves many requests, does not see it. It + also changes [`PauseAccept`](/docs/graceful-shutdown#pause-and-resume-accept), which + carries a not-yet-sent client across a pause by lingering about 1.5 s with the option + cleared: with `DisableDeferAccept` the pause closes the listen sockets at once. +- **`SocketBusyPoll`** (new in v1.6.0) — turns on `SO_BUSY_POLL`, Linux's per-socket + busy polling of the network device's receive queue (NAPI), for the connections the + `epoll`, `io_uring` and `adaptive` engines accept. **It is off by default.** Before + v1.6.0 the engines set it to 50 µs on every accepted socket with no way to turn it + off; `SocketBusyPoll: 50 * time.Microsecond` restores that + ([celeris#1036](https://github.com/goceleris/celeris/issues/1036), + [celeris#1227](https://github.com/goceleris/celeris/pull/1227)). With it set, a read + that finds a socket empty first runs one pass of the device's receive polling on its + CPU. The engines' sockets are non-blocking, so the value only switches that on: any + positive value behaves alike (a fraction of a microsecond is dropped). A negative + value or one under 1 µs is a [configuration error](#validation-behaviour). The + `std` engine, other operating systems and the unsupported io_uring fixed-files mode + (`CELERIS_IOURING_FIXED_FILES`) ignore it. The kernel needs + `CONFIG_NET_RX_BUSY_POLL`; where it refuses the option the connections are served + without it and a Warn is logged once per process. **Why it is off:** the maintainers + measured throughput 3.0% higher with it off on an arm64 big.LITTLE host with a + single-queue NIC (with `ksoftirqd` taking over receive processing in 5 of 32 runs + with it on and in none with it off) and 1.1% higher on an amd64 host with a 16-queue + NIC. Those numbers are for accepted sockets with 512 connections of GET on `io_uring`, + the stall result is indicative, and the mechanism is inferred, not observed. Set it + only where you have measured a gain for your own traffic and NIC, and keep the NIC's + interrupt vectors off the CPUs that run loops when you do. The Redis, PostgreSQL and + Memcached drivers opened `WithEngine(srv)` follow this value; see + [Stores and database drivers](/docs/data-stores#busy-polling-for-driver-connections). - **`BufferSize`** — the per-connection I/O buffer. Zero uses the engine default (8192 bytes, `celeris/internal/resource/preset.go` (`resolveDefaults`)). If you set it, a value below 4096 is a config error (`bufferSize must be >= 4096 if set`, @@ -600,6 +698,7 @@ Common errors under the `config validation:` prefix (`celeris/internal/resource/ | `initialWindowSize must be 0-2147483647` | `InitialWindowSize` too large | | `maxConcurrentStreams must be <= 2147483647` | `MaxConcurrentStreams` too large | | `readTimeout must be >= -1` (also write/idle) | A timeout set below `-1` | +| `socketBusyPoll must be 0 (off) or between …` | `SocketBusyPoll` negative, under 1µs, or over 2147483647µs | `TrustedProxies` is parsed *after* validation passes and surfaces its own error (not under the `config validation:` prefix): `celeris: invalid TrustedProxies entry: …` diff --git a/src/content/docs/core-concepts.md b/src/content/docs/core-concepts.md index 3f4fc35..1f6346c 100644 --- a/src/content/docs/core-concepts.md +++ b/src/content/docs/core-concepts.md @@ -321,14 +321,37 @@ of response data that its client has not yet granted flow-control window for that much, a new stream's handler runs on the HTTP/2 worker pool, **even on a sync route**, and its response write (`c.JSON`, `c.Blob`, `c.File`, ...) can block until the client sends `WINDOW_UPDATE`, as a `net/http` handler's `Write` -does. The wait lasts at most `Config.WriteTimeout` (60 s by default) from the -write, for the whole response: progress does not extend it, so a client that -grants window too slowly is cut off too. Then the stream is reset (`RST_STREAM`, -`INTERNAL_ERROR`), the request's context is done, and the write returns an error -that wraps `os.ErrDeadlineExceeded`. With `WriteTimeout: -1` there is no -deadline: the wait lasts until the client grants window or the connection -closes, and never ends if the I/O worker that would see either is the one -waiting (below). +does. `Config.WriteTimeout` (60 s by default) bounds that wait as a **minimum rate**, +not as an absolute deadline +([celeris#1167](https://github.com/goceleris/celeris/pull/1167), +[celeris#1180](https://github.com/goceleris/celeris/pull/1180)): in each +`WriteTimeout` window, a stream that holds bytes waiting for the window must have had +at least 64 KiB of them (or all it held, if less) sent, released by the client's +`WINDOW_UPDATE`s. If it did not, the stream is reset (`RST_STREAM`, `INTERNAL_ERROR`), +the request's context is done, and the write returns an error that wraps +`os.ErrDeadlineExceeded`. There is no ceiling: a stream that releases at least that +on average is not cut however long its response takes, and a `PING`, or a +`WINDOW_UPDATE` of one byte, is not progress. Clients on slow links need a +`WriteTimeout` of 2 s or more, and 5 s where the links are slower than a megabyte a +second. With `WriteTimeout: -1` there is no deadline: the wait lasts until the client +grants window or the connection closes, and never ends if the I/O worker that would +see either is the one waiting (below). Source: `celeris/config.go` +(`Config.WriteTimeout`). + +**If the event loop itself is stuck, the rate rule is replaced by a bound.** While an +event loop of the server has been in one batch of events for `max(WriteTimeout/2, 2 s)` +(a synchronous handler that waits for a lock a waiting handler holds across its +response write does this, and so does any slow blocking call), the rate cannot be +judged, and every such wait that has lasted a `WriteTimeout` is bounded by +`WriteTimeout` in all, as before the rate rule: at most `max(WriteTimeout/2, 2 s) + WriteTimeout` from the start of the freeze +(1.5 `WriteTimeout`s for a `WriteTimeout` of 4 s or more; the default's threshold is +30 s). The 2 s floor +([celeris#1202](https://github.com/goceleris/celeris/pull/1202)) matters only for a +`WriteTimeout` under 4 s: a healthy loop of an overloaded or instrumented host has +batches of a second or two, and a 250 ms floor cut healthy downloads there. The server +logs a Warn once per episode ("the minimum rate of a pool handler's wait for window is +suspended") and the reason of each such reset at Debug level (`rate`, or +`loop-blocked`). So do not hold a lock across a response write that a sync handler also takes. The sync handler would wait for that lock on its I/O worker, and every diff --git a/src/content/docs/data-stores.md b/src/content/docs/data-stores.md index 3ef9967..da8662d 100644 --- a/src/content/docs/data-stores.md +++ b/src/content/docs/data-stores.md @@ -303,14 +303,80 @@ is reachable via `rdb.Do(ctx, args...)`. Common options | `WithPassword` / `WithUsername` | AUTH credentials (ACL on Redis 6+) | none | | `WithDB(n)` | `SELECT` database index | 0 | | `WithPoolSize(n)` | Total connection cap | `NumWorkers * 4` | -| `WithDialTimeout(d)` | TCP dial timeout | 5s | +| `WithDialTimeout(d)` | Bounds the TCP connect and, separately, the connection handshake (`HELLO`, then `AUTH` and `SELECT`) | 5s | | `WithProto(2\|3)` | RESP version target | 3 (RESP2 fallback) | | `WithForceRESP2()` | Skip `HELLO`; speak RESP2 (ElastiCache classic) | off | +| `WithSocketBusyPoll(d)` | `SO_BUSY_POLL` for the connections the driver puts on an event loop; see [busy polling](#busy-polling-for-driver-connections) | off | + +Since v1.6.0 `DialTimeout` bounds the handshake as well as the TCP connect +([celeris#1209](https://github.com/goceleris/celeris/pull/1209)). Before, only the +connect was bounded, so a node that accepted the connection and never answered `HELLO` +held `Do`, `Subscribe` and every call that needed a new connection until the caller's +context ended, which is for ever under `context.Background()`. It now fails after +`DialTimeout`, also when the context has no deadline; a shorter deadline on the caller's +context still wins. The two are separate budgets, so a slow connect followed by a silent +server can take up to about twice `DialTimeout`. The pool's command and Pub/Sub dials, +`Pipeline`, and the node and sentinel clients below are bounded the same way +(`celeris/driver/redis/options.go` (`Config.DialTimeout`)). > TLS (`rediss://`) is **not** supported — `NewClient` rejects the scheme with a > clear error. Deploy over a VPC, loopback, or a sidecar TLS terminator > (`celeris/driver/redis/client.go` (`NewClient`)). +#### Redis Cluster and Sentinel + +`redis.NewClusterClient(redis.ClusterConfig{Addrs: ...})` and +`redis.NewSentinelClient(redis.SentinelConfig{...})` build on the same `Client` +(`celeris/driver/redis/cluster.go`, `celeris/driver/redis/sentinel.go`). What they do +that is easy to miss, as of v1.6.0: + +- **A Pub/Sub follows its subscription.** The `PubSub` from `ClusterClient.Subscribe` + or `SentinelClient.Subscribe` subscribes again on another node (a cluster) or on the + new master (Sentinel) when its connection goes away, and keeps delivering to the same + `PubSub.Channel()` + ([celeris#1154](https://github.com/goceleris/celeris/pull/1154)). A message + published while it switches is lost (a gap), and on a cluster a message in flight + around the switch can in rare cases be delivered twice. While it has no connection, + its control calls return `ErrPubSubConnDown`; it retries with backoff and no retry + limit until `PubSub.Close`. When the client is closed, a `PubSub` whose connection is + down (or goes down afterwards) gives up and its `Channel()` closes. +- **Sharded Pub/Sub is per slot.** `ClusterClient.SSubscribe` connects to the node that + owns the slot of the first channel and follows that slot when it moves. All channels + of one call must hash to one slot (use hash tags), or the call returns `ErrCrossSlot` + before anything is sent, and so does `PubSub.SSubscribe` with channels of different + slots. After a reroute the driver replays `SSUBSCRIBE` one slot at a time + ([celeris#1210](https://github.com/goceleris/celeris/pull/1210)). One `PubSub` holds + one connection, so a channel of another slot that the node does not own is answered + `-MOVED`, which is not reported, and is not subscribed; use a separate `PubSub` for + it. +- **A moved slot does not cause a herd of topology reads.** Commands that meet the same + `-MOVED` or the same lost primary share one `CLUSTER SLOTS` read, and so do the + reroutes of sharded `PubSub`s + ([celeris#1215](https://github.com/goceleris/celeris/pull/1215), + [celeris#1211](https://github.com/goceleris/celeris/pull/1211)). On the PR's test + cluster, 200 concurrent `GET`s on a moved slot sent 200 `CLUSTER SLOTS` before and 2 + to 6 after; the figure is that fake cluster's, not a guarantee. +- **One `Client` per node address.** A seed address listed more than once in `Addrs` now + builds a single node `Client`; before, the first was overwritten and never closed + ([celeris#1241](https://github.com/goceleris/celeris/pull/1241)). +- **`SentinelClient.Close` cancels and waits.** It cancels the background sentinel + subscription and any failover in progress and waits for them, so no `Client`, and no + reference to the driver's standalone event loop, outlives `Close` + ([celeris#1247](https://github.com/goceleris/celeris/pull/1247), + [celeris#1251](https://github.com/goceleris/celeris/pull/1251)). A failover dial that + is connecting or in the handshake ends at once. One that waits for the new master's + `ROLE` reply, in direct mode (a plain `net.Conn` client: an `Engine` that runs + `AsyncHandlers` on any OS, any `Engine` on macOS and Windows, and on Linux a started + `epoll`, `io_uring` or `adaptive` server), can hold `Close` for up to `DialTimeout` + (5 s by default). +- **`NewSentinelClient` discovery is bounded.** The query for the master address to each + sentinel, and the dial of the master with its `ROLE` check, are each bound by + `DialTimeout`, and discovery goes on to the next sentinel, so construction takes at + most (N+1) x `DialTimeout` for N sentinels (about 15 s for two at the 5 s default). A + failed construction returns the error of every sentinel tried (`errors.Join`); one that + timed out satisfies `errors.Is(err, context.DeadlineExceeded)` + ([celeris#1252](https://github.com/goceleris/celeris/pull/1252)). + ### PostgreSQL The Postgres driver offers two entry points (`celeris/driver/postgres/doc.go`): @@ -383,6 +449,29 @@ options (`celeris/driver/postgres/pool.go` (`WithMaxOpen`)): > The Postgres driver enforces SSL policy from the DSN's `sslmode`. Use a valid > `sslmode` for your deployment; the DSN is validated at `Open`/`NewConnector` time. +**Connect timeout.** `connect_timeout` (whole seconds, in the DSN) is one budget for the +dial and the startup together: the TCP connect, then the startup exchange +(authentication, `ParameterStatus`, `BackendKeyData`) up to `ReadyForQuery` +(`celeris/driver/postgres/dsn.go` (`Options.ConnectTimeout`), +[celeris#1216](https://github.com/goceleris/celeris/pull/1216), +[celeris#1223](https://github.com/goceleris/celeris/pull/1223)). A server that accepts +the connection and never completes the startup fails the dial with an error that matches +`context.DeadlineExceeded`, and a shorter deadline on the caller's context still wins. +Since v1.6.0: + +- A DSN **without** `connect_timeout` gets a **10 s default**. Before, such a dial + could wait for ever under `context.Background()`. +- `connect_timeout=0` means no client-side timeout, as in libpq; the caller's context is + then the only bound. A negative `connect_timeout` is rejected. +- In a DSN built in code (an `Options` value, not a string), a zero `ConnectTimeout` + is "not set" (the 10 s default applies) and a negative one means no timeout. +- The host name lookup before the dial is not bounded by it, nor by the caller's + context, and the `PG*` environment variables (`PGCONNECT_TIMEOUT` included) are not + read. + +`sslmode=require` and `verify-*` are refused up front, so there is no SSL negotiation to +bound. + ### Memcached ```go @@ -399,6 +488,19 @@ defer mc.Close() same shape as the Redis client and accepts `WithEngine` to colocate. TLS is not supported. +Two timeouts apply (`celeris/driver/memcached/options.go` (`Config.DialTimeout`, +`Config.Timeout`)): + +| Option | Bounds | Default | +| ------ | ------ | ------- | +| `WithDialTimeout(d)` | The TCP connect and, separately, the binary-protocol handshake (a `VERSION` round trip) of a new connection. The text protocol has no handshake. A slow connect followed by a silent server takes up to about twice `DialTimeout`. ([celeris#1217](https://github.com/goceleris/celeris/pull/1217)) | 5s | +| `WithTimeout(d)` | Each operation: a command (and `Ping`, `Version`, `Stats`, ...) not complete after `Timeout` fails with an error that matches `context.DeadlineExceeded`, and the connection it used is dropped. A multi-key get, `Stats` and `Flush` are one operation each. It does not cover waiting for a pool slot or dialling a new connection. A negative value turns it off, and the caller's context is then the only bound. ([celeris#1222](https://github.com/goceleris/celeris/pull/1222)) | 3s | + +`Timeout` applies whether or not the caller's context has a deadline; a caller's deadline +that is not later than `Timeout` wins and its error is returned as it is. Before v1.6.0 +`Config.Timeout` was documented but never applied, so a command on a server that +accepted the connection and never answered blocked until the caller's context ended. + ### Cache-aside: Redis in front of Postgres The `driver/redis` and `driver/postgres` packages ship in the core @@ -501,6 +603,57 @@ When you pass `WithEngine(srv)`, the driver pulls the server's `EventLoopProvide **Correctness is identical** with or without `WithEngine`; the difference is purely performance. +`WithEngine(srv)` of a server that has not started yet, or that is on the `std` engine +without `AsyncHandlers`, is answered with the driver's standalone loop. Closing a redis +or memcached `Client` (or a `ClusterClient`, or a redis `SentinelClient`) opened that +way gives its reference to that loop back since v1.6.0 +([celeris#1239](https://github.com/goceleris/celeris/pull/1239)); when the closed +`Client` was the last holder, the loop (an epoll or io_uring instance and its goroutine) +is shut down, so loops no longer accumulate across repeated open and close. A redis +`Client` that is the last holder takes its Pub/Sub connections down when it is closed; +another `Client` sharing the loop keeps them working. + +### Busy polling for driver connections + +The drivers no longer set `SO_BUSY_POLL` by default +([celeris#1233](https://github.com/goceleris/celeris/pull/1233); the server's own +`Config.SocketBusyPoll`, off by default, is in the +[Configuration reference](/docs/configuration#connection-socket-and-io)). In v1.5 the +redis, postgres and memcached drivers set 50 µs, but only on the connections they +register on an event loop, not on every connection they dial. Which those are depends +on how the driver was opened (`celeris/README.md`, "Busy polling"): + +| Driver opened | redis commands | memcached | postgres `Pool` | +| --- | --- | --- | --- | +| no engine | loop (was set) | netpoll | netpoll | +| `WithEngine` of a server that does not run async handlers and has not started yet, or is on the `std` engine | loop (was set) | loop (was set) | loop (was set) | +| `WithEngine` of a started `epoll`, `io_uring` or `adaptive` server (the default is `adaptive`) | netpoll | netpoll | netpoll | +| `WithEngine` of a server that runs async handlers, started or not | netpoll | netpoll | netpoll | + +Redis Pub/Sub connections and postgres `Connector` connections are always on a loop. The +connections that go through Go's netpoll were never set and still are not, so an +application that used only those is not affected and has nothing to change. + +Opened with `WithEngine(srv)`, the connections that are on a loop now take the server's +`Config.SocketBusyPoll` (`srv.SocketBusyPoll()`, callable before `Start`; it reports 0 +for the `std` engine), so one setting decides both. With no engine they are off unless +you ask: + +- redis: `redis.WithSocketBusyPoll(d)` or `Config.SocketBusyPoll` (`ClusterConfig` and + `SentinelConfig` forward it); +- postgres: `Connector.WithSocketBusyPoll(d)`; a `Pool` has no option of its own and + only follows the server; +- memcached: no option of its own; it only follows the server. + +A value of at least 1 µs on a driver option wins over the server's; anything less +(0, a negative value, a fraction of a microsecond) leaves the connection to the server's +value, so a driver option cannot turn off a server that has busy polling on. A refused +`setsockopt` is ignored silently (the server logs its own warning once). The reason it is +off is the one in the Configuration reference; the measurements were of accepted sockets, +and no driver traffic was measured. Source: `celeris/driver/redis/options.go` +(`Config.SocketBusyPoll`), `celeris/driver/postgres/connector.go` +(`Connector.WithSocketBusyPoll`), `celeris/server.go` (`Server.SocketBusyPoll`). + ## Standalone vs engine-colocated mode `WithEngine` is **always optional**. Drivers run perfectly well standalone (e.g. in diff --git a/src/content/docs/engines.md b/src/content/docs/engines.md index e3ef14d..dc51f92 100644 --- a/src/content/docs/engines.md +++ b/src/content/docs/engines.md @@ -78,6 +78,18 @@ latency for compatibility by choosing it. Epoll is also the engine that implemen zero-copy `sendfile(2)` for static-file responses (`celeris/internal/engine/engine.go` (`SendfileCapable`), `celeris/internal/engine/capability.go` (`CapabilityProfile.Sendfile`)). +Every epoll worker arms a 25 ms kernel timer (a `timerfd`) and sweeps its connections +for expired timeouts on each tick, whatever `ReadHeaderTimeout` and the other timeouts +are set to. Since v1.6.0 that includes `ReadHeaderTimeout: -1`; before, a loop without +the timer swept every 4 to 5 s, so `IdleTimeout`, `ReadTimeout` and `WriteTimeout` fired +seconds late +([celeris#1197](https://github.com/goceleris/celeris/pull/1197); see +[How promptly the native engines enforce a timeout](/docs/configuration#how-promptly-the-native-engines-enforce-a-timeout)). +Both native engines also set `TCP_DEFER_ACCEPT` on their listen sockets by default, so +the kernel hands them a connection when its first byte arrives, or about a second after +its SYN if the client stays silent; `Config.DisableDeferAccept` turns that off (see +[Configuration](/docs/configuration#connection-socket-and-io)). + ### IOUring Completion-based asynchronous I/O on Linux **5.10+**. Celeris detects the io_uring @@ -95,6 +107,30 @@ kernel transparently uses the `Base` feature set; a 6.1 kernel lights up the `Optional` set. io_uring requires `RLIMIT_MEMLOCK` headroom for its rings and provided buffers — see [Engine selection in practice](#engine-selection-in-practice). +**Startup probes.** Besides the tier, the engine probes four features at startup by +running them once on a throwaway ring: `SEND_ZC`, fixed files, provided buffers and +multishot accept. The result is cached for the process, but since v1.6.0 only the +kernel's own answer is cached +([celeris#1201](https://github.com/goceleris/celeris/pull/1201), +[celeris#1198](https://github.com/goceleris/celeris/pull/1198); +`celeris/internal/engine/iouring/probe.go`): + +- A probe that could not run, because setup failed before the kernel answered (out of + file descriptors or memory, a busy ring, a failed wait), is **not** cached. The + engine being built goes without that feature, and the probe runs again when the next + engine is built. Before, an `EMFILE` at the first engine build could leave the process + without the feature for good, while the log said the kernel had refused it. +- A wait that returns with nothing to read (a signal, a timer's wake) is waited out + within the same budget instead of being read as the kernel's answer. A probe whose + wait gets no answer in two runs is kept as unsupported. +- When a probe could not run, the engine logs one Warn per probe per process: + `io_uring startup probe could not run when an engine was built, so the feature was + off for the engine being built then; the probe runs again when the next engine is + built (restart the server, or build a new engine, to probe again; a probe whose wait + gets no answer in two runs is kept as unsupported)`, with the attributes `probe` + (`SEND_ZC`, `fixed files`, `provided buffers` or `multishot accept`) and `reason`. + Restart the server to probe again. + #### Tuning environment variables The engines read these at startup. None is needed for normal operation. The @@ -482,15 +518,16 @@ if m.RequestCount > 0 { To stop accepting new connections while continuing to serve existing ones — useful for graceful load shedding or coordinated draining — call `Server.PauseAccept()` -and later `Server.ResumeAccept()` (`celeris/server.go` (`Server.PauseAccept`)). These work on the -native engines. The **std engine does not support accept control**: both return -`celeris.ErrAcceptControlNotSupported` (`celeris/errors.go` (`ErrAcceptControlNotSupported`)), as does calling -them before `Start`. +and later `Server.ResumeAccept()` (`celeris/server.go` (`Server.PauseAccept`)). These work on +`epoll` and `io_uring`. The **`std` and `adaptive` engines do not support accept control** +(`celeris/server.go` (`Server.ResumeAccept`)): both calls return +`celeris.ErrAcceptControlNotSupported` (`celeris/errors.go` (`ErrAcceptControlNotSupported`)), as they do +before `Start`. ```go if err := s.PauseAccept(); err != nil { if errors.Is(err, celeris.ErrAcceptControlNotSupported) { - // std engine (or not started) — fall back to your own shedding strategy + // std or adaptive engine (or not started) — fall back to your own shedding strategy } } // ... drain / shed ... @@ -550,7 +587,7 @@ if p := s.EventLoopProvider(); p != nil { - **Streaming on the std engine without finishing before return.** On Std, `EngineSupportsAsyncDetach()` is false; a goroutine you spawn must complete before the handler returns, or `net/http` closes the connection out from under it. -- **`PauseAccept`/`ResumeAccept` on std.** They return +- **`PauseAccept`/`ResumeAccept` on std or adaptive.** They return `ErrAcceptControlNotSupported` — always check the error with `errors.Is`. - **Forgetting `RLIMIT_MEMLOCK` for io_uring.** Too-low memlock prevents io_uring setup; under Adaptive this silently keeps you on epoll, under a forced `IOUring` diff --git a/src/content/docs/graceful-shutdown.md b/src/content/docs/graceful-shutdown.md index 8fbaa59..57ce8bf 100644 --- a/src/content/docs/graceful-shutdown.md +++ b/src/content/docs/graceful-shutdown.md @@ -243,7 +243,11 @@ finished. Before celeris v1.6.0, on `epoll` and `io_uring` the hooks ran, and a ([celeris#703](https://github.com/goceleris/celeris/issues/703)). `Shutdown` returns the engine's shutdown error, or `ctx`'s error if the drain did not -finish within `ctx`, or `nil`. Note that hook panics are swallowed (recovered) — they +finish within `ctx`, or `nil`. On `epoll`, `io_uring` and `adaptive`, a drain that +`ctx`'s deadline ends can finish at that same instant, and `Shutdown` then returns +`nil`: **`nil` at the deadline does not mean nothing was cut** +([celeris#1225](https://github.com/goceleris/celeris/issues/1225), `celeris/server.go` +(`Server.Shutdown`)). Note that hook panics are swallowed (recovered) — they do not surface in the return value — so do your own error logging inside the hook. **Overlapping calls.** Several `Shutdown` calls can share one drain, for example a second @@ -328,7 +332,12 @@ handlers had returned, and such a response lost its tail ([celeris#760](https://github.com/goceleris/celeris/issues/760)). On `io_uring` the keep-accepting window is at most 250 ms, the start of that drain: past -it the worker stops accepting. Before [celeris#978](https://github.com/goceleris/celeris/issues/978) +it the worker stops accepting. A worker with nothing to send and no request in progress +does not keep accepting at all: if it is waiting only for a request head that is +already in the kernel, it closes its listener at once, and a connection that arrives +afterwards is not waited for +([celeris#1243](https://github.com/goceleris/celeris/pull/1243); see +[What each kind of connection gets](#what-each-kind-of-connection-gets)). Before [celeris#978](https://github.com/goceleris/celeris/issues/978) its send drain was a fixed 250 ms whatever the budget, so a client slower than that could lose the tail. `std` drains through net/http. @@ -348,6 +357,112 @@ above), so no byte count is given.) > manually) to cover both the request drain *and* the slowest resource you close in a > hook. +### What each kind of connection gets + +What a connection gets when `Shutdown` begins depends on what it has sent, and on the +engine. This is the contract the `Shutdown` godoc states +([celeris#1208](https://github.com/goceleris/celeris/issues/1208), +[celeris#1220](https://github.com/goceleris/celeris/pull/1220), +[celeris#1253](https://github.com/goceleris/celeris/pull/1253)); the timings are the +maintainers' measurements on arm64, in docker with 2 CPUs, on a 7.0 kernel with +`io_uring`'s high tier. `adaptive` does what the engine it is running does (`epoll` +when it starts). Source: `celeris/server.go` (`Server.Shutdown`). + +- **An idle keep-alive connection** (no request under way). `std` and `epoll` close it + at once, and so does an `io_uring` worker with no response going out. An `io_uring` + worker that has one serves on for the first 250 ms of its send drain, so its idle + connections are closed at 250 ms, and a request one of them sends before then is + answered with `Connection: close`. While a connection that has begun a head is open + (below), `epoll` closes the idle connections of the loop that serves it at about 250 + ms, and `io_uring` closes its idle connections at about 250 ms (251 to 258 ms with + six; with 2000, all were closed within 0.46 s on `epoll` and 0.71 s on `io_uring`, + against 11 ms and 17 ms with no such connection). +- **A request in progress, whose head the server has read.** Read on and answered, with + `Connection: close`, on every engine. +- **A head begun and not completed: a fragment, or a bare CRLF.** On `epoll` and + `io_uring` ([celeris#1225](https://github.com/goceleris/celeris/issues/1225)) this + *is* a request in progress, whether the connection was idle after a response or is + new: the engine reads on and waits for it, and does not close the connection as it + closes an idle one. `Shutdown` waits until the head completes (the request is then + answered, with `Connection: close`), until a limit of the connection ends it, or + until `ctx` is done. The limits are `ReadHeaderTimeout` (10 s by default), counted + from the head's first byte (for a new connection, from the engine's accept, which + [`TCP_DEFER_ACCEPT`](/docs/configuration#connection-socket-and-io) puts at that byte) + and **not extended** by the bytes that follow (a head trickled in at one byte per 150 + ms was cut at it), and `WriteTimeout`, from the start of the wait; when one of them + ends the wait, `Shutdown` returns `nil` if that was all it waited for. A client can + therefore hold a graceful shutdown for as long as the shortest of those by sending a + bare CRLF or one byte of a head. The bound you set is `ctx`'s deadline: when it comes + first, `Shutdown` returns at it, with `ctx`'s error (or with `nil`, see above), and + the engine closes the connection with a FIN. `std` counts differently: `net/http` + marks a connection active only once a request head has been read in full, so `std` + closes a keep-alive connection that has begun a head, or sent a bare CRLF, with the + idle ones, without an answer, and `Shutdown` returns at once (measured at 1 ms). A + *new* connection that has begun its first head is not idle for `net/http` until it is + about five seconds old, so on `std` `Shutdown` waits for it until the head completes, + `ReadHeaderTimeout` cuts it, it is five seconds old, or `ctx` is done (then + `Shutdown` returns `ctx`'s error and leaves the connection open; still open 4 s later + when measured). +- **A request head that has reached the kernel without the server having read it.** + Answered, with `Connection: close`, on `epoll`, `io_uring` and `adaptive`, as a + request in progress is (also for a connection whose head arrived while the loop was + held in another connection's `OnConnect` callback). A fragment or a bare CRLF found + there is read and is a request in progress like any other (the bullet above). An + `io_uring` worker with nothing to send makes this look **once**, when its send drain + begins (after the wait for the HTTP/2 handlers), and waits only for the connections + that had such a head then, with its listener closed + ([celeris#1226](https://github.com/goceleris/celeris/pull/1226), + [celeris#1243](https://github.com/goceleris/celeris/pull/1243)): a connection that + arrives afterwards does not hold it, however many keep coming, and a request it sends + is dropped. The head is answered on a connection the engine has accepted: one still in + the accept queue when the listener closes gets a reset, on `epoll` as well. A head + that reaches the kernel after the engine's gate has looked at the connection is + refused. On `std`, `net/http` reads the head after `Shutdown` began and drops the + request: the connection is closed without an answer, and reset if body bytes are + still unread (a small body, 2 KiB measured, is read with the head and the close is a + FIN; 64 KiB gave a reset). +- **A connection that has sent nothing.** `std`, `epoll` and an `io_uring` worker with + no response going out do not answer it, and a request it sends once `Shutdown` has + begun is dropped. `epoll` and such a worker close it at once if the engine has + accepted it. By default the kernel does not hand such a connection to the engine for + about a second (`TCP_DEFER_ACCEPT`, see `DisableDeferAccept` in the + [Configuration reference](/docs/configuration#connection-socket-and-io)); while it is + held back, `Shutdown` neither closes it nor lets a request through on `epoll` or on + such a worker: its first request gets a reset once the loops have closed their + listeners (an `epoll` loop busy sending can take a few milliseconds to, and answers + with `Connection: close` until then). An `io_uring` worker that has a response going + out keeps accepting for its first 250 ms: it closes an accepted silent connection at + 250 ms and answers, with `Connection: close`, a first request that arrives before + then, on an accepted connection or a held-back one. `std` keeps the connection open + until its first request is read, which is then dropped as above, or until `net/http` + closes it for being five seconds old, so a client that connects and sends nothing can + hold `Shutdown` on `std` for about five seconds, within `ctx`. + +### Every HTTP/1 response after `Shutdown` ends its connection + +Every HTTP/1 response written once `Shutdown` has begun ends its connection, as +`net/http`'s does: the response to a request that was already in its handler when +`Shutdown` began, and the response to a request that arrives on a keep-alive +connection before the connection is closed +([celeris#1169](https://github.com/goceleris/celeris/pull/1169), v1.6.0). Source: +`celeris/server.go` (`Server.Shutdown`). + +- On the native engines (`epoll`, `io_uring`, `adaptive`) it carries exactly one + `Connection: close`, an HTTP/1.0 client's included, and a handler's own `Connection` + header on it is replaced by that one. The `417` of a refused `Expect` says + `Connection: close` too. +- `std`, which is `net/http`, closes the connection after the response and says `close` + to an HTTP/1.1 client, but not to an HTTP/1.0 one. +- A request pipelined behind such a response is **not served**: its bytes, already + read, are dropped with the connection, and so are those the kernel holds when the + engine's gate looks at the connection (a request that reaches it after that can still + reset it, as it can `net/http`'s). +- Not changed: a response whose head went out before `Shutdown` began (a stream under + way keeps the head it sent; its connection is closed by the shutdown, at once or when + the gate looks at it, 250 ms into a wait), a detached stream (Server-Sent Events, a + WebSocket), which the engines end themselves, and a hijacked connection, which is + the application's. HTTP/2 connections get `GOAWAY` instead. + ## Drain hooks: `OnShutdown` `Server.OnShutdown(fn)` registers a function to run at the end of `Shutdown`, after the @@ -469,16 +584,33 @@ _ = s.ResumeAccept() | `PauseAccept() error` | Stop accepting new connections. Existing connections continue to be served. | | `ResumeAccept() error` | Resume accepting after a pause. | -**Native engines only.** Accept control is implemented by the native Linux engines -(`epoll`, `io_uring`, and the `adaptive` controller that drives them). The `std` -(net/http) engine does **not** support it: both methods return -`celeris.ErrAcceptControlNotSupported` on `std`, and also when the server has not been -started yet (no engine is installed). Always check the error and have a fallback (a -full `Shutdown`) for portability. Source: `celeris/server.go` (`PauseAccept`, -`ResumeAccept`), -`celeris/errors.go` (`ErrAcceptControlNotSupported`), `celeris/internal/engine/engine.go` (`AcceptController`). See +**`epoll` and `io_uring` only.** Accept control is implemented by the `epoll` and +`io_uring` engines when you select one of them. The `std` (net/http) engine does +**not** support it, and neither does the `adaptive` engine, which is the default on +Linux: on both, `PauseAccept` and `ResumeAccept` return +`celeris.ErrAcceptControlNotSupported`, as they do when the server has not been +started yet (no engine is installed). So to pause accept, set `Engine: celeris.Epoll` +or `celeris.IOUring`, and always check the error and have a fallback (a full +`Shutdown`) for the rest. Source: `celeris/server.go` (`Server.PauseAccept`, +`Server.ResumeAccept`), `celeris/errors.go` (`ErrAcceptControlNotSupported`), +`celeris/internal/engine/engine.go` (`AcceptController`). See [Engines](/docs/engines) for which engine runs where. +**On `epoll` and `io_uring`, `PauseAccept` blocks for about 1.5 s by default.** +`TCP_DEFER_ACCEPT` is on by default on their listen sockets, and while it is on, the kernel +holds a client that has completed its handshake but not yet sent its request outside +the accept queue until about one second after its SYN. Closing the listen socket then +would reset that client, so `PauseAccept` first clears the option on each listen socket +and keeps it open, accepting and serving, for about 1.5 s, and only then closes it. The +call blocks for that long, and the server keeps admitting new connections meanwhile. +What a pause can still reset is a handshake still in flight when a listen socket +closes, which is inherent to closing one, and a client whose retransmitted SYN-ACK is +lost or goes unanswered on a lossy path. Set `Config.DisableDeferAccept` for a pause +that closes the listen sockets at once; the +[Configuration reference](/docs/configuration#connection-socket-and-io) says what the +option saves. Source: `celeris/server.go` (`Server.PauseAccept`), +`celeris/config.go` (`Config.DisableDeferAccept`). + > Pause/resume is for *temporary* quiescing. It does not drain — in flight requests > keep running, and paused connections are simply not accepted. For an orderly stop > that drains and runs your hooks, use `Shutdown`. @@ -692,8 +824,8 @@ std via the shared inherited fd), so no client connection is refused. Run it on its own goroutine. - **Expecting hook panics in the return value.** Hook panics are recovered and *not* reflected in `Shutdown`'s return. Log errors inside the hook. -- **Using `PauseAccept` on the std engine.** It returns - `ErrAcceptControlNotSupported`. Accept control is native-engines-only. +- **Using `PauseAccept` on the std or adaptive engine.** It returns + `ErrAcceptControlNotSupported`. Accept control is `epoll`- and `io_uring`-only. ## FAQ @@ -757,7 +889,7 @@ Keep handlers shorter than the budget, and give long-running ones (streams) a st signal of your own that your signal handler fires before it cancels the context. **Can I pause accept instead of shutting down for a maintenance window?** -On native engines, yes — `PauseAccept` then `ResumeAccept`. It keeps in flight requests +On `epoll` and `io_uring`, yes — `PauseAccept` then `ResumeAccept`. It keeps in flight requests running but does not drain; it is not a substitute for `Shutdown` when you actually want to stop. diff --git a/src/content/docs/performance.md b/src/content/docs/performance.md index d13b55f..367daaa 100644 --- a/src/content/docs/performance.md +++ b/src/content/docs/performance.md @@ -250,6 +250,8 @@ These are all in `Config` (`celeris/config.go`): | `BufferSize int` | engine default | Per-connection I/O buffer size in bytes. `0` = engine default. | | `SocketRecvBuf int` | OS default | `SO_RCVBUF` on accepted connections. `0` = OS default. | | `SocketSendBuf int` | OS default | `SO_SNDBUF` on accepted connections. `0` = OS default. | +| `SocketBusyPoll time.Duration` | `0` (off) | `SO_BUSY_POLL` on accepted connections (Linux native engines). Off since v1.6.0; it was 50 µs before. | +| `DisableDeferAccept bool` | `false` | Turn `TCP_DEFER_ACCEPT` off on the `epoll` and `io_uring` listen sockets. | | `MaxConns int` | unlimited | Max simultaneous connections **per worker**. `0` = unlimited. | ```go @@ -270,6 +272,19 @@ A few tuning notes grounded in the engine signals: - **`MaxConns` is per worker, not per server.** With 8 workers and `MaxConns: 10_000` your ceiling is ~80k connections. It's an admission bound, not a tuning knob — use it to fail fast rather than thrash when a flood arrives. +- **Leave `SocketBusyPoll` off unless you have measured it.** The maintainers found + throughput 3.0% higher with it off on an arm64 big.LITTLE host with a single-queue NIC + (where it also coincided with `ksoftirqd` takeovers and 1 to 7 s stalls, an indicative result) and 1.1% + higher on an amd64 host with a 16-queue NIC; the drivers opened `WithEngine(srv)` + follow the server's value. Details, caveats and the exact conditions are in + [Configuration](/docs/configuration#connection-socket-and-io) and + [Stores and database drivers](/docs/data-stores#busy-polling-for-driver-connections). +- **`DisableDeferAccept` gives up a saved wakeup per new connection.** With the + default, a client that connects and sends its request at once is accepted with the + request already queued, saving a wakeup per new connection; turning it off gives that + up, a cost on connection churn only (keep-alive traffic does not see it), and removes + the second of deferral a silent connection otherwise waits + ([Configuration](/docs/configuration#connection-socket-and-io)). - **The bytes-per-request signal.** The engine tracks `BytesRead` and `BytesWritten` alongside `RequestCount` (`celeris/observe/engine_metrics.go` (`EngineMetrics`)). Large average payloads (link-bound workloads) make epoll and io_uring tie, so the diff --git a/src/content/docs/sse.md b/src/content/docs/sse.md index 45f5266..6cd7660 100644 --- a/src/content/docs/sse.md +++ b/src/content/docs/sse.md @@ -586,7 +586,7 @@ reading is dealt with per engine: queued for it; `-1` sets no such bound. The disconnect comes between about 3/4 and 5/4 of `WriteTimeout` after the client last took a byte (its progress is sampled every quarter of `WriteTimeout`; a send completion restarts the clock at once), plus the - engine's sweep interval (up to about 0.1 s on `epoll`, up to 2 s on `io_uring`; a + engine's sweep interval (25 ms on `epoll`, up to 2 s on `io_uring`; a `WriteTimeout` shorter than that can double). The handler returning or a shutdown ends the stream sooner. From e6258a62578317b2f5ba759c6acde1130c7610a2 Mon Sep 17 00:00:00 2001 From: Albert Bausili Date: Wed, 4 Nov 2026 02:44:47 +0100 Subject: [PATCH 2/3] docs: fix the review findings on the v1.6.0 catch-up (WriteTimeout sweep, colocation, accept control) and add the HTTP/2 deadlines, static Root and dotfiles, cache SetTimeout --- src/content/docs/configuration.md | 50 +++++++++++++++++++++----- src/content/docs/core-concepts.md | 9 ++--- src/content/docs/data-stores.md | 35 +++++++++++------- src/content/docs/engines.md | 5 ++- src/content/docs/middleware-content.md | 3 +- src/content/docs/static-files.md | 22 ++++++++++++ 6 files changed, 96 insertions(+), 28 deletions(-) diff --git a/src/content/docs/configuration.md b/src/content/docs/configuration.md index 4c1c66e..9e2d920 100644 --- a/src/content/docs/configuration.md +++ b/src/content/docs/configuration.md @@ -217,21 +217,24 @@ With `DisableDeferAccept` the deferral is gone: a silent connection is closed at ### How promptly the native engines enforce a timeout `epoll`, `io_uring` and `adaptive` do not arm one timer per connection for -`ReadTimeout`, `IdleTimeout` and `WriteTimeout`; they look at their connections in a -sweep, so each is enforced up to a sweep period after its time. Source: -`celeris/config.go` (`Config.ReadTimeout`). +`ReadTimeout` and `IdleTimeout`; they look at their connections in a sweep, so those +two are enforced up to a sweep period after their time. `WriteTimeout` is judged in +windows that end at a sweep (see `Config.WriteTimeout`): on `epoll`, a 1 s +`WriteTimeout` closed a client that reads nothing at about 2 s in the PR's measurement, +which is the documented stall-rule windows, and the sweep only adds its own lateness to +that. Source: `celeris/config.go` (`Config.ReadTimeout`, `Config.WriteTimeout`). - **`epoll`: every 25 ms, whatever the timeouts are.** Each worker arms a 25 ms kernel - timer (a `timerfd`) at start, so a timeout fires within about 30 ms (a 1 s - `IdleTimeout` closed an idle connection at 1.00 to 1.03 s on docker arm64 with 2 + timer (a `timerfd`) at start, so `ReadTimeout` and `IdleTimeout` fire within about 30 ms + (a 1 s `IdleTimeout` closed an idle connection at 1.00 to 1.03 s on docker arm64 with 2 CPUs). Since v1.6.0 that holds with `ReadHeaderTimeout: -1` too. Before, the timer was armed only when `ReadHeaderTimeout` was set, and without it a quiet loop swept every 4 to 5 s: a 1 s `IdleTimeout` closed at 4.4 to 5.0 s ([celeris#1197](https://github.com/goceleris/celeris/pull/1197), the numbers are that PR's measurement). `adaptive` follows `epoll` while it runs on it. - **`io_uring`: about 0.8 s apart on an idle worker** (0.83 s measured; about 1 s with - `ReadHeaderTimeout` off), so a timeout of a few seconds or less is approximate by - that much. The HTTP/1 header deadline and the budget of a connection whose protocol + `ReadHeaderTimeout` off), so a `ReadTimeout` or `IdleTimeout` of a few seconds or + less is approximate by that much. The HTTP/1 header deadline and the budget of a connection whose protocol is not decided yet have a timer of their own on `io_uring` and are not late; the HTTP/2 header-block bound is checked by the sweep and is. @@ -260,6 +263,30 @@ limiter, cache, session and JWT expiry, the stores). > migration and snapshot restore typically keep the guest's monotonic clock continuous > (kvmclock, Firecracker), which it did not verify either. +### HTTP/2 timeouts on the native engines + +On `epoll`, `io_uring` and `adaptive`, the HTTP/2 deadlines are per request and per +connection, not per read +([celeris#1150](https://github.com/goceleris/celeris/pull/1150); `celeris/config.go` +(`Config.ReadTimeout`, `Config.ReadHeaderTimeout`)): + +- **`ReadTimeout` bounds each request**, from its `HEADERS` frame, as an absolute + deadline, as `net/http`'s is. A request that is not whole by then (its body, or its + trailers, still to come) is answered `408` and its stream is reset with + `RST_STREAM(NO_ERROR)`; the connection and its other streams go on. A `DATA` frame or + a `PING` does not move the deadline, so **an HTTP/2 upload has to fit in + `ReadTimeout`** (as on `std`; there is no per-route override), where an HTTP/1.1 body + is bounded by the time since the last read. A request without a body has no deadline. +- **`ReadHeaderTimeout` bounds a header block.** One that begins with a `HEADERS` frame + without `END_HEADERS` and is not complete that long after ends the connection with + `GOAWAY(ENHANCE_YOUR_CALM)`. +- **`IdleTimeout` ignores `PING`s.** A connection with no stream open that receives only + `PING`, `SETTINGS` or connection-level `WINDOW_UPDATE` frames is closed at + `IdleTimeout` from the last stream activity, which those frames do not move. A + connection that has a stream open whose request is whole is not closed by the + connection's own clock; a response waiting for the client's flow-control window is + bounded by `WriteTimeout` (see [HTTP/2 response writes](/docs/core-concepts)). + ### SSE needs no `-1`; other long-lived responses do An SSE stream is a **detached** stream (the middleware calls @@ -288,7 +315,12 @@ A response that is **not** detached is still under `WriteTimeout`: a slow large download, or a chunked body written with `StreamWriter` without `Detach`. On `std` the deadline is absolute for the whole response, so a response that takes longer than `WriteTimeout` ends however steadily the client reads (measured: a streaming body that -sends every 200 ms is cut at `WriteTimeout`). On HTTP/2 over `epoll`, `io_uring` and +sends every 200 ms is cut at `WriteTimeout`). On `epoll` (and `adaptive` while it runs on +it), a response the kernel did not accept whole is instead bounded by a minimum rate on +the bytes the client acknowledges, 64 KiB per `WriteTimeout`, with no elapsed-time +ceiling: a client that reads steadily above that floor is not cut however long the +response takes ([celeris#1119](https://github.com/goceleris/celeris/pull/1119), +`celeris/config.go` (`Config.WriteTimeout`)). On HTTP/2 over `epoll`, `io_uring` and `adaptive` a write that waits for the client's flow-control window is bounded by `WriteTimeout` ([HTTP/2 response writes](/docs/core-concepts)). Raise the timeout for those routes, or disable it server-wide only when such responses dominate: @@ -421,7 +453,7 @@ the floor. | Field | Type | Default | Constraint | Effect | | ------------------ | ------ | -------------- | ----------------------- | ---------------------------------------------- | | `DisableKeepAlive` | `bool` | `false` | — | One request per connection when `true` | -| `DisableDeferAccept` | `bool` | `false` | `epoll` / `io_uring` only | Turns `TCP_DEFER_ACCEPT` off on the listen sockets | +| `DisableDeferAccept` | `bool` | `false` | Linux native engines (`epoll`, `io_uring`, `adaptive`) | Turns `TCP_DEFER_ACCEPT` off on the listen sockets | | `BufferSize` | `int` | `8192` | `>= 4096`; clamped to `262144` | Per-connection I/O buffer size in bytes | | `SocketRecvBuf` | `int` | OS default (0) | — | `SO_RCVBUF` for accepted connections | | `SocketSendBuf` | `int` | OS default (0) | — | `SO_SNDBUF` for accepted connections | diff --git a/src/content/docs/core-concepts.md b/src/content/docs/core-concepts.md index 1f6346c..1ee9aea 100644 --- a/src/content/docs/core-concepts.md +++ b/src/content/docs/core-concepts.md @@ -330,10 +330,11 @@ at least 64 KiB of them (or all it held, if less) sent, released by the client's `WINDOW_UPDATE`s. If it did not, the stream is reset (`RST_STREAM`, `INTERNAL_ERROR`), the request's context is done, and the write returns an error that wraps `os.ErrDeadlineExceeded`. There is no ceiling: a stream that releases at least that -on average is not cut however long its response takes, and a `PING`, or a -`WINDOW_UPDATE` of one byte, is not progress. Clients on slow links need a -`WriteTimeout` of 2 s or more, and 5 s where the links are slower than a megabyte a -second. With `WriteTimeout: -1` there is no deadline: the wait lasts until the client +on average (lagging the rate by no more than four windows' worth) is not cut however +long its response takes, and a `PING`, or a `WINDOW_UPDATE` of one byte, is not +progress. Clients on slow links need a `WriteTimeout` of 2 s or more, and 5 s where the +links are slower than a megabyte a second; on `io_uring`, where the engine's own send +clock closes such a connection at 1 s, use 5 s for clients on slow links. With `WriteTimeout: -1` there is no deadline: the wait lasts until the client grants window or the connection closes, and never ends if the I/O worker that would see either is the one waiting (below). Source: `celeris/config.go` (`Config.WriteTimeout`). diff --git a/src/content/docs/data-stores.md b/src/content/docs/data-stores.md index da8662d..ac37c98 100644 --- a/src/content/docs/data-stores.md +++ b/src/content/docs/data-stores.md @@ -341,17 +341,18 @@ that is easy to miss, as of v1.6.0: limit until `PubSub.Close`. When the client is closed, a `PubSub` whose connection is down (or goes down afterwards) gives up and its `Channel()` closes. - **Sharded Pub/Sub is per slot.** `ClusterClient.SSubscribe` connects to the node that - owns the slot of the first channel and follows that slot when it moves. All channels - of one call must hash to one slot (use hash tags), or the call returns `ErrCrossSlot` - before anything is sent, and so does `PubSub.SSubscribe` with channels of different - slots. After a reroute the driver replays `SSUBSCRIBE` one slot at a time + owns the slot of the first channel and follows that slot when it moves. A single + `ClusterClient.SSubscribe` or `PubSub.SSubscribe` call whose channels span slots + returns `ErrCrossSlot` before anything is sent (use hash tags). A separate + `PubSub.SSubscribe` call for another slot's channel is subscribed only if the node the + `PubSub` is on owns that slot; otherwise the node answers `-MOVED`, which is not + reported, and the channel is not subscribed. After a reroute the driver replays + `SSUBSCRIBE` one slot at a time ([celeris#1210](https://github.com/goceleris/celeris/pull/1210)). One `PubSub` holds - one connection, so a channel of another slot that the node does not own is answered - `-MOVED`, which is not reported, and is not subscribed; use a separate `PubSub` for - it. + one connection, so it is on one node. - **A moved slot does not cause a herd of topology reads.** Commands that meet the same - `-MOVED` or the same lost primary share one `CLUSTER SLOTS` read, and so do the - reroutes of sharded `PubSub`s + `-MOVED` or the same lost primary share a few `CLUSTER SLOTS` reads instead of one + each, and the reroutes of sharded `PubSub`s share reads too ([celeris#1215](https://github.com/goceleris/celeris/pull/1215), [celeris#1211](https://github.com/goceleris/celeris/pull/1211)). On the PR's test cluster, 200 concurrent `GET`s on a moved slot sent 200 `CLUSTER SLOTS` before and 2 @@ -595,10 +596,18 @@ When you pass `WithEngine(srv)`, the driver pulls the server's `EventLoopProvide - Without `WithEngine`, each driver resolves a **standalone** event loop, shared and reference-counted across all drivers that omit `WithEngine`. -- With `WithEngine`, driver FDs land on the same worker goroutine as HTTP handlers. - This saves one epoll/io_uring syscall per I/O and improves data locality — the - Redis driver docs cite lower latency from better data locality for serial - queries (`celeris/driver/redis/doc.go`). +- With `WithEngine` of a started `epoll`, `io_uring` or `adaptive` server, the + connections the driver registers on an event loop (redis Pub/Sub, postgres + `Connector`) land on the server's worker loops, next to the HTTP handlers; that saves + an epoll/io_uring syscall per I/O and improves data locality — the Redis driver docs + cite lower latency from better data locality for serial queries + (`celeris/driver/redis/doc.go`). The command connections (redis, memcached) and the + postgres `Pool` then use Go's netpoll on the handler's goroutine, and so do all three + with a server that runs async handlers. Before `Start`, or on `std` without async + handlers, the driver uses its standalone loop. The table under + [Busy polling for driver connections](#busy-polling-for-driver-connections) lists each + case (`celeris/driver/redis/options.go` (`Config.SocketBusyPoll`), + `celeris/driver/internal/eventloop/registry.go` (`Resolve`)). **Correctness is identical** with or without `WithEngine`; the difference is purely performance. diff --git a/src/content/docs/engines.md b/src/content/docs/engines.md index dc51f92..031e3cb 100644 --- a/src/content/docs/engines.md +++ b/src/content/docs/engines.md @@ -206,6 +206,8 @@ and let Adaptive/Std resolve automatically. | Accept control (Pause/Resume) | Yes | Yes | — | | Driver event-loop colocation | Yes | Yes | — | +`adaptive` has no column: it switches between `epoll` and `io_uring`. It does not implement accept control, so `PauseAccept` and `ResumeAccept` return `ErrAcceptControlNotSupported` on it. + Sources: `celeris/internal/engine/capability.go`, `celeris/internal/engine/engine.go` (`AcceptController`, `SwitchFreezer`, `SendfileCapable`), `celeris/server.go` (`Server.EventLoopProvider`), `celeris/context_response.go` (`Context.EngineSupportsAsyncDetach`). @@ -614,7 +616,8 @@ Yes — `EngineInfo().Type` reflects the *active* engine, and **Will my code change between engines?** No. The router, middleware, `*Context`, and handler signature are identical on all four. The only engine-visible behavioural differences are the native-only features -(async-detach, accept control, driver colocation, sendfile) — and Celeris exposes +(async-detach, driver colocation, sendfile, and accept control, which only `epoll` and +`io_uring` implement, not `adaptive`) — and Celeris exposes runtime predicates (`EngineSupportsAsyncDetach`, the `ErrAcceptControlNotSupported` sentinel, a `nil` `EventLoopProvider`) so portable code can adapt cleanly. diff --git a/src/content/docs/middleware-content.md b/src/content/docs/middleware-content.md index 64b333b..847e5d0 100644 --- a/src/content/docs/middleware-content.md +++ b/src/content/docs/middleware-content.md @@ -244,10 +244,11 @@ store transport error sets `X-Cache: ERROR` and passes through uncached. Source: | `IncludeHeaders` | `[]string` | `nil` (all) | Whitelist of response headers to store. When set, only these are kept. | | `ExcludeHeaders` | `[]string` | `["set-cookie"]` | Response headers to drop from the stored set (applied after `IncludeHeaders`). | | `IgnoreCacheControl` | `bool` | `false` | Store responses whatever their `Cache-Control` says. By default `no-store`/`private` skip and `max-age=N` caps the TTL. | +| `SetTimeout` | `time.Duration` | `1s` | How long the middleware waits for each `Store.Set`. Zero selects the default; negative waits for ever. A `Set` not back in time is abandoned silently: the response is sent as usual with `X-Cache: MISS` and is not stored. A store whose `Set` can legitimately take longer (Redis over a WAN, a saturated pool) needs a larger value, for example `5 * time.Second`. New in v1.6.0 ([celeris#1151](https://github.com/goceleris/celeris/pull/1151)). | | `Skip` | `func(*Context) bool` | `nil` | Return `true` to skip caching for a request. | | `SkipPaths` | `[]string` | `nil` | Exact-match paths to skip. | -Source: `celeris/middleware/cache/config.go` (`Config`). +Source: `celeris/middleware/cache/config.go` (`Config`, `Config.SetTimeout`). ### Cache keys and `VaryHeaders` diff --git a/src/content/docs/static-files.md b/src/content/docs/static-files.md index 440ed9b..050ffa1 100644 --- a/src/content/docs/static-files.md +++ b/src/content/docs/static-files.md @@ -195,6 +195,23 @@ s.Static("/assets", "./public") // GET /assets/css/app.css → ./public/css/app.css (traversal-safe) ``` +> **Upgrading to v1.6.0: a `root` of `"."` or `"./"` now serves the tree below it.** +> Before v1.6.0 `Static` (and `static.Config.Root`) with `"."` or `"./"` served nothing +> below the root, since v1.3.3. A configuration that was inert +> then **starts to serve the working directory after the upgrade**, with its sources and +> configuration, so name a directory that holds only what is public +> ([celeris#1152](https://github.com/goceleris/celeris/pull/1152)). Since v1.6.0 files +> and directories whose name starts with a dot (`.env`, `.git`, `.ssh`) are not served +> by default (a 404 whether or not they exist; `.well-known` is served), and `Static` +> has no opt-out. A `root` that resolves to the root of a filesystem (`"/"`, `"."` when +> the working directory is `"/"`, as in a systemd service or a container without a +> `WORKDIR`) panics at registration; to serve one on purpose use the middleware with +> `AllowFilesystemRoot`, which serves every file the process can read. The dotfile rule +> looks only at names in the request path and at where symlinks lead: it is not a +> secrets filter, so keep secrets out of the served directory. Source: +> `celeris/server.go` (`Server.Static`), `celeris/middleware/static/config.go` +> (`Config.Root`, `Config.Dotfiles`). + On a group, the group's prefix is prepended as usual: ```go @@ -263,6 +280,8 @@ s.Use(static.New(static.Config{ | `SPA` | `bool` | `false` | Single-page-app mode: requests for non-existent files serve `Index` instead of falling through. | | `MaxAge` | `time.Duration` | `0` | Sets `Cache-Control: public, max-age=N`. Zero means no `Cache-Control` header. | | `Compress` | `bool` | `false` | Serve pre-compressed `.br`/`.gz` siblings when the client's `Accept-Encoding` allows. Brotli is preferred over gzip. Works with both `Root` and `FS`. | +| `Dotfiles` | `static.DotfilesPolicy` | `static.DotfilesIgnore` | What to do with a request whose path below the root has a segment that starts with a dot (`.env`, `.git`): `DotfilesIgnore` answers as if the file did not exist (the next handler, usually the router's 404), `DotfilesDeny` answers `403`, `DotfilesAllow` serves it, as before v1.6.0. `.well-known` is served. A symlink that leads to a hidden file is refused like it. Not a secrets filter. | +| `AllowFilesystemRoot` | `bool` | `false` | Acknowledges a `Root` that resolves to the root of a filesystem, which `New` otherwise refuses with a panic naming the resolved path. Such a `Root` serves every file the process can read. | | `Skip` | `func(c *celeris.Context) bool` | `nil` | Skip the middleware for requests where this returns `true`. | | `SkipPaths` | `[]string` | `nil` | Skip the middleware for these exact request paths. | @@ -473,6 +492,9 @@ does not coalesce a request with a `Range` header. [`StreamWriter`](#files-larger-than-100-mb). - **Setting disposition after writing.** Call `Attachment`/`Inline` (and any `SetHeader`) **before** `File`/`FileFromDir`/`FileFromFS` writes the response. +- **A `Root` of `.` serves the working directory.** Since v1.6.0 it is served (without + its hidden files); name the directory that holds what is public. See the callout under + [`Static`](#one-line-static-serving--static). - **`Root` *and* `FS` both set.** The middleware uses `FS` and ignores `Root`. Set exactly one. - **Prefix that isn't a clean segment.** `Prefix` must start with `/`, and it From a5ea41d076eb315127eff4ea4c4af3f76f1efd5f Mon Sep 17 00:00:00 2001 From: Albert Bausili Date: Wed, 4 Nov 2026 02:53:47 +0100 Subject: [PATCH 3/3] docs: review nits on the v1.6.0 catch-up (static security fix and FileFromDir, WriteTimeout example, version markers, cache SetTimeout) --- src/content/docs/configuration.md | 19 ++++++++++------- src/content/docs/core-concepts.md | 7 ++++--- src/content/docs/middleware-content.md | 2 +- src/content/docs/static-files.md | 29 +++++++++++++++++++++----- 4 files changed, 41 insertions(+), 16 deletions(-) diff --git a/src/content/docs/configuration.md b/src/content/docs/configuration.md index 9e2d920..0a4024b 100644 --- a/src/content/docs/configuration.md +++ b/src/content/docs/configuration.md @@ -220,9 +220,14 @@ With `DisableDeferAccept` the deferral is gone: a silent connection is closed at `ReadTimeout` and `IdleTimeout`; they look at their connections in a sweep, so those two are enforced up to a sweep period after their time. `WriteTimeout` is judged in windows that end at a sweep (see `Config.WriteTimeout`): on `epoll`, a 1 s -`WriteTimeout` closed a client that reads nothing at about 2 s in the PR's measurement, -which is the documented stall-rule windows, and the sweep only adds its own lateness to -that. Source: `celeris/config.go` (`Config.ReadTimeout`, `Config.WriteTimeout`). +`WriteTimeout` closed a client whose application reads nothing at about 2 s in +[celeris#1197](https://github.com/goceleris/celeris/pull/1197)'s measurement. Those are +the documented stall-rule windows, and the sweep only adds its own lateness to them. The +2 s is the case where the client's kernel acknowledged at least the 64 KiB floor of +[the minimum-rate rule below](#sse-needs-no--1-other-long-lived-responses-do) in the +first window (its receive buffer took it); a client that acknowledges less is closed at +the end of the first window, about one `WriteTimeout` (the PR records one close at +1.242 s). Source: `celeris/config.go` (`Config.ReadTimeout`, `Config.WriteTimeout`). - **`epoll`: every 25 ms, whatever the timeouts are.** Each worker arms a 25 ms kernel timer (a `timerfd`) at start, so `ReadTimeout` and `IdleTimeout` fire within about 30 ms @@ -265,8 +270,8 @@ limiter, cache, session and JWT expiry, the stores). ### HTTP/2 timeouts on the native engines -On `epoll`, `io_uring` and `adaptive`, the HTTP/2 deadlines are per request and per -connection, not per read +Since celeris v1.6.0, on `epoll`, `io_uring` and `adaptive`, the HTTP/2 deadlines are +per request and per connection, not per read ([celeris#1150](https://github.com/goceleris/celeris/pull/1150); `celeris/config.go` (`Config.ReadTimeout`, `Config.ReadHeaderTimeout`)): @@ -315,8 +320,8 @@ A response that is **not** detached is still under `WriteTimeout`: a slow large download, or a chunked body written with `StreamWriter` without `Detach`. On `std` the deadline is absolute for the whole response, so a response that takes longer than `WriteTimeout` ends however steadily the client reads (measured: a streaming body that -sends every 200 ms is cut at `WriteTimeout`). On `epoll` (and `adaptive` while it runs on -it), a response the kernel did not accept whole is instead bounded by a minimum rate on +sends every 200 ms is cut at `WriteTimeout`). Since celeris v1.6.0, on `epoll` (and +`adaptive` while it runs on it), a response the kernel did not accept whole is instead bounded by a minimum rate on the bytes the client acknowledges, 64 KiB per `WriteTimeout`, with no elapsed-time ceiling: a client that reads steadily above that floor is not cut however long the response takes ([celeris#1119](https://github.com/goceleris/celeris/pull/1119), diff --git a/src/content/docs/core-concepts.md b/src/content/docs/core-concepts.md index 1ee9aea..bd716aa 100644 --- a/src/content/docs/core-concepts.md +++ b/src/content/docs/core-concepts.md @@ -334,9 +334,10 @@ on average (lagging the rate by no more than four windows' worth) is not cut how long its response takes, and a `PING`, or a `WINDOW_UPDATE` of one byte, is not progress. Clients on slow links need a `WriteTimeout` of 2 s or more, and 5 s where the links are slower than a megabyte a second; on `io_uring`, where the engine's own send -clock closes such a connection at 1 s, use 5 s for clients on slow links. With `WriteTimeout: -1` there is no deadline: the wait lasts until the client -grants window or the connection closes, and never ends if the I/O worker that would -see either is the one waiting (below). Source: `celeris/config.go` +clock closes such a connection at 1 s, use 5 s for clients on slow links. With +`WriteTimeout: -1` there is no deadline: the wait lasts until the client grants window +or the connection closes, and never ends if the I/O worker that would see either is the +one waiting (below). Source: `celeris/config.go` (`Config.WriteTimeout`). **If the event loop itself is stuck, the rate rule is replaced by a bound.** While an diff --git a/src/content/docs/middleware-content.md b/src/content/docs/middleware-content.md index 847e5d0..4d1b1e3 100644 --- a/src/content/docs/middleware-content.md +++ b/src/content/docs/middleware-content.md @@ -244,7 +244,7 @@ store transport error sets `X-Cache: ERROR` and passes through uncached. Source: | `IncludeHeaders` | `[]string` | `nil` (all) | Whitelist of response headers to store. When set, only these are kept. | | `ExcludeHeaders` | `[]string` | `["set-cookie"]` | Response headers to drop from the stored set (applied after `IncludeHeaders`). | | `IgnoreCacheControl` | `bool` | `false` | Store responses whatever their `Cache-Control` says. By default `no-store`/`private` skip and `max-age=N` caps the TTL. | -| `SetTimeout` | `time.Duration` | `1s` | How long the middleware waits for each `Store.Set`. Zero selects the default; negative waits for ever. A `Set` not back in time is abandoned silently: the response is sent as usual with `X-Cache: MISS` and is not stored. A store whose `Set` can legitimately take longer (Redis over a WAN, a saturated pool) needs a larger value, for example `5 * time.Second`. New in v1.6.0 ([celeris#1151](https://github.com/goceleris/celeris/pull/1151)). | +| `SetTimeout` | `time.Duration` | `1s` | How long the middleware waits for each `Store.Set`. Zero selects the default; negative waits for ever. A `Set` not back in time is abandoned silently: the response is sent as usual with `X-Cache: MISS` and is not stored (a store that ignores the `Set`'s context may still write it later). `MemoryStore` and `store.MemoryKV` are not bounded. A store whose `Set` can legitimately take longer (Redis over a WAN, a saturated pool) needs a larger value, for example `5 * time.Second`. New in v1.6.0 ([celeris#1151](https://github.com/goceleris/celeris/pull/1151)). | | `Skip` | `func(*Context) bool` | `nil` | Return `true` to skip caching for a request. | | `SkipPaths` | `[]string` | `nil` | Exact-match paths to skip. | diff --git a/src/content/docs/static-files.md b/src/content/docs/static-files.md index 050ffa1..94ab994 100644 --- a/src/content/docs/static-files.md +++ b/src/content/docs/static-files.md @@ -109,6 +109,14 @@ s.GET("/files/*name", func(c *celeris.Context) error { }) ``` +`FileFromDir` does not inherit the dotfile rule of `Static` and the static middleware: it +serves hidden files (`.env`), because the caller chooses the path. Since v1.6.0 it accepts +a `baseDir` of `"."` or `"./"` (the working directory) and of `"/"`, which reaches every +file the process can read; before, those were refused with a `400`. An empty `baseDir` +is still refused. Choose `baseDir` and `userPath` with care +(`celeris/context_response.go` (`Context.FileFromDir`), +[celeris#1152](https://github.com/goceleris/celeris/pull/1152)). + A request for `/files/../../etc/passwd` returns `400 invalid file path` rather than reading outside `./public`. A request that resolves to a directory also returns `400` — `FileFromDir` serves files only, not listings (use the @@ -202,13 +210,24 @@ s.Static("/assets", "./public") > configuration, so name a directory that holds only what is public > ([celeris#1152](https://github.com/goceleris/celeris/pull/1152)). Since v1.6.0 files > and directories whose name starts with a dot (`.env`, `.git`, `.ssh`) are not served -> by default (a 404 whether or not they exist; `.well-known` is served), and `Static` -> has no opt-out. A `root` that resolves to the root of a filesystem (`"/"`, `"."` when -> the working directory is `"/"`, as in a systemd service or a container without a -> `WORKDIR`) panics at registration; to serve one on purpose use the middleware with +> by default (`Static` answers 404 whether or not they exist; the middleware by default +> passes them on as missing files, and in SPA mode serves the index; `.well-known` is +> served), and `Static` has no opt-out. A `root` that resolves to the root of a +> filesystem (`"/"`, `"."` when the working directory is `"/"`, as in a systemd service +> or a container without a `WORKDIR`) panics at registration (`Static`) or at +> `static.New` (the middleware); to serve one on purpose use the middleware with > `AllowFilesystemRoot`, which serves every file the process can read. The dotfile rule > looks only at names in the request path and at where symlinks lead: it is not a -> secrets filter, so keep secrets out of the served directory. Source: +> secrets filter, so keep secrets out of the served directory. +> +> **Security fix in the same release.** A `root` made only of `..` segments (`".."`, +> `"../.."`, `"./.."`) served files above the directory it names (`GET /../secret`) from +> v1.3.3 through v1.5.11, for `Static`, the middleware and `FileFromDir`. Since v1.6.0 +> such a root is resolved against the working directory once, at registration or +> `static.New`, and nothing outside that directory is served; prefer an absolute root. +> Also since v1.6.0: devices, FIFOs and sockets are not served (the middleware +> answers them as a missing file; `File`, `FileFromDir` and `FileFromFS` return a `404` +> "not a regular file"), and `Static` answers a missing file with 404 (it was 500). Source: > `celeris/server.go` (`Server.Static`), `celeris/middleware/static/config.go` > (`Config.Root`, `Config.Dotfiles`).