From 0bef0c00eee834af77131c16eca9949c4376427d Mon Sep 17 00:00:00 2001 From: Steven Sklar Date: Mon, 14 Sep 2026 09:42:43 -0400 Subject: [PATCH] docs(operator): document native cold storage (spec.coldStorage) Operator v0.3.0 adds native cold storage. Document the structured spec.coldStorage block and its operational contract: - configuration: the coldStorage block, default identity-scoped root, version floor, ColdStorageHealthy/status.coldStorage, legacy cold.storage.* key handling, and the new immutability rules including QuestDBObjectStore physical coordinates - high availability: the settled cold-manager Planned precheck (ColdManagerMoveRequired) and Emergency behavior while the manager is lost - database operations: a "Move the cold-storage manager" runbook - known limitations: the manager-off-the-departing-primary requirement and the default manager: 1 friction - database-level cold storage pages: steer operator-managed clusters to spec.coldStorage.manager instead of SWITCH COLD STORAGE ROLE The generated releases.md and reference/api.md pages follow separately via make docs-sync once v0.3.0 is tagged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01ASL866WftGbwTF6PUqV76Y --- documentation/concepts/cold-storage.md | 2 +- .../configuration.md | 91 +++++++++++++++++-- .../high-availability.md | 36 +++++++- .../known-limitations.md | 17 ++++ .../operations/database.md | 43 +++++++++ documentation/operations/cold-storage.md | 11 +++ 6 files changed, 192 insertions(+), 8 deletions(-) diff --git a/documentation/concepts/cold-storage.md b/documentation/concepts/cold-storage.md index c150195c1..2d7979755 100644 --- a/documentation/concepts/cold-storage.md +++ b/documentation/concepts/cold-storage.md @@ -166,7 +166,7 @@ A legitimate change of version, after restoring onto a different host or migrati - **No materialized views.** A materialized view accepts no storage policy at all, so it cannot be tiered to object storage. It uses [TTL](/docs/query/sql/alter-mat-view-set-ttl/) for retention instead. - **No downgrade after the first upload.** Enabling remote upload is a one-way version change. - **Upgrade replicas first.** Older instances and open source nodes do not understand the seal event and suspend WAL apply rather than skipping it. -- **Operator-driven manager failover.** The manager role moves at runtime, but nothing detects manager death or elects a replacement, and a hot-switched role does not survive a restart. +- **Operator-driven manager failover.** The manager role moves at runtime, but nothing detects manager death or elects a replacement, and a hot-switched role does not survive a restart. On Kubernetes, the [Enterprise Kubernetes Operator](/docs/enterprise-kubernetes-operator/configuration/#cold-storage) declares the manager in `spec.coldStorage.manager` and re-applies it across restarts; electing a replacement is still a human decision. - **Backups do not contain cold data.** A backup captures a cold partition's local metadata and symbol indexes, but not its `data.parquet` bytes, which exist only in the object store. The database backup and the object store prefix have to be kept, and recovered, as one set. See [Backup and restore](/docs/operations/backup/#cold-storage-partitions). - **No idle warm cache.** Chunks are shared only while an active read holds a lease. - **Fixed key layout.** Object key templates are not configurable. diff --git a/documentation/enterprise-kubernetes-operator/configuration.md b/documentation/enterprise-kubernetes-operator/configuration.md index dcbfae50a..3d01c6de8 100644 --- a/documentation/enterprise-kubernetes-operator/configuration.md +++ b/documentation/enterprise-kubernetes-operator/configuration.md @@ -69,10 +69,12 @@ isolation boundary: - `spec.replication.root` defaults to the identity-scoped `db///`. Leave it unset unless deliberately adopting an existing WAL stream. +- `spec.coldStorage.objectStoreRef.root` defaults to the identity-scoped + `cold///` and is immutable once the cluster is created. A provider-level `root` on `QuestDBObjectStore` does **not** add a base prefix; -the per-use backup or replication root overrides it. Scope cloud permissions to -the effective per-use prefixes. +the per-use backup, replication, or cold-storage root overrides it. Scope cloud +permissions to the effective per-use prefixes. ## Backup and HA @@ -118,6 +120,69 @@ runbooks for routing and failover. The WAL cleaner is enabled by default. Do not disable it unless another process owns replication-WAL retention; otherwise WAL grows without bound. +## Cold storage + +Native [cold storage](/docs/concepts/cold-storage/) tiers sealed partitions to +object storage. On QuestDB Enterprise 4.0.0 or later, enable it through +`spec.coldStorage`: + +```yaml +spec: + coldStorage: + objectStoreRef: + name: questdb-cold-store + # root: cold/questdb/ # defaults to cold/// + # manager: 1 # default +``` + +`objectStoreRef` names a same-namespace `QuestDBObjectStore`. The operator +resolves it into a connection Secret named `-cold-storage-store`, +mounts that Secret on every database Pod, and enables cold storage on every +instance; QuestDB pods perform all object-store I/O. Recognizable QuestDB +Enterprise image tags below 4.0.0 are rejected at admission. Unknown custom +tags and digest-only images are allowed and fail closed at runtime if the +engine cannot answer cold-storage observations. + +`manager` selects the 1-based instance serial that holds the cold-storage +[manager role](/docs/concepts/cold-storage/#roles); every other instance is a +refresher. Changing `manager` requests an engine-level handoff: the operator +demotes the old manager, verifies it settled as a refresher, then promotes the +replacement — never two managers at once. Do not run +[`SWITCH COLD STORAGE ROLE`](/docs/query/sql/switch-cold-storage-role/) +against operator-managed instances; change `spec.coldStorage.manager` and +follow the +[manager-move runbook](/docs/enterprise-kubernetes-operator/operations/database/#move-the-cold-storage-manager). + +Run the manager on a replica. The default `manager: 1` normally selects the +primary, and a `Planned` promotion requires the manager settled on an instance +other than the departing primary — otherwise the promotion fails with +`ColdManagerMoveRequired`. See +[Planned prechecks](/docs/enterprise-kubernetes-operator/high-availability/#planned-prechecks). + +Watch the `ColdStorageHealthy` condition and `status.coldStorage`: + +```sh +kubectl get questdbcluster -n \ + -o jsonpath='{range .status.conditions[?(@.type=="ColdStorageHealthy")]}{.type}{"="}{.status}{" reason="}{.reason}{" message="}{.message}{"\n"}{end}{.status.coldStorage}{"\n"}' +``` + +`True/ManagerReady` with `status.coldStorage.currentManager` naming the +desired instance and a positive `managerTerm` is the healthy steady state. +While a handoff is in flight, `handoffSource` names the demoted manager until +the replacement is observed stable. `False/ManagerHandoffBlocked` fails closed +— for example while the current manager is unreachable — rather than risking +two managers. + +The operator configures and observes cold storage; it never reads or deletes +cold objects, never assigns +[storage policies](/docs/concepts/storage-policy/) to tables, and never +garbage-collects the prefix. Assign policies with +[`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/). +`spec.coldStorage` may be added to a running cluster but cannot be removed, +and `objectStoreRef` is immutable: changing the store or root would re-point +live cold data. `manager` must not exceed `instances`, so lower `instances` +only after moving the manager to a remaining serial. + ## Storage Each instance receives its own persistent volume: @@ -330,9 +395,16 @@ Object-store keys are **not** rejected, but operator-provided primary Never put access keys, passwords, or credential-bearing connection strings in `spec.config`: the custom resource and rendered ConfigMap are plaintext. -Additional backup destinations (`backup.object.store.1` through `.9`) and -`cold.storage.object.store` are suitable only for credential-free, -ambient-identity settings until a Secret-backed mechanism is available. +Additional backup destinations (`backup.object.store.1` through `.9`) are +suitable only for credential-free, ambient-identity settings until a +Secret-backed mechanism is available. + +For cold storage, use [`spec.coldStorage`](#cold-storage): its connection is +Secret-backed. When `spec.coldStorage` is absent, the legacy +`cold.storage.enabled`, `cold.storage.object.store`, and `cold.storage.role` +keys remain supported for credential-free settings. When it is present, those +three keys are operator-owned and rejected, while advanced `cold.storage.*` +and `storage.policy.*` tuning stays user-managed. The `qwp.udp.*` receiver keys are also operator-owned: `qwp.udp.enabled` and `qwp.udp.bind.to` are set through [`spec.protocols.qwp.udp`](#qwp-udp), and @@ -375,11 +447,18 @@ I/O even though the operator preserves the single-writer gate. Important immutable choices include: - `spec.objectStoreRef` once set; +- `spec.coldStorage` in presence once set, and `spec.coldStorage.objectStoreRef` + in value; - `spec.storage.storageClassName`; - `spec.bootstrap` in presence and value; - `spec.protocols.pgwire.tls` in presence; - `spec.replication.root` in presence and value; and -- `QuestDBObjectStore.spec.provider`. +- `QuestDBObjectStore.spec.provider` and its physical coordinates: the S3 + bucket, region, and endpoint, and the Azure container, account name, and + endpoint. Credential references, Secret contents, and non-coordinate + transport options remain mutable, so + [credential rotation](#rotate-static-object-store-credentials-safely) is + unaffected. Storage size is expand-only. For exact transition rules and less common fields, use the diff --git a/documentation/enterprise-kubernetes-operator/high-availability.md b/documentation/enterprise-kubernetes-operator/high-availability.md index 325e9fd07..750fdf018 100644 --- a/documentation/enterprise-kubernetes-operator/high-availability.md +++ b/documentation/enterprise-kubernetes-operator/high-availability.md @@ -52,7 +52,11 @@ It does not use the database's in-place [role switch](/docs/high-availability/failover/) (`SWITCH ROLE`, `POST /lifecycle/switch`). Do not run those against instances managed by the operator: `status.currentPrimary` and the PVC role labels would no longer -describe the cluster. +describe the cluster. The same applies to +[`SWITCH COLD STORAGE ROLE`](/docs/query/sql/switch-cold-storage-role/): +change +[`spec.coldStorage.manager`](/docs/enterprise-kubernetes-operator/operations/database/#move-the-cold-storage-manager) +instead. ::: @@ -84,6 +88,27 @@ kubectl get questdbcluster -n \ -o jsonpath='{.status.replication.activePromotion}{"\n"}' ``` +On a cluster with +[`spec.coldStorage`](/docs/enterprise-kubernetes-operator/configuration/#cold-storage), +also require settled cold-manager ownership away from the departing primary: +current-generation `ColdStorageHealthy=True/ManagerReady`, +`status.coldStorage.currentManager` equal to `spec.coldStorage.manager` and +not the current primary, a positive `managerTerm`, and an empty +`handoffSource`: + +```sh +kubectl get questdbcluster -n \ + -o jsonpath='manager={.spec.coldStorage.manager}{" current="}{.status.coldStorage.currentManager}{" term="}{.status.coldStorage.managerTerm}{" handoffSource="}{.status.coldStorage.handoffSource}{"\n"}' +``` + +Otherwise a non-no-op `Planned` promotion fails fast with +`ColdManagerMoveRequired`. The promotion never moves the cold manager itself: +[move `spec.coldStorage.manager`](/docs/enterprise-kubernetes-operator/operations/database/#move-the-cold-storage-manager) +to a ready replica, wait for `ManagerReady`, and create a new promotion +object. Moving it to the promotion target is allowed but leaves the manager on +the new primary after the cutover. An active promotion pauses new cold-manager +handoffs, so move the manager before creating the promotion. + The defaults reflect two different costs: - `catchUpTimeoutSeconds: 900` bounds pre-drain catch-up while the old primary @@ -190,6 +215,15 @@ promotion, the fenced loss state must have zero ready `-rw` EndpointSlice endpoints; afterward the selected target is the only live primary. Record the accepted recovery point and any expected lost-write window. +Emergency promotion never waits on cold-storage ownership. If the lost primary +was also the cold manager, `ColdStorageHealthy` reports +`False/ManagerHandoffBlocked` and the handoff fails closed rather than risking +two managers: refreshers continue serving already-offloaded partitions, while +new uploads and remote garbage collection wait. Cold health recovers through +the ordinary handoff once the fenced instance returns as a replica. To place +the manager elsewhere afterward, +[move `spec.coldStorage.manager`](/docs/enterprise-kubernetes-operator/operations/database/#move-the-cold-storage-manager). + ## If promotion stalls or fails Read the promotion's status first: diff --git a/documentation/enterprise-kubernetes-operator/known-limitations.md b/documentation/enterprise-kubernetes-operator/known-limitations.md index d1442207b..f3b72408d 100644 --- a/documentation/enterprise-kubernetes-operator/known-limitations.md +++ b/documentation/enterprise-kubernetes-operator/known-limitations.md @@ -51,6 +51,23 @@ Increasing `replication.primary.keepalive.interval` lengthens this stale-direct-client window. Leave its 10-second default unless QuestDB advises otherwise. +### Planned promotion requires the cold manager off the departing primary + +With +[`spec.coldStorage`](/docs/enterprise-kubernetes-operator/configuration/#cold-storage), +a `Planned` promotion starts only when the cold-storage manager is settled on +an instance other than the departing primary; otherwise it fails fast with +`ColdManagerMoveRequired`, and there is no override. The default `manager: 1` +normally selects the primary, so +[move the manager](/docs/enterprise-kubernetes-operator/operations/database/#move-the-cold-storage-manager) +to a replica before the first `Planned` cutover. The operator never elects a +replacement manager on its own: changing `spec.coldStorage.manager` is the +only path, and an unreachable current manager blocks the handoff fail-closed +(`ManagerHandoffBlocked`) rather than risking two managers. `Emergency` +promotion never waits on cold ownership; while the manager is lost, new +uploads and remote garbage collection pause and refreshers continue serving +already-offloaded partitions. + ### Promotion can be unbounded A live but hung final upload can hold `Draining`: the operator cannot diff --git a/documentation/enterprise-kubernetes-operator/operations/database.md b/documentation/enterprise-kubernetes-operator/operations/database.md index 6de848695..e1d17c5a0 100644 --- a/documentation/enterprise-kubernetes-operator/operations/database.md +++ b/documentation/enterprise-kubernetes-operator/operations/database.md @@ -304,6 +304,49 @@ writer-health or separate follower contract, and the expected retained or deleted replica PVCs. Confirm each current primary/replica PVC's `questdb.io/role` label matches that role. +## Move the cold-storage manager + +With +[`spec.coldStorage`](/docs/enterprise-kubernetes-operator/configuration/#cold-storage), +the cold-storage [manager role](/docs/concepts/cold-storage/#roles) is moved +by changing `spec.coldStorage.manager`, never by running +[`SWITCH COLD STORAGE ROLE`](/docs/query/sql/switch-cold-storage-role/) +against instances. Run the manager on a replica: that moves upload, manifest, +and garbage-collection work off the primary, and a `Planned` promotion +requires the manager settled away from the departing primary. + +Before the change, require current generation, +`ColdStorageHealthy=True/ManagerReady`, and an empty `handoffSource`: + +```sh +kubectl get questdbcluster -n \ + -o jsonpath='{range .status.conditions[?(@.type=="ColdStorageHealthy")]}{.type}{"="}{.status}{" reason="}{.reason}{" observed="}{.observedGeneration}{"\n"}{end}manager={.spec.coldStorage.manager}{" current="}{.status.coldStorage.currentManager}{" term="}{.status.coldStorage.managerTerm}{" handoffSource="}{.status.coldStorage.handoffSource}{"\n"}' +``` + +Choose the serial of a ready replica (`-3` has serial `3`) and patch: + +```sh +kubectl patch questdbcluster -n --type merge \ + -p '{"spec":{"coldStorage":{"manager":3}}}' +``` + +The operator demotes the old manager, verifies it settled as a refresher, +then promotes the replacement — never two managers at once. While the handoff +is in flight, `status.coldStorage.handoffSource` names the demoted instance. +Uploads and remote garbage collection pause during the short managerless +interval; reads continue everywhere. Wait for `True/ManagerReady` with +`currentManager` naming the target and a positive `managerTerm`. + +`False/ManagerHandoffBlocked` fails closed rather than risking two managers — +for example while the current manager is unreachable. Recover the named +instance; do not work around the block with direct SQL. Two interactions to +plan around: + +- An active promotion pauses new cold-manager handoffs. Finish or resolve the + cutover first. +- `manager` must not exceed `instances`, so scale in only after moving the + manager to a remaining serial. + ## Grow storage Storage is expand-only, and `spec.storage.storageClassName` is immutable. diff --git a/documentation/operations/cold-storage.md b/documentation/operations/cold-storage.md index 59512518e..f10538eb9 100644 --- a/documentation/operations/cold-storage.md +++ b/documentation/operations/cold-storage.md @@ -132,6 +132,17 @@ WHERE timestamp IN '2026-02-10'; The manager role moves between instances at runtime, with no restart. Role switching requires database administrator (system admin) privileges. See [`SWITCH COLD STORAGE ROLE`](/docs/query/sql/switch-cold-storage-role/) for the full syntax. +:::note + +On a cluster managed by the +[Enterprise Kubernetes Operator](/docs/enterprise-kubernetes-operator/), do not +run these statements directly. Change `spec.coldStorage.manager` and follow the +[operator's manager-move runbook](/docs/enterprise-kubernetes-operator/operations/database/#move-the-cold-storage-manager): +the operator performs this same demote-then-promote sequence and would treat a +hand-switched role as drift. + +::: + The supported handoff is two steps, in this order: ```questdb-sql title="1. On the current manager"