diff --git a/template-catalog/templates/airflow.mdx b/template-catalog/templates/airflow.mdx index b7605a49..58c75f46 100644 --- a/template-catalog/templates/airflow.mdx +++ b/template-catalog/templates/airflow.mdx @@ -490,7 +490,7 @@ Template version 1.5.0 removed the credentials that 1.4.x took as plain Helm val Template versions through 1.5.0 created their own GVC. Version 2.0.0 deploys into an existing GVC and does not declare one — and Helm deletes what a chart stops declaring. -**Never `helm upgrade` a 1.x release onto 2.0.0.** The upgrade drops the GVC from the manifest and deletes it, with every workload, volume set and identity inside — your metadata database and your DAGs — while reporting success. The chart refuses to render when your values still carry the `gvc` key, and the refusal leaves the running release untouched: +**Never run a plain `helm upgrade` on a 1.x release onto 2.0.0.** The upgrade drops the GVC from the manifest and deletes it, with every workload, volume set and identity inside — your metadata database and your DAGs — while reporting success. The chart refuses to render when your values still carry the `gvc` key, and the refusal leaves the running release untouched: ```text Error: execution error at (airflow/templates/workload-webserver.yaml:1:4): airflow 2.0.0: the `gvc` @@ -499,7 +499,7 @@ into, and `gvc.locations` became the single top-level `location`. DO NOT `helm u release onto 2.0.0 ... ``` -An upgrade run with no values at all has no `gvc` key to catch and proceeds, so the rule matters more than the guard. The only supported path is a **new release** against an existing GVC. +An upgrade run with no values at all has no `gvc` key to catch and proceeds, so the rule matters more than the guard. To keep the GVC, follow [Upgrading In Place from 1.5.0](#upgrading-in-place-from-1-5-0). The alternative is a **new release** against an existing GVC, as below. @@ -553,6 +553,50 @@ Values that moved or were removed in 2.0.0: | — | `airflow.celeryWorker.replicas`, the fixed worker count used when KEDA is off | | — | `internalAccess.type` / `internalAccess.workloads` | +### Upgrading In Place from 1.5.0 + +Template versions through 1.5.0 created their own GVC, and 2.1.0 deploys into an existing one. A plain `cpln helm upgrade` across that change deletes the GVC the 1.x release created, with every workload, volume set and identity inside it. These steps mark that GVC "keep" first, so the upgrade leaves it and everything in it in place. It covers a 1.5.0 release (upgrade an older 1.x to 1.5.0 first) that runs in one location; with more than one entry in `gvc.locations`, use [Migrating from 1.x](#migrating-from-1-x) instead. + +Create the database credentials secret under a **new** name first, holding the values the database runs with today. 1.5.0 keeps them in `RELEASE_NAME-airflow-config`, and the upgrade deletes that secret: + +```bash +cpln secret reveal RELEASE_NAME-airflow-config -o yaml # username and password in use today +cpln secret create-dictionary --name RELEASE_NAME-airflow-db \ + --entry username='YOUR_CURRENT_USERNAME' --entry password='YOUR_CURRENT_PASSWORD' --entry database='YOUR_CURRENT_DATABASE' +``` + +`YOUR_CURRENT_DATABASE` is `postgres.config.database` of the 1.5.0 release, `airflow` unless you changed it. Then mark the GVC and upgrade: + +```bash +cpln helm get values RELEASE_NAME -o yaml > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (see below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/airflow --version 2.1.0 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- `OLD_GVC_NAME` is the GVC the 1.x release created (its `gvc.name`), not the GVC you ran the original install against. +- The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`. If it does not, stop: the GVC was not protected. + +`values-new.yaml` is `values-old.yaml` with these changes: + +```yaml +# gvc: delete the whole block +location: aws-eu-central-1 # the one entry of the old gvc.locations +postgres: + config: # delete username, password and database + credentialsSecretName: RELEASE_NAME-airflow-db # the secret created above +keda: + enabled: true # what 1.5.0 ran with (its default was true) +``` + +If `values-old.yaml` has no `gvc` key, the release ran on 1.5.0's defaults: `OLD_GVC_NAME` is `airflow` and `location` is `aws-eu-central-1`. Keep `keda.enabled` as 1.5.0 had it: the GVC it created has KEDA on exactly when that was `true`, and `true` against a GVC with KEDA off stops the webserver from starting. + +The upgrade restarts PostgreSQL, Redis, the webserver and the workers, so Airflow is unavailable for a few minutes. A workload can wait with `is not allowed to reveal the secret` in `get-deployments` while access to the new secrets propagates; it clears by itself (see [Upgrading from 2.0.0](#upgrading-from-2-0-0)). + +Afterwards the GVC belongs to you, not the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME`. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Upgrading from 2.0.0 Template version 2.1.0 removed `postgres.config.username`, `postgres.config.password` and `postgres.config.database`: the metadata database credentials are now a prerequisite secret. Carrying the old keys forward fails the render. @@ -627,9 +671,9 @@ The upgrade restarts PostgreSQL, the webserver and the workers (they now read th **Symptom:** upgrading a 1.x release to 2.0.0 fails at render with the message quoted in [Migrating from 1.x](#migrating-from-1-x). - **Cause:** your values still carry the 1.x `gvc` key. The refusal is deliberate; the in-place upgrade would delete the GVC the 1.x chart created. + **Cause:** your values still carry the 1.x `gvc` key. The refusal is deliberate; a plain upgrade would delete the GVC the 1.x chart created. - **Fix:** do not remove the key and retry. Install 2.0.0 as a new release and move the data as described in [Migrating from 1.x](#migrating-from-1-x). + **Fix:** do not remove the key and retry. Follow [Upgrading In Place from 1.5.0](#upgrading-in-place-from-1-5-0), or install 2.0.0 as a new release and move the data as described in [Migrating from 1.x](#migrating-from-1-x). **Symptom:** the webserver is ready but its public URL returns `403`. @@ -658,7 +702,7 @@ The upgrade restarts PostgreSQL, the webserver and the workers (they now read th - **Create the auth and database credentials secrets before installing.** Without either the deployment wedges with no log output; `cpln workload get-deployments` is the one place the missing secret is named. - **The Fernet key can never be rotated.** Generate it once, keep it, and back it up with the auth secret. -- **Never upgrade a 1.x release onto 2.0.0 in place** — it deletes the GVC the 1.x chart created and everything in it. See [Migrating from 1.x](#migrating-from-1-x). +- **Never run a plain `helm upgrade` on a 1.x release onto 2.0.0** — it deletes the GVC the 1.x chart created and everything in it. See [Upgrading In Place from 1.5.0](#upgrading-in-place-from-1-5-0) or [Migrating from 1.x](#migrating-from-1-x). - **`location` must be a location of your GVC.** Otherwise the install succeeds and nothing starts anywhere. - **Enable KEDA on the GVC before setting `keda.enabled: true`**, or the workers sit at `keda.minScale` forever with no error. - **Rotating the database password needs `ALTER ROLE` plus a forced redeployment of the webserver and workers** — editing the secret alone changes nothing; see [Rotating the Database Password](#rotating-the-database-password). diff --git a/template-catalog/templates/clickhouse.mdx b/template-catalog/templates/clickhouse.mdx index 0d2e3e40..910780ba 100644 --- a/template-catalog/templates/clickhouse.mdx +++ b/template-catalog/templates/clickhouse.mdx @@ -521,12 +521,53 @@ The server startup script writes the `password` key of your credentials secret i This procedure follows from how the chart's startup script applies the secret; it has not been exercised end to end on this template. Confirm the new password with the verify snippet under [Connecting](#connecting) after the redeployment completes. +### Upgrading In Place from 2.8.0 + +Template versions through 2.8.0 created their own GVC, and 3.0.0 deploys into an existing one. A plain `cpln helm upgrade` across that change deletes the GVC the 2.x release created, with every workload, volume set and identity inside it. These steps mark that GVC "keep" first, so the upgrade leaves it and everything in it in place. It covers a 2.8.0 release (upgrade an older 2.x to 2.8.0 first) whose locations all have at least 1 replica; if one has `replicas: 0`, use [Upgrading from 2.x](#upgrading-from-2-x) instead. + +```bash +cpln helm get values RELEASE_NAME -o yaml > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (see below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/clickhouse --version 3.0.0 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- `OLD_GVC_NAME` is the GVC the 2.x release created (its `gvc.name`), not the GVC you ran the original install against. +- The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`. If it does not, stop: the GVC was not protected. + +`values-new.yaml` is `values-old.yaml` with these changes: + +```yaml +# gvc: delete the whole block +locations: # the old gvc.locations: same entries, SAME ORDER (it sets shard numbers and Keeper members) + - name: aws-us-east-2 + replicas: 1 +server: + internalAccess: # renamed from server.internal_access, same content + type: same-gvc + workloads: [] +keeper: + internalAccess: # renamed from keeper.internal_access, same content + type: same-gvc + workloads: [] +``` + +- If `values-old.yaml` has no `gvc` key, the release ran on 2.8.0's defaults: `OLD_GVC_NAME` is `clickhouse-gvc`, and `locations` is `aws-us-east-2`, `aws-us-west-2`, `aws-us-east-1` in that order, 1 replica each. +- Keep your storage settings exactly as they are. If `values-old.yaml` has no `aws` block, 2.8.0 used `bucket: clickhouse-s3-bucket`, `cloudAccountName: clickhouse-s3-cloudaccount` and `policyName: clickhouse-s3-policy`; write those out, because 3.0.0's defaults name a different bucket. +- With `type: workload-list`, `workloads` is now enforced (2.x ignored it): list every client that connects. This release's own workloads are added for you. + +Every server and Keeper restarts, in every location at once, so queries fail for a few minutes. + +Afterwards the GVC belongs to you, not the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME`. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Upgrading from 2.x Template versions through 2.8.0 created their own GVC, so that GVC is part of a 2.x release's manifest. Version 3.0.0 does not declare a GVC, and Helm deletes what a chart stops declaring. -**An in-place `helm upgrade` from any 2.x release to 3.0.0 is not supported and destroys the deployment.** The upgrade would delete the GVC the 2.x chart created and every workload, volume set and identity inside it — including your ClickHouse metadata and Keeper state — while reporting success. +**A plain `helm upgrade` from any 2.x release to 3.0.0 destroys the deployment.** To keep the GVC, follow [Upgrading In Place from 2.8.0](#upgrading-in-place-from-2-8-0) instead. The upgrade would delete the GVC the 2.x chart created and every workload, volume set and identity inside it — including your ClickHouse metadata and Keeper state — while reporting success. The chart refuses to render when your values still carry the 2.x `gvc` key, so an upgrade that passes your old values file stops before a single resource is touched: @@ -540,7 +581,7 @@ against an existing GVC, reload your data, then uninstall the old release. See ` in the README. ``` -**The guard cannot cover an upgrade run with no values at all.** A 2.x release installed on pure defaults has no `gvc` key for the chart to see, so nothing fires. Do not run `cpln helm upgrade` on a 2.x release against the 3.0.0 chart under any circumstances. The only supported path is a new release against an existing GVC, reloading the data, then removing the old release. +**The guard cannot cover an upgrade run with no values at all.** A 2.x release installed on pure defaults has no `gvc` key for the chart to see, so nothing fires. Never run a plain `cpln helm upgrade` on a 2.x release against the 3.0.0 chart. The alternatives are the in-place steps above, or a new release against an existing GVC, reloading the data, then removing the old release. What changed between 2.x and 3.0.0: @@ -663,7 +704,7 @@ Any removed key still present in your values is refused at render with a message **Cause:** your values still carry the 2.x `gvc` key. The refusal is deliberate — the upgrade it blocks would delete the GVC the 2.x chart created. - **Fix:** do not retry the upgrade with the key removed. Install 3.0.0 as a new release against an existing GVC, reload the data, and remove the old release as described in [Upgrading from 2.x](#upgrading-from-2-x). + **Fix:** do not retry the upgrade with the key removed. Follow [Upgrading In Place from 2.8.0](#upgrading-in-place-from-2-8-0), or install 3.0.0 as a new release against an existing GVC, reload the data, and remove the old release as described in [Upgrading from 2.x](#upgrading-from-2-x). **Symptom:** in a cluster mode, the first server replica in the first location exits with: @@ -695,7 +736,7 @@ Any removed key still present in your values is refused at render with a message ## Important Notes -- **Never run `cpln helm upgrade` on a 2.x release against 3.0.0** — it deletes the GVC the 2.x chart created and everything in it. Install a new release: [Upgrading from 2.x](#upgrading-from-2-x). +- **Never run a plain `cpln helm upgrade` on a 2.x release against 3.0.0** — it deletes the GVC the 2.x chart created and everything in it. See [Upgrading In Place from 2.8.0](#upgrading-in-place-from-2-8-0) or [Upgrading from 2.x](#upgrading-from-2-x). - **Create the credentials secret before installing.** A missing one wedges the deployment with no log output; `cpln workload get-deployments` is the one command that names it. - **The GVC must contain every location you list**, and may contain more. Undeclared GVC locations run nothing. - **Two locations is not supported.** Use one (single-node or single-shard) or three or more. diff --git a/template-catalog/templates/cockroach.mdx b/template-catalog/templates/cockroach.mdx index 438b0693..6c6c705d 100644 --- a/template-catalog/templates/cockroach.mdx +++ b/template-catalog/templates/cockroach.mdx @@ -429,12 +429,44 @@ cpln workload exec RELEASE_NAME-cockroach --gvc GVC_NAME \ Running the second statement without the first fails with `full cluster restore can only be run on a cluster with no tables or databases`. Follow progress with `SHOW JOBS` the same way, and verify afterwards with `SHOW DATABASES`. +### Upgrading In Place from 1.5.0 + +Template versions through 1.5.0 created their own GVC, and 2.0.0 deploys into an existing one. A plain `cpln helm upgrade` across that change deletes the GVC the 1.x release created, with every workload, volume set and identity inside it. These steps mark that GVC "keep" first, so the upgrade leaves it and everything in it in place. It covers a 1.5.0 release (upgrade an older 1.x to 1.5.0 first) whose locations all have at least 1 replica; if one has `replicas: 0`, use [Upgrading from 1.x](#upgrading-from-1-x) instead. + +```bash +cpln helm get values RELEASE_NAME -o yaml > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (see below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/cockroach --version 2.0.0 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- `OLD_GVC_NAME` is the GVC the 1.x release created (its `gvc.name`), not the GVC you ran the original install against. +- The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`. If it does not, stop: the GVC was not protected. + +`values-new.yaml` is `values-old.yaml` with one change: + +```yaml +# gvc: delete the whole block +locations: # the old gvc.locations: same entries, same order, same replicas + - name: aws-us-east-1 + replicas: 3 +``` + +- If `values-old.yaml` has no `gvc` key, the release ran on 1.5.0's defaults: `OLD_GVC_NAME` is `cockroach-gvc`, and `locations` is `aws-us-east-1`, `aws-eu-central-1`, `aws-us-west-2` in that order, 3 replicas each. +- With `backup.enabled: true`, `backup.location` must be one of `locations`. + +Every CockroachDB node and PgBouncer replica restarts, and the platform does not serialize the restart across locations, so expect retried or failed queries for a few minutes (see [Application Retry Logic](#application-retry-logic)). + +Afterwards the GVC belongs to you, not the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME`. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Upgrading from 1.x Template versions through 1.5.0 created their own GVC, so that GVC is part of a 1.x release's manifest. Version 2.0.0 does not declare a GVC — and Helm deletes what a chart stops declaring. -**Never `cpln helm upgrade` a 1.x release onto 2.0.0.** The upgrade would delete the GVC the 1.x release created and every workload, volume set and identity inside it — including your data — while reporting success. The only supported path is a **new** release against an existing GVC, a restore of the data, and removal of the old release. +**Never run a plain `cpln helm upgrade` on a 1.x release onto 2.0.0.** To keep the GVC, follow [Upgrading In Place from 1.5.0](#upgrading-in-place-from-1-5-0). The upgrade would otherwise delete the GVC the 1.x release created and every workload, volume set and identity inside it — including your data — while reporting success. The alternative is a **new** release against an existing GVC, a restore of the data, and removal of the old release. The chart refuses to render when your values still carry the 1.x `gvc` key, so an upgrade that passes your old values file stops before any API call is made: @@ -444,7 +476,7 @@ key was REMOVED. This chart no longer creates a GVC -- it deploys into the GVC y `gvc.locations` moved to the top-level `locations`. DO NOT `helm upgrade` a 1.x release onto 2.0.0 ... ``` -**The guard cannot cover an upgrade run with no values file at all.** A 1.x release installed on pure defaults has no `gvc` key for the chart to see, so nothing fires. Do not run an upgrade of a 1.x release against the 2.0.0 chart under any circumstances. +**The guard cannot cover an upgrade run with no values file at all.** A 1.x release installed on pure defaults has no `gvc` key for the chart to see, so nothing fires. Never run a plain upgrade of a 1.x release against the 2.0.0 chart. @@ -561,7 +593,7 @@ Values keys that changed in 2.0.0, for the new release's values file: ## Important Notes - **Every location in `locations` must already exist in the GVC.** A missing one is accepted at install and refused by the nodes at boot; a GVC location you did not list runs nothing. -- **Never `helm upgrade` a 1.x release onto 2.0.0.** It deletes the GVC the 1.x release created and everything in it. Install a new release and move the data — see [Upgrading from 1.x](#upgrading-from-1-x). +- **Never run a plain `helm upgrade` on a 1.x release onto 2.0.0.** It deletes the GVC the 1.x release created and everything in it. See [Upgrading In Place from 1.5.0](#upgrading-in-place-from-1-5-0), or install a new release and move the data: [Upgrading from 1.x](#upgrading-from-1-x). - **Surviving a region loss needs at least three locations.** The default single location survives a node failure, not a location failure. - **The cluster runs in insecure mode.** There are no SQL credentials; `internal_access`, `pgbouncer.internal_access` and the GVC boundary are the only access control. - **Confirm multi-zone support in every GVC location before setting `multiZone: true`.** An unsupported location wedges the deployment with no log output. diff --git a/template-catalog/templates/etcd-multi-location.mdx b/template-catalog/templates/etcd-multi-location.mdx index 1a06fb2e..c30333b9 100644 --- a/template-catalog/templates/etcd-multi-location.mdx +++ b/template-catalog/templates/etcd-multi-location.mdx @@ -314,10 +314,10 @@ If you must recover a keyspace, follow the [etcd disaster recovery guide](https: Versions before 2.0.0 created their own GVC; 2.0.0 deploys into one you already have. -**Do not `cpln helm upgrade` a 1.x release onto 2.0.0.** Once the chart stops declaring a GVC, Helm removes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it. The chart refuses to render when your values still carry the 1.x `global.gvc` key, so a values-carrying upgrade fails safely before any resource is touched; that guard cannot fire on an upgrade run with no values at all. The procedure below is the safety; the render guard is only a backstop. +**Do not run a plain `cpln helm upgrade` on a 1.x release.** [Upgrading in Place From 1.0.2](#upgrading-in-place-from-1-0-2) is the one supported in-place path. Once the chart stops declaring a GVC, Helm removes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it. The chart refuses to render when your values still carry the 1.x `global.gvc` key, so a values-carrying upgrade fails safely before any resource is touched; that guard cannot fire on an upgrade run with no values at all. Without the in-place steps, the safe alternative is the procedure below; the render guard is only a backstop. -Install 2.0.0 as a **new release against an existing GVC**, move the keyspace across, then remove the old release. +To keep the old GVC, follow [Upgrading in Place From 1.0.2](#upgrading-in-place-from-1-0-2). Otherwise install 2.0.0 as a **new release against an existing GVC**, move the keyspace across, then remove the old release. @@ -346,6 +346,36 @@ Install 2.0.0 as a **new release against an existing GVC**, move the keyspace ac +### Upgrading in Place From 1.0.2 + +A plain `cpln helm upgrade` from 1.x deletes the GVC the old release created, with everything in it. These steps keep the GVC, the cluster and its data, and apply to any 1.x release: + +```bash +cpln helm get values RELEASE_NAME > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/etcd-multi-location --version 2.0.0 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- **`--gvc` is the GVC the old release created** (`global.gvc.name` in `values-old.yaml`), not the GVC you ran the original install against. A different `--gvc` starts an empty install there instead of upgrading the one that holds your data. +- **The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`.** If it does not, stop: the GVC was not protected. + +Write `values-new.yaml` from `values-old.yaml`: + +```yaml +global: + locations: # was global.gvc.locations; copy the list exactly as it is + - name: aws-us-east-1 + - name: aws-us-west-2 + # delete global.gvc entirely; the chart refuses to render while it is present +``` + +Keep every other key as it is, and keep the location list unchanged: a different list reprovisions the cluster. All members restart together, so expect a short loss of quorum. + +Afterwards the GVC belongs to you rather than to the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME` when you no longer need it. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Scaling and Availability - **Fault tolerance follows the location count, not the template.** The cluster keeps serving while a majority of members is up: 3 locations tolerate the loss of 1, 5 tolerate 2, and 2 tolerate none (see [Locations and Quorum](#locations-and-quorum)). If you want automatic failover, use three locations. @@ -492,7 +522,7 @@ cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-etcd"} |= "[etcd]"' --limit 5 ## Important Notes -- **Do not `cpln helm upgrade` a 1.x release onto this version.** It deletes the GVC the old release created and everything inside it. Follow [Migrating From Version 1](#migrating-from-version-1). +- **Never run a plain `cpln helm upgrade` on a 1.x release.** It deletes the GVC the old release created and everything inside it. Follow [Upgrading in Place From 1.0.2](#upgrading-in-place-from-1-0-2) or [Migrating From Version 1](#migrating-from-version-1). - **`global.locations` must match the GVC's locations.** A declared location the GVC lacks makes a fresh member refuse to start with a named `[etcd] FATAL` line; a GVC location you did not list runs nothing. Read the GVC's list with `cpln gvc get GVC_NAME -o yaml` before installing. - **Quorum arithmetic decides your failure tolerance.** Two locations survive zero losses. If you want automatic failover, use three. - **Every `cpln helm upgrade` restarts the members together, including a no-op one.** Expect a brief write outage; the data survives. Plan upgrades as a short planned outage. diff --git a/template-catalog/templates/grafana-multi-location.mdx b/template-catalog/templates/grafana-multi-location.mdx index 079a1de9..c45a29b3 100644 --- a/template-catalog/templates/grafana-multi-location.mdx +++ b/template-catalog/templates/grafana-multi-location.mdx @@ -575,12 +575,12 @@ Two Grafana-specific rules apply to any restore: Every release before 2.0.0 created its own GVC. 2.0.0 creates none and deploys into the GVC you install it into. -**Never `helm upgrade` a 1.x release onto 2.0.0.** Once the chart stops declaring a GVC, Helm prunes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it: both Grafana tiers, the whole database cluster and its data volumes, the consensus store and the Redis coordination tier, while the command reports success. The chart refuses to render if your values still carry the 1.x `global.gvc` key, but that guard **cannot** fire on an upgrade run with no values at all. The procedure below is the safety; the guard is only a backstop. +**Never run a plain `helm upgrade` on a 1.x release onto 2.0.0.** To keep the GVC, follow [Upgrading In Place From 1.1.1](#upgrading-in-place-from-1-1-1) instead. Otherwise, once the chart stops declaring a GVC, Helm prunes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it: both Grafana tiers, the whole database cluster and its data volumes, the consensus store and the Redis coordination tier, while the command reports success. The chart refuses to render if your values still carry the 1.x `global.gvc` key, but that guard **cannot** fire on an upgrade run with no values at all. The in-place steps and the procedure below are the safety; the guard is only a backstop. The refusal message usually names `etcd-multi-location` or `postgres-multi-location` rather than this chart, because every chart in the stack carries the same guard on the same key and a subchart's copy aborts the render first. The remedy is identical. -Install 2.0.0 as a **new release against an existing GVC**, restore the database into it, then remove the old release: +To migrate instead of upgrading in place, install 2.0.0 as a **new release against an existing GVC**, restore the database into it, then remove the old release: @@ -609,6 +609,40 @@ Values that changed: | `postgresML` pinned at 1.x | Pinned at **2.0.0**, which creates no GVC of its own | | `redisML` pinned at 2.x | Pinned at **3.0.0**, which creates no GVC of its own and adds `redisML.engine` | +### Upgrading In Place From 1.1.1 + +Template versions through 1.1.1 created their own GVC, and 2.0.0 deploys into an existing one. A plain `cpln helm upgrade` across that change deletes the GVC the 1.x release created, with every workload, volume set and identity inside it. These steps mark that GVC "keep" first, so the upgrade leaves it and everything in it in place. It keeps the release's database (dashboards, users, datasources, alert rules), its etcd store and its Redis tier, and covers a 1.1.1 release; upgrade an older 1.x to 1.1.1 first. + +```bash +cpln helm get values RELEASE_NAME -o yaml > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (see below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/grafana-multi-location --version 2.0.0 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- `OLD_GVC_NAME` is the GVC the 1.x release created (its `global.gvc.name`), not the GVC you ran the original install against. +- The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`. If it does not, stop: the GVC was not protected. + +`values-new.yaml` is `values-old.yaml` with one change: + +```yaml +global: + # gvc: delete the whole block (name and locations) + locations: # the old global.gvc.locations: same entries, same order, same replicas + - name: aws-us-east-1 + replicas: 1 +``` + +- If `values-old.yaml` has no `global.gvc` key, the release ran on 1.1.1's defaults: `OLD_GVC_NAME` is `grafana-multi-location-gvc`, and `global.locations` is `aws-us-east-1`, `aws-eu-central-1`, `aws-us-west-2` in that order, 1 replica each. +- Leave `redisML.engine` unset (`redis`): the Redis data cannot move to `valkey` in place. +- Nothing else changes. The bundled database, etcd and Redis tiers move to their own converted versions with the same resource names and secrets. + +Every tier restarts, the database in every location included, so Grafana is unavailable for a few minutes. + +Afterwards the GVC belongs to you, not the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME`. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Alert Evaluation `alerting.enabled` is the parent switch. Setting it to `false` creates no evaluator workload and turns rule execution off on every instance: alert rules and contact points can still be created and viewed, and nothing ever evaluates them. While it is on, `alerting.highAvailability.enabled` picks one of two shapes: @@ -724,7 +758,7 @@ Turning HA on or off changes which workloads exist: enabling it deletes `RELEASE **Cause:** Your values file is from 1.x, which named a GVC under `global.gvc`. - **Fix:** Do not retry as an upgrade. Follow [Migrating From Version 1](#migrating-from-version-1). + **Fix:** Do not retry as a plain upgrade. Follow [Upgrading In Place From 1.1.1](#upgrading-in-place-from-1-1-1) or [Migrating From Version 1](#migrating-from-version-1). **Symptom:** The install refuses with `the database connection budget is exceeded — N Grafana instances ... above the 80-connection ceiling`. @@ -745,7 +779,7 @@ Turning HA on or off changes which workloads exist: enabling it deletes `RELEASE ## Important Notes - **Create the admin password, encryption key and database credentials secrets before installing.** Without them the deployment waits silently on secrets that do not exist while `helm install` reports success. -- **There is no upgrade path from 1.x.** A `helm upgrade` across the 2.0.0 boundary deletes the old GVC and everything in it. Follow [Migrating From Version 1](#migrating-from-version-1). +- **Never run a plain `helm upgrade` on a 1.x release onto 2.0.0.** It deletes the old GVC and everything in it. Follow [Upgrading In Place From 1.1.1](#upgrading-in-place-from-1-1-1) or [Migrating From Version 1](#migrating-from-version-1). - **Every location in `global.locations` must already exist in the GVC you install into**, and `alerting.location` and `postgresML.primaryLocation` must be among them. GVC locations you do not list run nothing. - **Never rotate or delete the encryption key.** Every stored datasource credential becomes unreadable in every location, and alert rules that query them fail. - **Treat every `helm upgrade` that touches the database tier as a planned outage**, and expect the first upgrade after any install to re-apply every tier. diff --git a/template-catalog/templates/mongodb-cluster.mdx b/template-catalog/templates/mongodb-cluster.mdx index 3cf0fd5c..dc88c615 100644 --- a/template-catalog/templates/mongodb-cluster.mdx +++ b/template-catalog/templates/mongodb-cluster.mdx @@ -470,7 +470,7 @@ The password in the secret creates the root user on first boot; afterwards the m ### Migrating from 1.x -Template versions through 1.1.1 created their own GVC. Version 2.0.0 does not — and Helm deletes what a chart stops declaring, so **an in-place upgrade of a 1.x release onto 2.0.0 deletes the GVC the 1.x release created, with every workload, volume set and identity in it, including your data.** The chart refuses to render when your values still carry the 1.x `gvc` key, so the usual path fails safely: +Template versions through 1.1.1 created their own GVC. Version 2.0.0 does not — and Helm deletes what a chart stops declaring, so **a plain upgrade of a 1.x release onto 2.0.0 deletes the GVC the 1.x release created, with every workload, volume set and identity in it, including your data.** The chart refuses to render when your values still carry the 1.x `gvc` key, so the usual path fails safely: ```text Error: execution error at (mongodb-cluster/templates/identity.yaml:1:4): mongodb-cluster 2.0.0: the @@ -480,7 +480,7 @@ release onto 2.0.0 ... ``` -**Do not run `cpln helm upgrade` on a 1.x release with the 2.0.0 chart under any circumstances**, including with no values at all — the guard can only see values you pass. Migrate to a new release instead. +**Never run a plain `cpln helm upgrade` on a 1.x release with the 2.0.0 chart**, including with no values at all — the guard can only see values you pass. To keep the GVC, follow [Upgrading In Place from 1.1.1](#upgrading-in-place-from-1-1-1); otherwise migrate to a new release as below. @@ -515,6 +515,38 @@ Values that moved or were removed in 2.0.0. Each is named at render time rather cpln secret reveal RELEASE_NAME-mongo-config -o yaml ``` +### Upgrading In Place from 1.1.1 + +Template versions through 1.1.1 created their own GVC, and 2.0.0 deploys into an existing one. A plain `cpln helm upgrade` across that change deletes the GVC the 1.x release created, with every workload, volume set and identity inside it. These steps mark that GVC "keep" first, so the upgrade leaves it and everything in it in place. It covers a 1.1.1 release (upgrade an older 1.x to 1.1.1 first) with **at most 7 members in total, never exactly 2, and at least 1 in every location**. The 1.x default of 3 locations × 3 is 9 members, which 2.0.0 refuses; for that shape, use [Migrating from 1.x](#migrating-from-1-x) instead. + +```bash +cpln helm get values RELEASE_NAME -o yaml > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (see below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/mongodb-cluster --version 2.0.0 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- `OLD_GVC_NAME` is the GVC the 1.x release created (its `gvc.name`), not the GVC you ran the original install against. +- The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`. If it does not, stop: the GVC was not protected. + +`values-new.yaml` is `values-old.yaml` with these changes: + +```yaml +# gvc: delete the whole block +locations: # the old gvc.locations: same entries, same order, same replicas + - name: aws-us-east-1 + replicas: 3 +backup: + mode: logical # 2.0.0 refuses physical + location: aws-us-east-1 # with backup.enabled: true, the location the job ran in (where your bucket is); must be one of locations +``` + +The credentials and keyfile secrets stay as they are. Every member and HAProxy replica restarts, so expect a primary election and failed writes for a few minutes. + +Afterwards the GVC belongs to you, not the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME`. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Scaling and Availability **What the cluster survives.** Only the primary accepts writes; when it is lost, the remaining members elect a new primary and the proxy re-routes to it, so clients using the proxy address keep the same connection string. With three members in one location the set survives losing one member; spread across three locations with one member each it survives losing a whole location. A member that comes back rejoins as a secondary on its own volume. Members are voting members, so size against the [table under Locations](#locations). @@ -579,9 +611,9 @@ Then upgrade the release with the lower count. To stop running in a location ent **Symptom:** `cpln helm install` or `cpln helm upgrade` exits with a `mongodb-cluster 2.0.0:` render error naming the `gvc` key or `backup.mode 'physical'`, and nothing is changed. - **Cause:** Your values file is from a 1.x release. The `gvc` block and physical backups were removed in 2.0.0, and an in-place upgrade from 1.x is refused deliberately. + **Cause:** Your values file is from a 1.x release. The `gvc` block and physical backups were removed in 2.0.0, and a plain upgrade from 1.x is refused deliberately. - **Fix:** Follow [Migrating from 1.x](#migrating-from-1-x). For a 2.0.0 values file, remove the `gvc` block and set `backup.mode: logical`. + **Fix:** Follow [Upgrading In Place from 1.1.1](#upgrading-in-place-from-1-1-1) or [Migrating from 1.x](#migrating-from-1-x). For a 2.0.0 values file, remove the `gvc` block and set `backup.mode: logical`. @@ -589,7 +621,7 @@ Then upgrade the release with the lower count. To stop running in a location ent - **Create both prerequisite secrets before installing.** A missing secret wedges the deployment with no log output; the only diagnostic is `status.versions[].message` from `cpln workload get-deployments`. - **Generate the keyfile with `openssl rand -base64 756`**, never a passphrase, and never change it after the cluster is initialized. -- **Never `helm upgrade` a 1.x release onto 2.0.0.** It deletes the GVC the 1.x chart created and everything in it. Migrate to a new release. +- **Never run a plain `helm upgrade` on a 1.x release onto 2.0.0.** It deletes the GVC the 1.x chart created and everything in it. See [Upgrading In Place from 1.1.1](#upgrading-in-place-from-1-1-1) or migrate to a new release. - **The GVC must already have every location you list**, and may have more; unlisted locations run nothing. Never reorder `locations` after install — only append. - **At most 7 members in total and 6 in one location.** A 2-member set is refused; 3 is the usual minimum. - **Connection strings need `authSource=admin`**, and clients should use the proxy address so failover needs no client change. diff --git a/template-catalog/templates/nats.mdx b/template-catalog/templates/nats.mdx index c044a67a..c7126c48 100644 --- a/template-catalog/templates/nats.mdx +++ b/template-catalog/templates/nats.mdx @@ -342,7 +342,7 @@ The stream backup command above follows the upstream guide but has not been exer Template versions through 2.0.0 created their own GVC; 3.0.0 deploys into one you already have. -**Do not `cpln helm upgrade` a 2.x release onto 3.0.0.** A 2.x release owns the GVC it created, and Helm deletes what a chart stops declaring — the upgrade destroys that GVC and **every workload, volume set and identity inside it**, including your JetStream data. The chart refuses to render when your values still carry the 2.x `gvc` key, so a values-carrying upgrade fails safely before any resource is touched: +**Do not run a plain `cpln helm upgrade` on a 2.x release.** [Upgrading in Place from 2.0.0](#upgrading-in-place-from-2-0-0) is the one supported in-place path. A 2.x release owns the GVC it created, and Helm deletes what a chart stops declaring — the upgrade destroys that GVC and **every workload, volume set and identity inside it**, including your JetStream data. The chart refuses to render when your values still carry the 2.x `gvc` key, so a values-carrying upgrade fails safely before any resource is touched: ```text Error: execution error at (nats/templates/workloads.yaml:3:4): nats 3.0.0: the `gvc` values key was @@ -350,10 +350,10 @@ REMOVED. This chart no longer creates a GVC -- it deploys into the GVC you insta `gvc.locations` moved to the top-level `locations`. DO NOT `helm upgrade` a 2.x release onto 3.0.0 ... ``` -That guard cannot fire on an upgrade run with **no values at all** — a 2.x release installed on pure defaults has no `gvc` key for the chart to see. The procedure below is the safety; the render guard is only a backstop. +That guard cannot fire on an upgrade run with **no values at all** — a 2.x release installed on pure defaults has no `gvc` key for the chart to see. Without the in-place steps, the safe alternative is the procedure below; the render guard is only a backstop. -Install 3.0.0 as a **new release against an existing GVC**, move the traffic, then remove the old release. +To keep the old GVC and its streams, follow [Upgrading in Place from 2.0.0](#upgrading-in-place-from-2-0-0). Otherwise install 3.0.0 as a **new release against an existing GVC**, move the traffic, then remove the old release. @@ -387,6 +387,46 @@ Values that moved or were removed in 3.0.0. The chart names each one at render t | `volumeset.autoscaling` | Gone. `volumeset.capacity` is the only storage knob | | `locations[].replicas: 0` | Not allowed. Remove the location instead | +### Upgrading in Place from 2.0.0 + +A plain `cpln helm upgrade` from 2.x deletes the GVC the old release created, with everything in it, including your JetStream data. These steps keep the GVC, the servers and their streams: + +```bash +cpln helm get values RELEASE_NAME > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/nats --version 3.1.0 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- **`--gvc` is the GVC the old release created** (`gvc.name` in `values-old.yaml`, `nats-gvc` if you never set it), not the GVC you ran the original install against. A different `--gvc` starts an empty install there instead of upgrading the one that holds your data. +- **The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`.** If it does not, stop: the GVC was not protected. + +Write `values-new.yaml` from `values-old.yaml`. `cpln helm get values` returns only what you set, so also write in any 2.x default you relied on: + +```yaml +locations: # was gvc.locations: same names and replicas (2.x default shown) + - name: aws-us-east-1 + replicas: 2 + - name: aws-us-west-2 + replicas: 2 + - name: aws-eu-central-1 + replicas: 2 +allowCIDR: # the 2.x default was 0.0.0.0/0; 3.x defaults to [] (no public access) + - 0.0.0.0/0 # keep only if clients use the public WebSocket endpoint +# delete: gvc, nats_defaults.cluster.listen, nats_defaults.gateway.listen, +# nats_defaults.cluster.noAdvertise (the chart refuses them), volumeset.autoscaling (no longer used) +``` + +- **With JetStream on, 3.x refuses a total of exactly 2 servers.** A 2.x release with one location and `replicas: 2` must change it to `replicas: 3`, which adds one server. +- Remove any location with `replicas: 0`; the chart refuses it. +- Keep `image`, `resources`, `jetstream`, `volumeset.capacity`, `internalAccess` and `nats_extra_config` as they are. + +The servers restart one at a time. + +Afterwards the GVC belongs to you rather than to the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME` when you no longer need it. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Rotating the Credentials Both turning authentication on and changing the password take effect only when the servers restart. @@ -494,7 +534,7 @@ cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-nats"} |= "[nats]"' --limit 5 ## Important Notes -- **Do not `cpln helm upgrade` a 2.x release onto this version.** It deletes the GVC the old release created and everything inside it. Follow [Migrating from 2.x](#migrating-from-2-x). +- **Never run a plain `cpln helm upgrade` on a 2.x release.** It deletes the GVC the old release created and everything inside it. Follow [Upgrading in Place from 2.0.0](#upgrading-in-place-from-2-0-0) or [Migrating from 2.x](#migrating-from-2-x). - **`locations` must match the GVC's locations.** A declared location the GVC lacks is not caught at install; a fresh JetStream server refuses to start on it, and every other server warns. Read the GVC's list with `cpln gvc get GVC_NAME -o yaml` before installing. - **A GVC location you did not list runs nothing.** Its deployment reads `This workload location is deactivated because maxScale is set to 0`; that is expected. - **No public access by default, and no authentication.** Opening `allowCIDR` without `auth.credentialsSecretName` publishes an unauthenticated message bus. Turn authentication on before adding ranges. Never pin a single `/32`. diff --git a/template-catalog/templates/pgedge.mdx b/template-catalog/templates/pgedge.mdx index 065d6458..a2060d8a 100644 --- a/template-catalog/templates/pgedge.mdx +++ b/template-catalog/templates/pgedge.mdx @@ -498,7 +498,7 @@ Each node's volume is also snapshotted on `volumeset.snapshots.schedule`. Restor Template versions through 1.1.1 created their own GVC, so that GVC is part of a 1.x release's manifest. Version 2.0.0 and later do not declare one — and Helm deletes what a chart stops declaring. -**An in-place `helm upgrade` from 1.x to any later version is not supported and destroys the cluster.** The upgrade would delete the GVC the 1.x chart created together with every workload, volume set and identity inside it — including your data. The only supported path is the migration below: install the current version as a new release into an existing GVC, move the data, then remove the old release. +**A plain `helm upgrade` from 1.x to any later version destroys the cluster.** The upgrade would delete the GVC the 1.x chart created together with every workload, volume set and identity inside it — including your data. The supported paths are [Upgrading in Place from 1.1.1](#upgrading-in-place-from-1-1-1), which keeps the GVC, or the migration below: install the current version as a new release into an existing GVC, move the data, then remove the old release. The chart refuses to render when your values still carry the 1.x `gvc` key, so an upgrade that passes your old values file stops before a single API call is made: @@ -506,7 +506,7 @@ The chart refuses to render when your values still carry the 1.x `gvc` key, so a Error: execution error at (pgedge/templates/identity.yaml:2:4): pgedge 2.0.0: the `gvc` values key was REMOVED. This chart no longer creates a GVC -- it deploys into the GVC you install into, and `gvc.locations` moved to the top-level `locations`. DO NOT `helm upgrade` a 1.x release onto 2.0.0: the upgrade drops `kind: gvc` from the manifest and Helm deletes what a chart no longer declares, which DESTROYS that GVC and every workload, volumeset and identity inside it. Install 2.0.0 as a NEW release against an existing GVC, move your data, then uninstall the old release. See `Migrating from 1.x` in the README. ``` -**The guard cannot cover an upgrade run with no values at all.** A 1.x release installed on pure defaults has no `gvc` key for the chart to see, so nothing fires. Do not run `helm upgrade` on a 1.x release against any later chart under any circumstances. +**The guard cannot cover an upgrade run with no values at all.** A 1.x release installed on pure defaults has no `gvc` key for the chart to see, so nothing fires. Do not run a plain `helm upgrade` on a 1.x release against any later chart. The 1.0.x credential values (`postgres.username`, `postgres.password`, `postgres.database`) are refused at render as well, with their own message, so pasting an old values file into a new install produces an error rather than a silent misconfiguration. @@ -557,6 +557,47 @@ Subscriptions move to `replicating` shortly after the reload. Check on every nod SELECT subscription_name, status FROM spock.sub_show_status(); ``` +### Upgrading in Place from 1.1.1 + +A plain `cpln helm upgrade` from 1.x deletes the GVC the old release created, with everything in it. These steps keep the GVC, the nodes and their data, and go straight from 1.x to 3.0.0 (no 2.x step needed): + +```bash +cpln helm get values RELEASE_NAME > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/pgedge --version 3.0.0 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- **`--gvc` is the GVC the old release created** (`gvc.name` in `values-old.yaml`, `pgedge-gvc` if you never set it), not the GVC you ran the original install against. A different `--gvc` starts an empty install there instead of upgrading the one that holds your data. +- **The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`.** If it does not, stop: the GVC was not protected. + +Write `values-new.yaml` from `values-old.yaml`. `cpln helm get values` returns only what you set, so write out the 1.x defaults you relied on too (the 1.1.1 default is shown): + +```yaml +locations: # was gvc.locations; same names and replicas + - name: aws-us-west-2 + replicas: 3 + - name: aws-us-east-2 + replicas: 3 + - name: aws-eu-central-1 + replicas: 3 +pgbouncer: # was pgcat; keep poolMode, defaultPoolSize, maxClientConn, resources, + poolMode: transaction # minReplicas, maxReplicas; delete pgcat.image +# delete: gvc, pgcat (the chart refuses them) +``` + +- If `values-old.yaml` still has `postgres.username`, `password` and `database` (1.0.x), create the credentials secret (see [Prerequisites](#prerequisites)) with exactly those values first, set `postgres.credentialsSecretName` to it and delete the three keys. If the nodes are still waiting on the new secret after the upgrade, run `cpln workload force-redeployment RELEASE_NAME-pgedge --gvc OLD_GVC_NAME`. +- Keep `image`, `resources`, `postgres.credentialsSecretName`, `multiZone`, `volumeset`, `internal_access` and `backup` as they are. Volume snapshots now run daily by default; set `volumeset.snapshots.schedule: ""` to keep them off. + +Every pgEdge node restarts once, one at a time within each location (with one node per location, writes pause briefly). The `-pgcat` workload is replaced by `-pgbouncer`, so **point applications at `RELEASE_NAME-pgbouncer.OLD_GVC_NAME.cpln.local:5432`** (port unchanged). Then: + +- Check replication: `SELECT subscription_name, status FROM spock.sub_show_status();` on each node must show every row `replicating`. If not, apply [Switching Replication On for an Existing 1.x Cluster](#switching-replication-on-for-an-existing-1-x-cluster). +- Run a backup: the backup format changed in 3.0.0 and the restore job reads only the new one (see [Backing Up](#backing-up)). + +Afterwards the GVC belongs to you rather than to the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME` when you no longer need it. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Upgrading from 2.x Version 3.0.0 replaces the pgcat pooler with PgBouncer and makes the HAProxy failover tier permanent. **The client endpoint changes** from `RELEASE_NAME-pgcat` to `RELEASE_NAME-pgbouncer`; the port stays `5432`. The pgEdge nodes, their volumes and their data are kept. @@ -733,7 +774,7 @@ Then open `http://localhost:8405/stats` in a browser. - **Your application must retry failed connections and statements** — without it, every upgrade and replica restart shows up as errors in your app. Retry with a short backoff for 10–15 seconds and make retried writes idempotent; see [Scaling and Availability](#scaling-and-availability). - **3.0.0 changes the client endpoint** to `RELEASE_NAME-pgbouncer`, and backups change format — take a backup right after upgrading. See [Upgrading from 2.x](#upgrading-from-2-x). -- **Never run `helm upgrade` on a 1.x release against any later version** — it deletes the GVC the 1.x chart created and everything in it. Migrate to a new release instead: [Upgrading from 1.x](#upgrading-from-1-x). +- **Never run a plain `helm upgrade` on a 1.x release against any later version** — it deletes the GVC the 1.x chart created and everything in it. Follow [Upgrading in Place from 1.1.1](#upgrading-in-place-from-1-1-1), or migrate to a new release: [Upgrading from 1.x](#upgrading-from-1-x). - **The GVC must already contain every location you list**, and may contain more. A missing location is not caught at install — the nodes exit with `[pgedge] FATAL: locations declared in values are not in GVC ...`. Locations you do not list start nothing. - **DDL does not replicate** — create every table on every node (or broadcast it and add it to the replication set on the others), and give every table a `PRIMARY KEY` or `CREATE TABLE` fails outright. - **Use at least 3 nodes per location in production**, and at least 2 for upgrades without a write interruption. diff --git a/template-catalog/templates/postgres-multi-location.mdx b/template-catalog/templates/postgres-multi-location.mdx index 9c502cba..11459a6f 100644 --- a/template-catalog/templates/postgres-multi-location.mdx +++ b/template-catalog/templates/postgres-multi-location.mdx @@ -688,10 +688,10 @@ Since every 1.x release also created its own GVC, a cluster on 1.0.0 or 1.0.1 sh Every release before 2.0.0 created its own GVC; 2.0.0 deploys into one you already have. -**Do not `cpln helm upgrade` a 1.x release onto 2.0.0.** Once the chart stops declaring a GVC, Helm removes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it, including the volumes holding your database. The chart refuses to render when your values still carry the 1.x `global.gvc` key, so a values-carrying upgrade fails safely before any resource is touched; that guard cannot fire on an upgrade run with no values at all. The procedure below is the safety; the render guard is only a backstop. +**Do not run a plain `cpln helm upgrade` on a 1.x release.** [Upgrading in Place From 1.1.0](#upgrading-in-place-from-1-1-0) is the one supported in-place path. Once the chart stops declaring a GVC, Helm removes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it, including the volumes holding your database. The chart refuses to render when your values still carry the 1.x `global.gvc` key, so a values-carrying upgrade fails safely before any resource is touched; that guard cannot fire on an upgrade run with no values at all. Without the in-place steps, the safe alternative is the procedure below; the render guard is only a backstop. -Install 2.0.0 as a **new release against an existing GVC**, move the data across, then remove the old release. +To keep the old GVC and its volumes, follow [Upgrading in Place From 1.1.0](#upgrading-in-place-from-1-1-0). Otherwise install 2.0.0 as a **new release against an existing GVC**, move the data across, then remove the old release. @@ -723,6 +723,41 @@ Values that changed: | `global.gvc.locations` | `global.locations` | | Bundled etcd Multi-Location 1.x | Pinned to **2.0.0**, which also creates no GVC | +### Upgrading in Place From 1.1.0 + +A plain `cpln helm upgrade` from 1.x deletes the GVC the old release created, with everything in it. These steps keep the GVC, the cluster and its data volumes: + +```bash +cpln helm get values RELEASE_NAME > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/postgres-multi-location --version 2.0.1 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- **`--gvc` is the GVC the old release created** (`global.gvc.name` in `values-old.yaml`, `postgres-multi-location-gvc` if you never set it), not the GVC you ran the original install against. A different `--gvc` starts an empty install there instead of upgrading the one that holds your data. +- **The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`.** If it does not, stop: the GVC was not protected. + +Write `values-new.yaml` from `values-old.yaml`. `cpln helm get values` returns only what you set, so write out the locations even if you used the 1.x default (shown): + +```yaml +global: + locations: # was global.gvc.locations; copy the list exactly as it is + - name: aws-us-east-1 + replicas: 1 + - name: aws-eu-central-1 + replicas: 1 + - name: aws-us-west-2 + replicas: 1 + # delete global.gvc entirely; the chart refuses to render while it is present +# image: delete it, or set controlplanecorporation/patroni-postgres:0.8 (the chart refuses 0.7 and earlier) +``` + +Keep every other key as it is, including `postgres.credentialsSecretName`, and keep the location list unchanged. All Postgres members and etcd members restart together, so **stop writes for the upgrade**: writes fail until the cluster is back, and writes acknowledged just before the restart can be lost (see [Important Notes](#important-notes)). + +Afterwards the GVC belongs to you rather than to the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME` when you no longer need it. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Upgrading From 2.0.0 Version 2.0.1 moves the default image to `controlplanecorporation/patroni-postgres:0.8`, which carries pgvector 0.8.7 and fixes CVE-2026-103484 (a database user who can create an IVFFlat index can run arbitrary code). It also changes how the HAProxy tier starts: a new proxy no longer waits for every member, and the old proxies keep serving until a new one is up. Nothing for you to do there. @@ -902,7 +937,7 @@ cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres"} |= "[patroni]"' -- ## Important Notes -- **Do not `cpln helm upgrade` a 1.x release onto 2.0.0.** It deletes the GVC the old release created and every volume set in it. Follow [Migrating From Version 1](#migrating-from-version-1): new release, restore, then uninstall the old one. +- **Never run a plain `cpln helm upgrade` on a 1.x release.** It deletes the GVC the old release created and every volume set in it. Follow [Upgrading in Place From 1.1.0](#upgrading-in-place-from-1-1-0), or [Migrating From Version 1](#migrating-from-version-1): new release, restore, then uninstall the old one. - **Create the credentials secret before installing.** A reference to a secret that does not exist wedges the workload in every location with no log output; see [Prerequisites](#prerequisites) for the one command that shows the reason. - **`global.locations` must match the GVC you install into.** A declared location the GVC lacks makes a fresh member refuse to start with a named `[patroni] FATAL` line; a GVC location you did not list runs nothing. Read the GVC's list with `cpln gvc get GVC_NAME -o yaml` before installing. - **Quorum arithmetic decides your failure tolerance.** Two locations survive zero losses and promotion is manual; three give automatic failover. diff --git a/template-catalog/templates/redis-multi-location.mdx b/template-catalog/templates/redis-multi-location.mdx index 37a33a05..09e2551d 100644 --- a/template-catalog/templates/redis-multi-location.mdx +++ b/template-catalog/templates/redis-multi-location.mdx @@ -446,10 +446,10 @@ What the backup is: a standard, gzipped RDB file. The upstream-documented way to ### Upgrading from Version 2 -Every release before 3.0.0 created its own GVC. **Do not `helm upgrade` a 2.x release onto 3.0.0.** Once the chart stops declaring a GVC, Helm deletes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it, while the command prints `upgraded successfully`. +Every release before 3.0.0 created its own GVC. **Do not run a plain `helm upgrade` on a 2.x release.** [Upgrading in Place from 2.2.0](#upgrading-in-place-from-2-2-0) is the one supported in-place path. Once the chart stops declaring a GVC, Helm deletes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it, while the command prints `upgraded successfully`. -The chart refuses to render when your values still carry the 2.x `global.gvc` key, so an upgrade that passes your old values file fails safely before any resource is touched, with a message saying the `global.gvc` values key was REMOVED. That guard **cannot** fire on an upgrade run with no values at all, because it then sees only 3.0.0's own defaults. The procedure below is the safety; the render guard is only a backstop. +The chart refuses to render when your values still carry the 2.x `global.gvc` key, so an upgrade that passes your old values file fails safely before any resource is touched, with a message saying the `global.gvc` values key was REMOVED. That guard **cannot** fire on an upgrade run with no values at all, because it then sees only 3.0.0's own defaults. Without the in-place steps, installing a new release and migrating is the safe alternative; the render guard is only a backstop. Install 3.0.0 as a **new release against an existing GVC**, copy the keyspace across, then remove the old release: @@ -481,6 +481,37 @@ Values that changed: Two behaviours also changed with the GVC: public access needs a dedicated load balancer you enable on your GVC yourself (2.x enabled it on the GVC it created — see [Public Access](#public-access)), and with `firewall.internalAllowType: workload-list` this release's own workloads are now added to the list for you, so remove them from `firewall.workloads` if you had listed them. +### Upgrading in Place from 2.2.0 + +A plain `cpln helm upgrade` from 2.x deletes the GVC the old release created, with everything in it. These steps keep the GVC, both tiers and their data, and every client hostname stays the same: + +```bash +cpln helm get values RELEASE_NAME > values-old.yaml +curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py +python3 keep-gvc.py RELEASE_NAME # shows what it will change +python3 keep-gvc.py RELEASE_NAME --apply +# write values-new.yaml from values-old.yaml (below) +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/redis-multi-location --version 3.0.0 --gvc OLD_GVC_NAME -f values-new.yaml +``` + +- **`--gvc` is the GVC the old release created** (`global.gvc.name` in `values-old.yaml`, `redis-multi-location-gvc` if you never set it), not the GVC you ran the original install against. A different `--gvc` starts an empty install there instead of upgrading the one that holds your data. +- **The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`.** If it does not, stop: the GVC was not protected. + +Write `values-new.yaml` from `values-old.yaml`. `cpln helm get values` returns only what you set, so write out the locations even if you used the 2.x default (shown): + +```yaml +global: + locations: # was global.gvc.locations; copy the list exactly as it is + - name: aws-us-east-1 + - name: aws-eu-central-1 + - name: aws-us-west-2 + # delete global.gvc entirely; the chart refuses to render while it is present +``` + +Keep every other key as it is, including `engine`, which cannot change on an existing install. Every Redis and Sentinel replica in every location restarts at once, so treat the upgrade as a planned write outage. + +Afterwards the GVC belongs to you rather than to the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME` when you no longer need it. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Scaling and Availability - **Shape.** The cluster has `redis.replicasPerLocation` Redis instances in each of your `global.locations` and one Sentinel per location — six Redis instances and three Sentinels by default. One Redis instance is the master; every other instance, in every location, is a replica of it. Write capacity is that of a single instance; replicas serve reads only. @@ -578,7 +609,7 @@ Two behaviours also changed with the GVC: public access needs a dedicated load b ## Important Notes -- **Never `helm upgrade` a 2.x release onto 3.0.0** — it deletes the GVC the old release created and everything in it. Install a new release and migrate; see [Upgrading from Version 2](#upgrading-from-version-2). +- **Never run a plain `helm upgrade` on a 2.x release** — it deletes the GVC the old release created and everything in it. Follow [Upgrading in Place from 2.2.0](#upgrading-in-place-from-2-2-0), or install a new release and migrate; see [Upgrading from Version 2](#upgrading-from-version-2). - **Every entry in `global.locations` must already be a location of the GVC**, and there must be at least two. A fresh install that names a missing location refuses to start and says which one; an already-running cluster warns and keeps serving. - **Choose `engine` before the first install.** Switching it on a running release crash-loops every instance until it is set back; migrate with a dump and restore instead. - **Create the password secrets before installing.** A `passwordSecretName` that points at a secret that does not exist wedges the deployment silently; the name appears only in `status.versions[].message`. diff --git a/template-catalog/templates/tidb.mdx b/template-catalog/templates/tidb.mdx index 91ffff23..f864c625 100644 --- a/template-catalog/templates/tidb.mdx +++ b/template-catalog/templates/tidb.mdx @@ -527,14 +527,14 @@ cpln workload cron start RELEASE_NAME-tidb-backup --gvc GVC_NAME --location LOCA Template versions through 1.8.1 created their own GVC, so that GVC is part of the 1.x release's manifest. Versions 2.0.0 and later do not declare it — and Helm deletes what a chart stops declaring. -**Never `helm upgrade` a 1.x release onto 2.x.** The upgrade deletes the GVC the 1.x chart created and every workload, volume set and identity inside it, including your data. The 2.x chart refuses to render when your values still carry a `gvc` key, so an upgrade run with your old values file fails before touching anything: +**Never run a plain `helm upgrade` on a 1.x release.** [Upgrading in Place from 1.8.1](#upgrading-in-place-from-1-8-1) is the one supported in-place path. A plain upgrade deletes the GVC the 1.x chart created and every workload, volume set and identity inside it, including your data. The 2.x chart refuses to render when your values still carry a `gvc` key, so an upgrade run with your old values file fails before touching anything: ```text Error: execution error at (tidb/templates/identity.yaml:1:4): tidb 2.0.0: the `gvc` values key was REMOVED. This chart no longer creates a GVC -- it deploys into the GVC you install into ... ``` -That guard cannot see an upgrade run with no values file at all — a 1.x release installed on pure defaults has no `gvc` key to catch. Do not run the upgrade under any circumstances. Install a new release and move the data across. +That guard cannot see an upgrade run with no values file at all — a 1.x release installed on pure defaults has no `gvc` key to catch. Do not run a plain upgrade. Either follow the in-place steps, or install a new release and move the data across, as below. @@ -572,6 +572,120 @@ Values keys that moved or were removed since 1.x: The guards fire on the mere presence of a removed key, so `exposeServer: false` and `devMode: false` fail too, each with an error naming the key and its replacement. +### Upgrading in Place from 1.8.1 + +A plain `cpln helm upgrade` from 1.x deletes the GVC the old release created, with everything in it. The steps below keep the GVC, the cluster and its data, from any 1.x release. Plan a maintenance window: PD and TiKV restart in every location at once, and reads and writes pause until they are back. Take a backup first (see [Backing Up](#backing-up)). + + + + ```bash + cpln helm get values RELEASE_NAME > values-old.yaml + ``` + + If `values-old.yaml` has `autoCreateDatabase.credentialsSecretName` (1.8.0 and later), that secret is already in place. If it has an `autoCreateDatabase.database` block instead (1.7.0 and earlier), create the secret now with the passwords the cluster uses today, which 1.x keeps in `RELEASE_NAME-tidb-user`: + + ```bash + cpln secret reveal RELEASE_NAME-tidb-user -o json + cpln secret create-dictionary --name RELEASE_NAME-tidb-credentials \ + --entry rootPassword='...' --entry user='...' --entry password='...' --entry db='...' + ``` + + + The upgrade replaces the policy that lets the cluster read its secrets, and a new grant takes several minutes to apply; this temporary grant covers the gap. Save as `prestage.yaml`, using your credentials secret's name: + + ```yaml + kind: policy + name: RELEASE_NAME-prestage-reveal + targetKind: secret + targetLinks: + - //secret/CREDENTIALS_SECRET_NAME + - //secret/RELEASE_NAME-tidb-pd-startup + - //secret/RELEASE_NAME-tidb-server-startup + - //secret/RELEASE_NAME-tidb-tikv-startup + - //secret/RELEASE_NAME-tidb-db-init + bindings: + - permissions: [reveal] + principalLinks: [//gvc/OLD_GVC_NAME/identity/RELEASE_NAME-tidb-identity] + --- + kind: policy + name: RELEASE_NAME-prestage-gvc + targetKind: gvc + targetLinks: [//gvc/OLD_GVC_NAME] + bindings: + - permissions: [view] + principalLinks: [//gvc/OLD_GVC_NAME/identity/RELEASE_NAME-tidb-identity] + ``` + + ```bash + cpln apply -f prestage.yaml + ``` + + + ```bash + curl -fsSLO https://raw.githubusercontent.com/controlplane-com/templates/main/scripts/keep-gvc.py + python3 keep-gvc.py RELEASE_NAME # shows what it will change + python3 keep-gvc.py RELEASE_NAME --apply + # write values-new.yaml from values-old.yaml (below) + cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/tidb --version 2.2.0 --gvc OLD_GVC_NAME -f values-new.yaml + ``` + + - **`--gvc` is the GVC the old release created** (`gvc.name` in `values-old.yaml`, `tidb-gvc` if you never set it), not the GVC you ran the original install against. A different `--gvc` starts an empty install there instead of upgrading the one that holds your data. + - **The upgrade output must include `Skipped deletion of /org/ORG_NAME/gvc/OLD_GVC_NAME`.** If it does not, stop: the GVC was not protected. + + Write `values-new.yaml` from `values-old.yaml`. `cpln helm get values` returns only what you set, so write out the 1.x defaults you relied on too (the 1.8.1 default was these three locations with 1 each): + + ```yaml + locations: # was gvc.locations; replicas: N becomes tikvReplicas: N + serverReplicas: N + - name: aws-us-east-2 + tikvReplicas: 1 + serverReplicas: 1 + - name: aws-eu-central-1 + tikvReplicas: 1 + serverReplicas: 1 + - name: aws-us-east-1 + tikvReplicas: 1 + serverReplicas: 1 + pdReplicas: 3 # was gvc.pdReplicas + autoCreateDatabase: + credentialsSecretName: RELEASE_NAME-tidb-credentials # your existing one, or the one from step 1 + internal_access: + server: + type: same-org # ONLY for 1.1.1 and earlier: they had one internal_access.type; move it here + images: + server: pingcap/tidb:v8.5.3 # ONLY if your servers run older than v8.5.7 (1.6.2 and earlier); see step 4 + # delete: gvc, devMode, exposeServer, autoCreateDatabase.database (the chart refuses them) + ``` + + Keep `resources`, `volumeset` capacities, `external_access`, `internal_access` and `backup` as they are. Volume snapshots now run daily by default; set `volumeset.tikv.snapshots.schedule` and `volumeset.pd.snapshots.schedule` to `""` to keep them off. + + Wait until every location of `RELEASE_NAME-pd`, `RELEASE_NAME-tikv` and `RELEASE_NAME-server` is ready (`cpln workload get-deployments RELEASE_NAME-pd --gvc OLD_GVC_NAME`). + + + Remove the `images` block from `values-new.yaml` and run the same `cpln helm upgrade` again. Only the SQL servers restart, new before old, and queries keep being served. The hold matters: upgrading the servers while PD and TiKV are restarting can leave the new servers stuck on TiDB's system-table upgrade. PD and TiKV cannot be downgraded after step 3. + + + A cluster first installed before 1.7.0 never told PD to spread copies by location. Check, and set it if `location-labels` is empty: + + ```bash + cpln workload exec RELEASE_NAME-pd --gvc OLD_GVC_NAME --location LOCATION_NAME -- \ + sh -c 'curl -s 127.0.0.1:2379/pd/api/v1/config/replicate' + cpln workload exec RELEASE_NAME-pd --gvc OLD_GVC_NAME --location LOCATION_NAME -- \ + sh -c 'curl -s -X POST -H "Content-Type: application/json" -d "{\"location-labels\":\"region\"}" 127.0.0.1:2379/pd/api/v1/config/replicate' + ``` + + PD then moves replicas in the background with no downtime. + + + ```bash + cpln policy delete RELEASE_NAME-prestage-reveal RELEASE_NAME-prestage-gvc + ``` + + Never `helm rollback` this release to a 1.x revision. + + + +Afterwards the GVC belongs to you rather than to the release: `cpln helm uninstall` leaves it in place, and you delete it with `cpln gvc delete OLD_GVC_NAME` when you no longer need it. Details: [the keep-gvc script documentation](https://github.com/controlplane-com/templates/blob/main/scripts/README.md). + ### Upgrading from 2.0.0 2.0.0 and 2.2.0 both deploy into an existing GVC, so this is a plain `cpln helm upgrade` with one values change. Everything in [Upgrading from 2.1.0](#upgrading-from-2-1-0) applies to this upgrade too. @@ -848,7 +962,7 @@ If the init job is still deployed, set `autoCreateDatabase.deployInitWorkload: f **Cause:** your values file still carries a key that 2.0.0 removed. The guards fire on the key's presence, even when it is set to `false`. - **Fix:** delete the key and use its replacement from the list in [Upgrading from 1.x](#upgrading-from-1-x). If this was an attempt to upgrade a 1.x release in place, stop — that path destroys the 1.x GVC. Install a new release instead. + **Fix:** delete the key and use its replacement from the list in [Upgrading from 1.x](#upgrading-from-1-x). If this was a plain `helm upgrade` of a 1.x release, stop — that path destroys the 1.x GVC. Follow [Upgrading in Place from 1.8.1](#upgrading-in-place-from-1-8-1) or install a new release instead. **Symptom:** rendering fails with `tidb: backup.location 'aws-us-west-2' is not one of locations (aws-us-east-1)`. @@ -886,7 +1000,7 @@ If the init job is still deployed, set `autoCreateDatabase.deployInitWorkload: f - **Root has a password from the start only when `autoCreateDatabase.enabled` is `true`** (the default). With it off, root starts with no password — set one with `ALTER USER` before clients can reach the cluster. - **Each TiDB server start makes one root login attempt with an empty password**, to confirm root is protected before serving. If you set `FAILED_LOGIN_ATTEMPTS` on root, set it above your total number of TiDB servers, or restarting them all locks root. - **Upgrading from 2.1.0 restarts every workload once** — on the one-PD-and-TiKV-per-location layout, writes pause for about 75 seconds. Check first that root refuses an empty password; see [Upgrading from 2.1.0](#upgrading-from-2-1-0). -- **Never `helm upgrade` a 1.x release onto 2.x** — it deletes the GVC the 1.x chart created and everything in it. Install a new release: [Upgrading from 1.x](#upgrading-from-1-x). +- **Never run a plain `helm upgrade` on a 1.x release** — it deletes the GVC the 1.x chart created and everything in it. Follow [Upgrading in Place from 1.8.1](#upgrading-in-place-from-1-8-1), or install a new release: [Upgrading from 1.x](#upgrading-from-1-x). - **Create the credentials secret before installing.** Without it the deployment wedges silently; the only diagnostic is `status.versions[].message` from `cpln workload get-deployments`. - **Upgrading from 2.0.0 needs one values change**: replace each location's `replicas` with `tikvReplicas` and `serverReplicas`. See [Upgrading from 2.0.0](#upgrading-from-2-0-0). - **The GVC must contain every location you list**, and may contain more. A missing location is not caught at install: PD exits with `FATAL: locations declared in values are not in GVC ...`.