diff --git a/template-catalog/templates/airflow.mdx b/template-catalog/templates/airflow.mdx index 745c2df5..b7605a49 100644 --- a/template-catalog/templates/airflow.mdx +++ b/template-catalog/templates/airflow.mdx @@ -26,16 +26,15 @@ The admin password, the JWT signing secret and the Fernet key are not template v | Volume set | `RELEASE_NAME-airflow-vs` | Shared Airflow home (`/opt/airflow`: DAGs and task logs) mounted by the webserver and every worker | — | | Volume set | `RELEASE_NAME-airflow-postgres-vs` | `ext4` volume holding the metadata database | — | | Volume set | `RELEASE_NAME-airflow-redis-vs` | `ext4` volume holding the broker's data | — | -| Secret | `RELEASE_NAME-airflow-config` | Dictionary secret with the metadata database username and password | — | | Secret | `RELEASE_NAME-airflow-webserver-startup` | Opaque secret holding the webserver's startup script | — | | Identity | `RELEASE_NAME-airflow-identity` | Identity shared by the webserver, workers and database | — | -| Policy | `RELEASE_NAME-airflow-policy` | `reveal` on the config secret, the startup script, your auth secret and, when set, your git token secret | — | +| Policy | `RELEASE_NAME-airflow-policy` | `reveal` on your database credentials secret, the startup script, your auth secret and, when set, your git token secret | — | | Policy | `RELEASE_NAME-airflow-gvc-policy` | `view` on the one GVC you install into, so the webserver can check its location and KEDA settings at boot | — | | Container | `git-sync` (sidecar on the webserver and worker workloads) | Clones a Git repository onto the shared volume and keeps it in sync | `gitSync.enabled: true` | ## Prerequisites -One secret must exist before you install, and the GVC must already have the location you configure. Secrets are org-level, so no GVC flag is involved. +Two secrets must exist before you install, and the GVC must already have the location you configure. Secrets are org-level, so no GVC flag is involved. @@ -72,6 +71,18 @@ One secret must exist before you install, and the GVC must already have the loca cpln secret reveal AIRFLOW_AUTH_SECRET_NAME -o yaml ``` + + A [dictionary secret](/guides/create-secret/dictionary) holding `username`, `password` and `database`. The bundled PostgreSQL initializes from it, and the webserver and workers connect with it: + + ```bash + cpln secret create-dictionary --name my-airflow-db-credentials \ + --entry username=airflow \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=airflow + ``` + + Set its name in `postgres.config.credentialsSecretName`. Secret names are org-wide, so give each Airflow release its own secret. The three values are part of Airflow's connection URLs, so use letters, digits, `-` and `_` only. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password). + Only when `gitSync.enabled` is `true` and the repository is private. An [opaque secret](/guides/create-secret/opaque) with encoding `plain` whose payload is a personal access token; the sidecar sends it as the Git password: @@ -101,7 +112,7 @@ One secret must exist before you install, and the GVC must already have the loca -**Create the auth secret before installing, or the deployment wedges silently.** A `secretName` pointing at a secret that does not exist installs "successfully" and then never starts. No container runs, so `cpln logs` returns **zero lines**. The one place the reason appears is `status.versions[].message`: +**Create the auth and database credentials secrets before installing, or the deployment wedges silently.** A `secretName` pointing at a secret that does not exist installs "successfully" and then never starts. No container runs, so `cpln logs` returns **zero lines**. The one place the reason appears is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-airflow-webserver --gvc GVC_NAME -o yaml @@ -124,13 +135,14 @@ cpln workload force-redeployment RELEASE_NAME-airflow-webserver --gvc GVC_NAME ## Installation -Install from the marketplace registry, naming the location your GVC has and the auth secret you created: +Install from the marketplace registry, naming the location your GVC has and the two secrets you created: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/airflow \ - --version 2.0.0 --gvc GVC_NAME \ + --version 2.1.0 --gvc GVC_NAME \ --set location=aws-us-east-1 \ - --set airflow.auth.secretName=AIRFLOW_AUTH_SECRET_NAME + --set airflow.auth.secretName=AIRFLOW_AUTH_SECRET_NAME \ + --set postgres.config.credentialsSecretName=my-airflow-db-credentials ``` @@ -176,14 +188,18 @@ postgres: minMemory: 512Mi maxMemory: 1024Mi config: - username: username - password: change-me-airflow-db # set at FIRST install; cannot be changed once the volume is initialized - database: airflow + # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL. If it does not + # exist the deployment WEDGES silently and `cpln logs` returns nothing at all. + # A `dictionary` secret holding exactly three keys: username, password and + # database. Postgres reads them only when its data directory is first + # initialized. Upgrading from 2.0.0: give it a NEW name (not + # RELEASE-airflow-config) — see `Upgrading from 2.0.0` in the README. + credentialsSecretName: my-airflow-db-credentials volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) ``` -The database serves Airflow only and is unreachable from outside the GVC, so its credentials stay values and the chart writes them into `RELEASE_NAME-airflow-config`. **Set `postgres.config.password` at first install:** the shipped placeholder is published, and PostgreSQL reads the password only when the data directory is initialized, so it cannot be changed through values afterwards. +The metadata database credentials come from your prerequisite dictionary secret (see [Prerequisites](#prerequisites)), named by `postgres.config.credentialsSecretName`. PostgreSQL reads them only when the data directory is first initialized, so editing the secret afterwards does not change the database password — see [Rotating the Database Password](#rotating-the-database-password). ### Redis Broker @@ -305,7 +321,7 @@ KEDA scales the Celery workers on the Redis queue length, including down to zero |---|---|---| | UI and REST API (closed default) | `cpln port-forward RELEASE_NAME-airflow-webserver 8080:8080 --gvc GVC_NAME`, then `http://localhost:8080` | The tunnel works while the canonical endpoint returns `403` | | UI and REST API (public) | `status.canonicalEndpoint` from `cpln workload get RELEASE_NAME-airflow-webserver --gvc GVC_NAME -o yaml` | Returns `403` until you add a CIDR to `firewallConfig.inboundAllowCIDR` | -| Metadata database (in-GVC) | `RELEASE_NAME-airflow-postgres.GVC_NAME.cpln.local:5432` | Credentials are `postgres.config.*` | +| Metadata database (in-GVC) | `RELEASE_NAME-airflow-postgres.GVC_NAME.cpln.local:5432` | Credentials are the `username`, `password` and `database` keys of the secret named by `postgres.config.credentialsSecretName` | | Redis broker (in-GVC) | `RELEASE_NAME-airflow-redis.GVC_NAME.cpln.local:6379` | No authentication | | Login | `airflow.admin.username` and the `adminPassword` key of your auth secret | `cpln secret reveal AIRFLOW_AUTH_SECRET_NAME -o yaml` | @@ -431,6 +447,26 @@ cpln workload force-redeployment RELEASE_NAME-airflow-webserver --gvc GVC_NAME `jwtSecret` can be rotated the same way — redeploy both the webserver and `RELEASE_NAME-airflow-celery-worker`, and every outstanding API token becomes invalid. **Never change `fernetKey`.** +### Rotating the Database Password + +PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with, so a rotation takes three steps in order: + +1. Change it in the database (`DB_USERNAME` and `DB_NAME` are the secret's `username` and `database`): + + ```bash + cpln workload exec RELEASE_NAME-airflow-postgres --gvc GVC_NAME --container postgresql -- psql -U DB_USERNAME -d DB_NAME -c "ALTER ROLE \"DB_USERNAME\" PASSWORD 'NEW_PASSWORD'" + ``` + +2. Update the `password` key of the secret: `cpln secret edit SECRET_NAME`. +3. Right away, redeploy the webserver and the workers, and wait until `get-deployments` shows a higher version number for each, alone and ready. Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. + + ```bash + cpln workload force-redeployment RELEASE_NAME-airflow-webserver --gvc GVC_NAME + cpln workload force-redeployment RELEASE_NAME-airflow-celery-worker --gvc GVC_NAME + ``` + +Connections Airflow already has open keep working after `ALTER ROLE`; new ones fail until the redeploy finishes, so rotate in a quiet window. Without step 3 the old password stays in use indefinitely: updating a secret does not restart the workloads that read it. + ### Upgrading from 1.4.x or Earlier Template version 1.5.0 removed the credentials that 1.4.x took as plain Helm values. Each removed key is rejected at render with a message naming its replacement, so an old values file fails loudly instead of being silently ignored. There is no in-place path from 1.4.x to 2.0.0 — follow [Migrating from 1.x](#migrating-from-1-x), carrying only the new keys: @@ -471,7 +507,7 @@ An upgrade run with no values at all has no `gvc` key to catch and proceeds, so Choose a GVC that already has the location you will set as `location`, and enable KEDA on it first if you plan to use it. See [Prerequisites](#prerequisites). - Use a **different release name** — policy and secret names are org-wide, so a same-named release collides with the 1.x one even in another GVC. Point `airflow.auth.secretName` at the **same auth secret** the 1.x release used: the `fernetKey` must be identical, or every restored Connection and Variable becomes unreadable. + Use a **different release name** — policy and secret names are org-wide, so a same-named release collides with the 1.x one even in another GVC. Point `airflow.auth.secretName` at the **same auth secret** the 1.x release used: the `fernetKey` must be identical, or every restored Connection and Variable becomes unreadable. With 2.1.0 or later, also point `postgres.config.credentialsSecretName` at a new database credentials secret (see [Prerequisites](#prerequisites)). Dump from the old release's database workload, in the GVC the 1.x chart created, and restore into the new one. Both go through base64 because `cpln workload exec` is not binary-safe: @@ -484,10 +520,10 @@ An upgrade run with no values at all has no `gvc` key to catch and proceeds, so ```bash base64 < airflow.dump | cpln workload exec RELEASE_NAME-airflow-postgres --gvc GVC_NAME \ --container postgresql --stdin --quiet -- \ - sh -c 'base64 -d | pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists' + sh -c 'base64 -d | pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists --no-owner --no-acl' ``` - This data move has not been exercised end to end; see [Restoring a Backup](#restoring-a-backup). + `--no-owner --no-acl` let the restore succeed when the new database credentials secret uses a different `username` from the old release. This data move has not been exercised end to end; see [Restoring a Backup](#restoring-a-backup). With `gitSync.enabled: true` there is nothing to do — the sidecar clones onto the new volume. Otherwise copy them off the old shared volume: @@ -517,6 +553,22 @@ Values that moved or were removed in 2.0.0: | — | `airflow.celeryWorker.replicas`, the fixed worker count used when KEDA is off | | — | `internalAccess.type` / `internalAccess.workloads` | +### Upgrading from 2.0.0 + +Template version 2.1.0 removed `postgres.config.username`, `postgres.config.password` and `postgres.config.database`: the metadata database credentials are now a prerequisite secret. Carrying the old keys forward fails the render. + +2.0.0 created that secret itself (`RELEASE_NAME-airflow-config`), so **this upgrade deletes it**, and the chart refuses that name for the new secret. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Airflow loses its database connection: + +```bash +cpln secret reveal RELEASE_NAME-airflow-config -o yaml # username and password the database runs with today +cpln secret create-dictionary --name RELEASE_NAME-airflow-db \ + --entry username='YOUR_CURRENT_USERNAME' --entry password='YOUR_CURRENT_PASSWORD' --entry database='YOUR_CURRENT_DATABASE' +``` + +`YOUR_CURRENT_DATABASE` is `postgres.config.database` of the 2.0.0 release — `airflow` unless you changed it; the old secret does not hold it. Then upgrade with `postgres.config.credentialsSecretName` set to the new name and the three old keys removed. An upgrade that passes no values at all uses the default name `my-airflow-db-credentials`, so create the secret under that name instead if you upgrade that way. + +The upgrade restarts PostgreSQL, the webserver and the workers (they now read the new secret), so Airflow is briefly unavailable. The new version of a workload can also wait for several minutes, with `get-deployments` reporting that it is not allowed to reveal the secret, while access to the new secret propagates; the previous version keeps running meanwhile, and it clears by itself (or wait a few minutes, then `cpln workload force-redeployment` the waiting workload). + ### Scaling and Availability - **Workers scale; the webserver does not.** Raise `airflow.celeryWorker.replicas` for a fixed pool, or enable KEDA to scale on queue length (see [KEDA Autoscaling](#keda-autoscaling)). The webserver, database and broker are each a single replica on a single volume. @@ -586,23 +638,30 @@ Values that moved or were removed in 2.0.0: **Fix:** use the port-forward from [First Run](#first-run), or add your CIDR ranges and allow a few minutes for the change to apply. - - **Symptom:** after an upgrade that changed `postgres.config.password`, the webserver and workers fail to authenticate against PostgreSQL. + + **Symptom:** after editing the `password` key of the database credentials secret and redeploying, the webserver and workers fail to authenticate against PostgreSQL. **Cause:** PostgreSQL reads `POSTGRES_PASSWORD` only when the data directory is initialized, so the database still has the old password while Airflow's connection string carries the new one. - **Fix:** revert the value to the password the volume was initialized with. The password can only be set at first install. + **Fix:** change the password in the database first, as described in [Rotating the Database Password](#rotating-the-database-password), or put the old value back in the secret and redeploy the webserver and workers. + + + **Symptom:** upgrading a 2.0.0 release fails at render with `airflow 2.1.0: postgres.config.username, postgres.config.password and postgres.config.database were REMOVED`. + + **Cause:** your values still carry the 2.0.0 plaintext database credentials. + + **Fix:** create the database credentials secret holding your current values and remove the old keys — see [Upgrading from 2.0.0](#upgrading-from-2-0-0). ## Important Notes -- **Create the auth secret before installing.** Without it the deployment wedges with no log output; `cpln workload get-deployments` is the one place the missing secret is named. +- **Create the auth and database credentials secrets before installing.** Without either the deployment wedges with no log output; `cpln workload get-deployments` is the one place the missing secret is named. - **The Fernet key can never be rotated.** Generate it once, keep it, and back it up with the auth secret. - **Never upgrade a 1.x release onto 2.0.0 in place** — it deletes the GVC the 1.x chart created and everything in it. See [Migrating from 1.x](#migrating-from-1-x). - **`location` must be a location of your GVC.** Otherwise the install succeeds and nothing starts anywhere. - **Enable KEDA on the GVC before setting `keda.enabled: true`**, or the workers sit at `keda.minScale` forever with no error. -- **Set `postgres.config.password` at first install.** The shipped placeholder is published and the value cannot be changed once the data directory exists. +- **Rotating the database password needs `ALTER ROLE` plus a forced redeployment of the webserver and workers** — editing the secret alone changes nothing; see [Rotating the Database Password](#rotating-the-database-password). - **The UI is closed by default.** Reach it with `cpln port-forward`; a firewall change takes a few minutes to apply. - **The first Helm upgrade after an install can restart the database and webserver** even with unchanged values; expect a brief interruption. - **Changing the admin password means editing the secret and forcing a webserver redeployment**; see [Rotating the Admin Password](#rotating-the-admin-password). diff --git a/template-catalog/templates/cdc-pipeline.mdx b/template-catalog/templates/cdc-pipeline.mdx index dfcbac10..fb1a74bc 100644 --- a/template-catalog/templates/cdc-pipeline.mdx +++ b/template-catalog/templates/cdc-pipeline.mdx @@ -1,13 +1,13 @@ --- title: CDC Pipeline icon: /template-catalog/templates/icons/cdc-pipeline.png -description: Change data capture in one install. A three-member PostgreSQL Highly Available source, a three-broker Kafka cluster and a Debezium Server connector, wired together with shared credentials and internal hostnames. Deploys into an existing GVC; needs a Kafbat UI configuration secret first. +description: Change data capture in one install. A three-member PostgreSQL Highly Available source, a three-broker Kafka cluster and a Debezium Server connector, wired together with shared credentials and internal hostnames. Deploys into an existing GVC; needs one credentials secret first. keywords: ['cdc', 'change data capture', 'debezium', 'kafka', 'postgresql', 'postgres to kafka', 'event streaming', 'data pipeline', 'patroni', 'kraft', 'logical replication', 'outbox', 'kafka connect alternative', 'fivetran alternative'] --- ## Overview -CDC Pipeline deploys a complete change data capture stack in a single release. It reimplements nothing: it bundles three catalog templates as dependencies — [PostgreSQL Highly Available](/template-catalog/templates/postgres-highly-available) as the source database, [Kafka](/template-catalog/templates/kafka) as the event transport, and [Debezium Server](/template-catalog/templates/debezium-server) as the connector that reads the database's change stream and publishes every row-level insert, update and delete to Kafka. This chart adds only the glue: one credentials secret for the database, the internal hostnames each component uses to reach the others, and checks at render time that the shared credentials agree. +CDC Pipeline deploys a complete change data capture stack in a single release. It reimplements nothing: it bundles three catalog templates as dependencies — [PostgreSQL Highly Available](/template-catalog/templates/postgres-highly-available) as the source database, [Kafka](/template-catalog/templates/kafka) as the event transport, and [Debezium Server](/template-catalog/templates/debezium-server) as the connector that reads the database's change stream and publishes every row-level insert, update and delete to Kafka. This chart adds only the glue: a policy letting the Kafka brokers read the one credentials secret you create, the internal hostnames each component uses to reach the others, and checks at render time that the components' values agree. This template deploys into an existing [GVC](/reference/gvc) that you already have. It does not create or manage a GVC. Every workload runs in each location the GVC has, so use a GVC with a single location. @@ -25,26 +25,56 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-postgres-ha-proxy` | HAProxy routing TCP `5432` to the current leader (standard, 2 replicas); container `haproxy` | — | | Workload | `RELEASE_NAME-etcd` | Consensus store for PostgreSQL leader election (stateful, 3 replicas); container `etcd` | — | | Workload | `RELEASE_NAME-pgbouncer` | PgBouncer connection pooler in front of HAProxy | `postgres-highly-available.pgbouncer.enabled: true` | -| Workload | `RELEASE_NAME-cluster` | Kafka brokers in KRaft mode (stateful, 3 replicas, SASL client listener on `9092`); containers `kafka`, `kafka-orchestrator`, `kafka-exporter`, `jmx-exporter` | — | -| Workload | `RELEASE_NAME-kafbat-ui` | Kafbat UI web console for Kafka, public by default | `kafka.kafbat_ui.enabled: true` (default) | +| Workload | `RELEASE_NAME-cluster` | Kafka brokers in KRaft mode (stateful, 3 replicas, SASL client listener on `9092`); containers `kafka`, `jmx-exporter` | — | +| Workload | `RELEASE_NAME-kafbat-ui` | Kafbat UI web console for Kafka, public when on | `kafka.kafbat_ui.enabled: true` (off by default) | | Workload | `RELEASE_NAME-client` | Kafka CLI tools container for testing from inside the GVC; container `kafka` | — | | Workload | `RELEASE_NAME-debezium` | Debezium Server connector (stateful, 1 replica, health on `8080`); container `debezium-server` | — | | Volume Set | `RELEASE_NAME-postgres-ha-vs` | One volume per PostgreSQL member holding the data directory | — | | Volume Set | `RELEASE_NAME-etcd-vs` | One volume per etcd member | — | | Volume Set | `RELEASE_NAME-logs-0`, `RELEASE_NAME-logs-1` | Kafka log directories, with daily snapshots | — | | Volume Set | `RELEASE_NAME-debezium-data` | Debezium offsets and schema history | — | -| Secret | `my-cdc-pipeline-db-credentials` | Dictionary with the database `username`, `password` and `database`, created by this chart from `database.*`. Its name is `postgres-highly-available.config.credentialsSecretName` and is **not** prefixed with the release name | — | -| Secret | `RELEASE_NAME-secrets` | KRaft cluster ID and the Kafka controller, inter-broker and admin passwords | — | +| Secret | `RELEASE_NAME-secrets` | KRaft cluster ID, and `${env:...}` references to the Kafka passwords in your credentials secret | — | | Secret | `RELEASE_NAME-controller-configuration`, `RELEASE_NAME-init`, `RELEASE_NAME-jmx-exporter-conf` | Kafka broker configuration, startup script and JMX exporter rules | — | -| Secret | `RELEASE_NAME-debezium-config`, `RELEASE_NAME-debezium-credentials`, `RELEASE_NAME-debezium-entrypoint` | Debezium `application.properties`, the connection details and passwords it reads, and its startup script | — | +| Secret | `RELEASE_NAME-debezium-config`, `RELEASE_NAME-debezium-credentials`, `RELEASE_NAME-debezium-entrypoint` | Debezium `application.properties`, the connection details it reads (its passwords come from your credentials secret), and its startup script | — | | Secret | `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-patroni-startup`, `RELEASE_NAME-etcd-startup`, `RELEASE_NAME-wal-g-backup-script` | PostgreSQL, HAProxy and etcd startup scripts | — | | Identity, Policy | `RELEASE_NAME-postgres-ha-identity` / `RELEASE_NAME-postgres-ha-policy`, `RELEASE_NAME-etcd-identity` / `RELEASE_NAME-etcd-policy`, `RELEASE_NAME` / `RELEASE_NAME`, `RELEASE_NAME-kafbat-ui` / `RELEASE_NAME-kafbat-ui`, `RELEASE_NAME-debezium-identity` / `RELEASE_NAME-debezium-policy` | One identity per component, each granted `reveal` on exactly the secrets that component mounts | — | +| Policy | `RELEASE_NAME-cdc-kafka-credentials` | Lets the Kafka brokers `reveal` your credentials secret | — | ## Prerequisites - - Kafbat UI is on by default and reads its whole configuration from an **opaque** secret mounted as a file, which must exist **before** you install. Replace `ADMIN_PASSWORD` with the value you will pass as `kafka.kafka.listeners.client.sasl.admin.password` and `UI_PASSWORD` with the login you want for the console. The full file format is in the [Kafbat configuration reference](https://ui.docs.kafbat.io/configuration/configuration-file). + + **One [dictionary secret](/guides/create-secret/dictionary) must exist before you install.** PostgreSQL initializes with it, the Kafka brokers take their SASL passwords from it, and Debezium connects with it, so it holds eight keys. `my-cdc-pipeline-credentials` is the name the values default to: + + ```bash + DB_PASSWORD="$(openssl rand -hex 24)" + cpln secret create-dictionary --name my-cdc-pipeline-credentials \ + --entry username=cdc_user \ + --entry password="$DB_PASSWORD" \ + --entry sourcePassword="$DB_PASSWORD" \ + --entry database=cdcdb \ + --entry kafkaSaslPassword="$(openssl rand -hex 24)" \ + --entry kafkaAdminPassword="$(openssl rand -hex 24)" \ + --entry kafkaInterBrokerPassword="$(openssl rand -hex 24)" \ + --entry kafkaControllerPassword="$(openssl rand -hex 24)" + ``` + + - `sourcePassword` must equal `password` (Debezium reads that key), and `username`/`database` must equal `debezium-server.source.database.user`/`.name`. `database` is used unquoted, so keep it to lowercase letters, digits and underscores. + - `kafkaSaslPassword` is the password of Kafka's `debezium` user, on the broker side and in Debezium alike. `kafkaAdminPassword` is the Kafka `admin` user's; the other two secure the brokers' own connections. Keep the Kafka passwords to letters and digits, as generated above: they are placed inside Kafka's login configuration, where a `"` or `;` breaks it. + - The secret's name appears in six values: `postgres-highly-available.config.credentialsSecretName`, `debezium-server.credentialsSecretName` and the four `kafka.kafka.env` entries (see [Database Credentials](#database-credentials)). **All six must name the same secret**; the chart refuses anything else. Secret names are org-wide, so give each release its own, and do not use `RELEASE_NAME-debezium-credentials`, which Debezium creates itself. + + + If the secret does not exist at install time, the deployment wedges silently: `cpln logs` returns zero lines, because the containers never start. The only diagnostic is `status.versions[].message` (also for `RELEASE_NAME-cluster` and `RELEASE_NAME-debezium`): + + ```bash + cpln workload get-deployments RELEASE_NAME-postgres-ha --gvc GVC_NAME -o yaml + ``` + + Plain `cpln workload get` has no `versions` key. Create the secret and the deployment recovers on its own after several minutes (poll rather than giving up), or skip the wait with `cpln workload force-redeployment`. + + + + Kafbat UI is off by default. It reads its whole configuration from an **opaque** secret mounted as a file, which must exist **before** you turn it on, and it is public when on. Replace `ADMIN_PASSWORD` with the credentials secret's `kafkaAdminPassword` (read it with `cpln secret reveal my-cdc-pipeline-credentials -o yaml`) and `UI_PASSWORD` with the login you want for the console. The full file format is in the [Kafbat configuration reference](https://ui.docs.kafbat.io/configuration/configuration-file). ```bash cat > kafbat-config.yaml <<'CONF' @@ -67,7 +97,7 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha cat kafbat-config.yaml | cpln secret create-opaque --name SECRET_NAME --encoding plain -f - ``` - Set `kafka.kafbat_ui.configuration_secret` to this name. If you do not want the console, install with `--set kafka.kafbat_ui.enabled=false` instead and skip this step. + Set `kafka.kafbat_ui.enabled` to `true` and `kafka.kafbat_ui.configuration_secret` to this name. If you do not want the console, skip this step. If the secret is missing, the install still succeeds, `cpln logs` returns nothing for `RELEASE_NAME-kafbat-ui`, and the workload never becomes ready. The only place the missing secret is named is `status.versions[].message`: @@ -79,8 +109,8 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha Create the secret and the deployment recovers on its own after a few minutes, or force it with `cpln workload force-redeployment RELEASE_NAME-kafbat-ui --gvc GVC_NAME`. - - The database password, the Kafka passwords and the KRaft cluster ID are template values. Every default starts with `change-me-cdc-pipeline-`; those defaults are placeholders, not working secrets, so replace each one. Generate the KRaft cluster ID with the Kafka tooling, using the same image the brokers run: + + The KRaft cluster ID is a template value, and its default `change-me-cdc-pipeline-kraft-cluster-id` is a placeholder, so replace it. Generate it with the Kafka tooling, using the same image the brokers run: ```bash docker run --rm apache/kafka:3.9.1 /opt/kafka/bin/kafka-storage.sh random-uuid @@ -92,22 +122,16 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha ## Installation -Install into an existing GVC. The database password and the `debezium` Kafka password are each passed twice, because the database and Kafka read one copy and Debezium the other: +Install into an existing GVC once the credentials secret exists. With the default secret name `my-cdc-pipeline-credentials`, only the KRaft cluster ID is required: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/cdc-pipeline \ - --version 1.0.4 --gvc GVC_NAME \ - --set database.password=DB_PASSWORD \ - --set debezium-server.source.database.password=DB_PASSWORD \ - --set kafka.kafka.secrets.kraft_cluster_id=KRAFT_CLUSTER_ID \ - --set kafka.kafka.secrets.inter_broker_password=INTER_BROKER_PASSWORD \ - --set kafka.kafka.secrets.controller_password=CONTROLLER_PASSWORD \ - --set kafka.kafka.listeners.client.sasl.admin.password=ADMIN_PASSWORD \ - --set kafka.kafka.listeners.client.sasl.passwords=DEBEZIUM_KAFKA_PASSWORD \ - --set debezium-server.sink.kafka.saslPassword=DEBEZIUM_KAFKA_PASSWORD \ - --set kafka.kafbat_ui.configuration_secret=SECRET_NAME + --version 1.1.1 --gvc GVC_NAME \ + --set kafka.kafka.secrets.kraft_cluster_id=KRAFT_CLUSTER_ID ``` +With a different secret name, set all six values that name it in a values file (`-f values.yaml`), as shown under [Database Credentials](#database-credentials). + Browse, install, and manage templates visually @@ -136,14 +160,26 @@ Each block shows the shipped defaults for one area of `values.yaml`. Only the ke ### Database Credentials -The bundled database exists only to serve the pipeline, so its credentials are values and the chart writes them into the secret named by `postgres-highly-available.config.credentialsSecretName`. They must match the `debezium-server.source.database` values, which the chart checks at render. +Every credential of the pipeline comes from the one secret created in [Prerequisites](#prerequisites). Its name appears in six values, and all six must name the same secret: ```yaml -database: - username: cdc_user - password: "change-me-cdc-pipeline-postgres" # replace; pass the same value as debezium-server.source.database.password - name: cdcdb - walLevel: logical # must stay logical; checked at render but not applied to the database in 1.0.4 +postgres-highly-available: + walLevel: logical # Debezium's pgoutput decoding needs it; the chart refuses anything else + config: + credentialsSecretName: my-cdc-pipeline-credentials # REQUIRED prerequisite secret, must exist BEFORE install +debezium-server: + credentialsSecretName: my-cdc-pipeline-credentials # the SAME secret +kafka: + kafka: + env: # the SAME secret again, in all four entries; keep the variable names and keys + - name: CDC_KAFKA_ADMIN_PASSWORD + value: cpln://secret/my-cdc-pipeline-credentials.kafkaAdminPassword + - name: CDC_KAFKA_DEBEZIUM_PASSWORD + value: cpln://secret/my-cdc-pipeline-credentials.kafkaSaslPassword + - name: CDC_KAFKA_INTER_BROKER_PASSWORD + value: cpln://secret/my-cdc-pipeline-credentials.kafkaInterBrokerPassword + - name: CDC_KAFKA_CONTROLLER_PASSWORD + value: cpln://secret/my-cdc-pipeline-credentials.kafkaControllerPassword ``` ### PostgreSQL Highly Available @@ -160,8 +196,10 @@ postgres-highly-available: image: controlplanecorporation/patroni-postgres:0.7 + walLevel: logical # Debezium's pgoutput decoding needs it; the chart refuses anything else + config: - credentialsSecretName: my-cdc-pipeline-db-credentials # created by this chart, not by you; give each release in an org its own name + credentialsSecretName: my-cdc-pipeline-credentials # REQUIRED prerequisite secret; must equal debezium-server.credentialsSecretName multiZone: false @@ -207,7 +245,7 @@ See [PostgreSQL Highly Available](/template-catalog/templates/postgres-highly-av ### Kafka -The broker, listener, ACL and secret settings, plus the optional consoles. The exporter blocks (`kafka.kafka_exporter`, `kafka.jmx_exporter`) and `kafka.kafka_client` ship with working defaults; see [Kafka](/template-catalog/templates/kafka) for them. +The broker, listener, ACL and secret settings, plus the optional consoles. Every Kafka password is a `${env:VAR}` reference, never the password: the broker resolves it from one of the `CDC_KAFKA_*` variables in `kafka.kafka.env` (Kafka's `EnvVarConfigProvider`), and the chart refuses a plain password. `kafka.jmx_exporter` and `kafka.kafka_client` ship with working defaults; see [Kafka](/template-catalog/templates/kafka) for them. ```yaml kafka: @@ -217,6 +255,15 @@ kafka: replicas: 3 multiZone: false logDirs: /opt/kafka/logs-0,/opt/kafka/logs-1 + env: # the brokers read their SASL passwords from the credentials secret through these variables; keep all four + - name: CDC_KAFKA_ADMIN_PASSWORD + value: cpln://secret/my-cdc-pipeline-credentials.kafkaAdminPassword + - name: CDC_KAFKA_DEBEZIUM_PASSWORD + value: cpln://secret/my-cdc-pipeline-credentials.kafkaSaslPassword + - name: CDC_KAFKA_INTER_BROKER_PASSWORD + value: cpln://secret/my-cdc-pipeline-credentials.kafkaInterBrokerPassword + - name: CDC_KAFKA_CONTROLLER_PASSWORD + value: cpln://secret/my-cdc-pipeline-credentials.kafkaControllerPassword volumes: logs: initialCapacity: 10 @@ -244,23 +291,28 @@ kafka: sasl: admin: username: admin # ACL super user; use it for consumers and for Kafbat UI - password: "change-me-cdc-pipeline-kafka-admin" + password: "${env:CDC_KAFKA_ADMIN_PASSWORD}" # the secret's kafkaAdminPassword users: "debezium" # comma-separated; must contain debezium-server.sink.kafka.saslUsername - passwords: "change-me-cdc-pipeline-kafka-debezium" # one per user, same order; the debezium entry must equal debezium-server.sink.kafka.saslPassword + passwords: "${env:CDC_KAFKA_DEBEZIUM_PASSWORD}" # the secret's kafkaSaslPassword acl: - superUsers: "User:admin" # semicolon-separated, e.g. User:admin;User:debezium + superUsers: "User:admin;User:debezium" # keep User:debezium: the chart cannot grant ACLs allowEveryoneIfNoAclFound: false secrets: kraft_cluster_id: "change-me-cdc-pipeline-kraft-cluster-id" # generate with kafka-storage.sh random-uuid - inter_broker_password: "change-me-cdc-pipeline-inter-broker" - controller_password: "change-me-cdc-pipeline-controller" + inter_broker_password: "${env:CDC_KAFKA_INTER_BROKER_PASSWORD}" # the secret's kafkaInterBrokerPassword + controller_password: "${env:CDC_KAFKA_CONTROLLER_PASSWORD}" # the secret's kafkaControllerPassword extra_configurations: + config.providers: env # keep both provider lines: they resolve the references + config.providers.env.class: org.apache.kafka.common.config.provider.EnvVarConfigProvider default.replication.factor: 3 auto.create.topics.enable: true # Debezium's topics are created on first write log.retention.hours: 168 + kafka_orchestrator: false # keep off: needs the admin password as a plain value + kafka_exporter: false # keep off: needs the admin password as a plain value + kafbat_ui: - enabled: true + enabled: false # needs an opaque config secret you create first (see Prerequisites), and is public when on name: kafbat-ui image: ghcr.io/kafbat/kafka-ui cpu: 300m @@ -268,18 +320,20 @@ kafka: minCpu: 100m minMemory: 400Mi replicas: 1 - configuration_secret: kafka-kafbat-ui-config # opaque secret you create before installing (see Prerequisites) + configuration_secret: kafka-kafbat-ui-config # opaque secret you create before turning it on (see Prerequisites) firewall: external_inboundAllowCIDR: "0.0.0.0/0" # the console is reachable from the internet by default external_outboundAllowCIDR: "0.0.0.0/0" kafka_rest_proxy: - enabled: false + enabled: false # refused: it needs its Kafka password inline in values kafka_ui: enabled: false ``` +To add another Kafka user, put its password in the credentials secret under a new key, add a `kafka.kafka.env` entry reading it (`cpln://secret/SECRET_NAME.NEW_KEY`), and append the user and `${env:THAT_VARIABLE}` to `users` and `passwords`. Kafka Connect (`kafka.kafka_connectors`) is refused at render, because its worker login and plugin configuration take credentials inline; Debezium Server is this pipeline's connector, so install the [Kafka](/template-catalog/templates/kafka) template on its own to run Kafka Connect. + ### Debezium Server The connector's source and sink. `hostname` and `bootstrapServers` are filled in from the release name when left empty, so the defaults already point at the bundled database and brokers. @@ -288,6 +342,8 @@ The connector's source and sink. `hostname` and `bootstrapServers` are filled in debezium-server: image: quay.io/debezium/server:3.0 + credentialsSecretName: my-cdc-pipeline-credentials # REQUIRED prerequisite secret; must equal postgres-highly-available.config.credentialsSecretName + resources: cpu: 500m memory: 512Mi @@ -298,9 +354,9 @@ debezium-server: database: hostname: "" # empty = RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local port: 5432 - name: cdcdb # must equal database.name - user: cdc_user # must equal database.username - password: "change-me-cdc-pipeline-postgres" # must equal database.password + name: cdcdb # must equal the secret's database key + user: cdc_user # must equal the secret's username key + # password: the secret's sourcePassword key (the same value as password) serverName: "dbserver1" # topic prefix; events go to dbserver1.SCHEMA.TABLE tableIncludeList: "" # comma-separated schema.table list, e.g. public.users,public.orders; empty = all tables @@ -326,8 +382,8 @@ debezium-server: topic: "" securityProtocol: "SASL_PLAINTEXT" saslMechanism: "PLAIN" - saslUsername: "debezium" # must appear in kafka.kafka.listeners.client.sasl.users - saslPassword: "change-me-cdc-pipeline-kafka-debezium" # must equal kafka.kafka.listeners.client.sasl.passwords + saslUsername: "debezium" # must be in kafka.kafka.listeners.client.sasl.users and acl.superUsers + # saslPassword: the secret's kafkaSaslPassword key (the debezium entry of sasl.passwords) format: key: json @@ -346,7 +402,7 @@ debezium-server: - 0.0.0.0/0 ``` -Before the connector starts, its startup step creates the `debezium_heartbeat` table and, once `wal_level` is `logical`, the `debezium` replication slot in the source database, so the heartbeat needs no manual SQL. Pointing `hostname` or `bootstrapServers` at a database or Kafka outside this release still deploys the bundled components, and the render-time credential checks still apply. This release bundles Debezium Server chart 1.1.1, where credentials are values; the [Debezium Server](/template-catalog/templates/debezium-server) page documents a later version that reads them from a secret. +Before the connector starts, its startup step creates the `debezium_heartbeat` table and, once `wal_level` is `logical`, the `debezium` replication slot in the source database, so the heartbeat needs no manual SQL. Pointing `hostname` or `bootstrapServers` at a database or Kafka outside this release still deploys the bundled components, and the render-time credential checks still apply. Debezium reads its database and Kafka passwords from the credentials secret named by `debezium-server.credentialsSecretName`. ## Connecting @@ -354,13 +410,12 @@ Nothing in the pipeline is public except Kafbat UI. Everything else is reachable | What | Address | Notes | |---|---|---| -| Source database, writes and reads | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | HAProxy routes to the current leader. User `database.username`, database `database.name` | -| Kafka bootstrap server | `RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092` | `security.protocol=SASL_PLAINTEXT`, `sasl.mechanism=PLAIN`. Read with the `admin` user | +| Source database, writes and reads | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | HAProxy routes to the current leader. `username`, `password` and `database` of the credentials secret | +| Kafka bootstrap server | `RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092` | `security.protocol=SASL_PLAINTEXT`, `sasl.mechanism=PLAIN`. Read with the `admin` user and the secret's `kafkaAdminPassword` | | Change event topics | `dbserver1.SCHEMA.TABLE`, for example `dbserver1.public.customers` | Prefix is `debezium-server.source.serverName`; JSON keys and values | | Debezium health | `http://RELEASE_NAME-debezium.GVC_NAME.cpln.local:8080/q/health` | Readiness at `/q/health/ready`, liveness at `/q/health/live` | | Kafbat UI | `status.canonicalEndpoint` of `RELEASE_NAME-kafbat-ui` | Login from `spring.security.user` in your configuration secret | -| Database credentials | `cpln secret reveal my-cdc-pipeline-db-credentials -o yaml` | Use the name you set in `postgres-highly-available.config.credentialsSecretName` | -| Kafka credentials | `cpln secret reveal RELEASE_NAME-secrets -o yaml` | KRaft cluster ID and the admin, controller and inter-broker passwords | +| Credentials | `cpln secret reveal my-cdc-pipeline-credentials -o yaml` | Your credentials secret: the database and Kafka passwords. Use the name you set in `postgres-highly-available.config.credentialsSecretName` | Read the Kafbat UI endpoint from the workload: @@ -409,7 +464,7 @@ If no `dbserver1.` topic appears, read the connector log and see [Troubleshootin cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-debezium"}' --limit 50 --since 10m ``` -Applications in the GVC consume the same way, with these client properties and the bootstrap address above. To give a consumer its own Kafka user rather than `admin`, append it to the comma-separated `kafka.kafka.listeners.client.sasl.users` list, with its password at the same position in `passwords`, and give it access, either with Kafka ACLs or by listing it in `kafka.kafka.acl.superUsers`. +Applications in the GVC consume the same way, with these client properties and the bootstrap address above. To give a consumer its own Kafka user rather than `admin`, add it as described under [Kafka](#kafka) (a new key in the credentials secret, a `kafka.kafka.env` entry, and the user and its `${env:VAR}` reference appended to `users` and `passwords`), and give it access, either with Kafka ACLs or by listing it in `kafka.kafka.acl.superUsers`. ## Operations @@ -448,7 +503,109 @@ An existing database keeps the credentials it was created with, so set `database cpln secret reveal RELEASE_NAME-postgres-config -o yaml ``` -This upgrade path has not been verified end to end. Snapshot the database volume set first, as shown under [Backing Up](#backing-up). +This upgrade path has not been verified end to end. Snapshot the database volume set first, as shown under [Backing Up](#backing-up). Then follow [Upgrading from 1.0.4](#upgrading-from-1-0-4). + +### Upgrading from 1.0.4 + +1.0.4 never delivered change events (`wal_level` was `replica` and Debezium had no Kafka permissions), so **a fresh install of 1.1.1 is the simplest path**: install it as a new release, move any data the database holds, then uninstall the old release. To upgrade in place instead: + +1. 1.0.4 created the database secret itself, and **this upgrade deletes it**; it also kept Kafka's passwords as values, which 1.1.x reads from the secret instead. **Before upgrading**, create the new secret under a **new** name holding the values the database and the Kafka cluster run with today, not new ones — the running cluster keeps its passwords, and nothing changes on it. Read the database credentials from `my-cdc-pipeline-db-credentials` (or the name in `postgres-highly-available.config.credentialsSecretName`), the Kafka `client-admin-password`, `inter-broker-password` and `controller-password` from `RELEASE_NAME-secrets`, and Debezium's `kafka-sasl-password` from `RELEASE_NAME-debezium-credentials`: + + ```bash + cpln secret reveal my-cdc-pipeline-db-credentials -o yaml + cpln secret reveal RELEASE_NAME-secrets -o yaml + cpln secret reveal RELEASE_NAME-debezium-credentials -o yaml + cpln secret create-dictionary --name RELEASE_NAME-cdc-credentials \ + --entry username=YOUR-CURRENT-USERNAME --entry password=YOUR-CURRENT-PASSWORD \ + --entry sourcePassword=YOUR-CURRENT-PASSWORD --entry database=YOUR-CURRENT-DATABASE \ + --entry kafkaSaslPassword=CURRENT-KAFKA-SASL-PASSWORD \ + --entry kafkaAdminPassword=CURRENT-CLIENT-ADMIN-PASSWORD \ + --entry kafkaInterBrokerPassword=CURRENT-INTER-BROKER-PASSWORD \ + --entry kafkaControllerPassword=CURRENT-CONTROLLER-PASSWORD + ``` + +2. In your values, delete the `database:` block, `debezium-server.source.database.password` and `debezium-server.sink.kafka.saslPassword`; set both `credentialsSecretName` keys to the new name; and if you changed `kafka.kafka.acl.superUsers`, add `User:debezium`. Replace every Kafka password with its reference and add the `kafka.kafka.env` entries and provider lines exactly as in [Kafka](#kafka) and [Database Credentials](#database-credentials), naming the new secret; delete any `kafka.kafka_exporter` block (1.1.x runs no Kafka exporter or health sidecar). The chart refuses the old keys and any plain Kafka password at render. +3. **Before upgrading**, switch the running cluster to `wal_level = logical` — 1.0.4 created it with `replica`, and 1.1.x replicas refuse to start on a `replica` cluster, which stalls the upgrade. The same command tells Patroni to leave logical slots alone and removes the install-time password 1.0.4 stored in the cluster configuration (`"authentication":null`; without it this command prints that password back to you). Then restart the members in place, the replicas first and the leader last (`patronictl` does not work on 1.0.4's cluster, so this uses Patroni's API; `LOCATION` is your GVC's location): + + ```bash + pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } + pgx 0 curl -s -XPATCH -d '{"postgresql":{"parameters":{"wal_level":"logical"},"authentication":null},"ignore_slots":[{"type":"logical"}]}' localhost:8008/config + # wait until every member shows "pending_restart": true; note which one is the "leader" + pgx 0 curl -s localhost:8008/cluster + # for each replica N, then the leader last: + pgx N curl -s -XPOST -d '{"restart_pending": true}' localhost:8008/restart + ``` + + Run the `cluster` call again: every member should be `running` or `streaming` with no `pending_restart`. +4. Upgrade. The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Debezium can briefly lose its database; it reconnects by itself. The Kafka brokers restart as well, now reading their passwords from the secret. The new version of a workload can also wait several minutes with `is not allowed to reveal the secret` in `get-deployments`, while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. +5. Confirm Debezium's slot is a failover slot: `pgx N psql -h 127.0.0.1 -d postgres -c "SELECT slot_name, failover FROM pg_replication_slots"` on the leader shows `failover = t`. If it shows `f` (1.0.4's Debezium created it), stop the connector, convert the slot from any `psql` that reaches the current leader (the slot must be inactive), then start the connector again: + + ```bash + cpln workload stop RELEASE_NAME-debezium --gvc GVC_NAME + psql "host=DB_HOST user=DB_USER dbname=DB_NAME replication=database" -c 'ALTER_REPLICATION_SLOT debezium ( FAILOVER true )' + cpln workload start RELEASE_NAME-debezium --gvc GVC_NAME + ``` + +The bundled components move too: PostgreSQL Highly Available 2.5.0 → 2.8.0, Kafka 4.0.1 → 4.3.0 (the brokers roll, which also applies the new super user), and Kafbat UI keeps the setting your release runs with. + +### Rotating the Database Password in HA Mode + +The HA database needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Change BOTH roles on -0 +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +``` + +Straight after step 3, put the new password in the secret as BOTH `password` and `sourcePassword`, confirm it, and restart the members: + +```bash +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only password and sourcePassword in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +``` + +Once the redeployment has finished, confirm one leader and two `streaming` replicas on one timeline (repeat step 1); `-0` restarts last, so the leader moves once more, to another member. For a release upgraded from 1.0.4 that skipped `"authentication":null` in its upgrade, also remove the install-time password Patroni kept in its cluster configuration: + +```bash +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +The procedure moves the leader twice. Debezium's replication slot is a failover slot that every replica keeps a synchronized copy of, so it resumes on each new leader from where it stopped; changes around a leader change can be delivered twice, and the database drops connections briefly at each leader change. Debezium fails to authenticate from the `ALTER ROLE` until you have edited the secret: its container then restarts by itself and reads the new `sourcePassword`, so it needs no redeploy of its own. If it still logs `password authentication failed` a few minutes after the edit, run `cpln workload force-redeployment RELEASE_NAME-debezium --gvc GVC_NAME`. + +**Any `helm upgrade` after the secret has been edited restarts `RELEASE_NAME-postgres-ha`, `RELEASE_NAME-cluster` and `RELEASE_NAME-debezium` again** (one more rolling restart of the database, with its leader changes), even when no value changed, because they are rolled whenever the secret they read has changed since the last upgrade. Plan the next upgrade into the same window. If a member stays `starting`, or `running` without `streaming` for several minutes, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + +### Rotating the Kafka Passwords + +All four Kafka passwords live only in the credentials secret, so a rotation is a secret edit and two redeployments, with no `helm upgrade`. The brokers and Debezium keep the passwords they started with until they restart, so run the steps back to back: a broker that restarts on its own in between comes back with passwords its peers do not have yet. + +```bash +# 1. new values for any of kafkaSaslPassword, kafkaAdminPassword, kafkaInterBrokerPassword, kafkaControllerPassword +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only the keys you are rotating in secret.yaml +cpln apply -f secret.yaml +# 2. confirm the new values BEFORE redeploying +cpln secret reveal SECRET_NAME -o yaml +# 3. the brokers restart with the new passwords +cpln workload force-redeployment RELEASE_NAME-cluster --gvc GVC_NAME +# 4. only when kafkaSaslPassword changed; right after step 3 +cpln workload force-redeployment RELEASE_NAME-debezium --gvc GVC_NAME +``` + +The brokers restart one at a time. When `kafkaInterBrokerPassword` or `kafkaControllerPassword` changed, a restarted broker cannot rejoin the brokers still on the old passwords, so partway through the roll Kafka refuses writes until the last broker has restarted; rotate those two in a quiet window. Debezium is refused from step 3 until step 4, then resumes from its replication slot, so nothing is lost and events around the switch can be delivered twice. Any other client that logs in as `admin` or `debezium` needs the new password too. ### Scaling and Availability @@ -461,7 +618,7 @@ This upgrade path has not been verified end to end. Snapshot the database volume - **Cause:** in version 1.0.4 this is expected with the defaults, for up to three reasons. The bundled PostgreSQL cluster runs with `wal_level = replica`, which `pgoutput` logical decoding cannot use; `database.walLevel` is validated but not applied. Kafka enforces ACLs with `allowEveryoneIfNoAclFound: false`, and the `debezium` user is neither a super user nor granted any ACL. Or `debezium-server.sink.kafka.saslPassword` differs from the `debezium` entry in `kafka.kafka.listeners.client.sasl.passwords` — the render-time check compares only the user name. + **Cause:** in version 1.0.4 this is expected with the defaults, for up to three reasons. The bundled PostgreSQL cluster runs with `wal_level = replica`, which `pgoutput` logical decoding cannot use; `database.walLevel` is validated but not applied. Kafka enforces ACLs with `allowEveryoneIfNoAclFound: false`, and the `debezium` user is neither a super user nor granted any ACL. **Fix:** read the connector log, filtered server-side, to see which applies: @@ -469,32 +626,37 @@ This upgrade path has not been verified end to end. Snapshot the database volume cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-debezium"} |= "ERROR"' --limit 50 --since 10m ``` - Align the two Kafka passwords if they differ. Adding `User:debezium` to `kafka.kafka.acl.superUsers` puts it in the brokers' super-user list. Patroni keeps `wal_level` in its cluster-wide configuration, so it is not changed by any value of this chart. Neither change has been verified end to end on this template. + Adding `User:debezium` to `kafka.kafka.acl.superUsers` puts it in the brokers' super-user list. Patroni keeps `wal_level` in its cluster-wide configuration, so it is not changed by any value of this chart. Neither change has been verified end to end on this template. **Cause:** the secret named in `kafka.kafbat_ui.configuration_secret` did not exist when the workload was created. The container never starts, so there is no log output. **Fix:** read `status.versions[].message` from `cpln workload get-deployments RELEASE_NAME-kafbat-ui --gvc GVC_NAME -o yaml` to see which secret is missing, create it as shown in [Prerequisites](#prerequisites), then wait a few minutes or run `cpln workload force-redeployment RELEASE_NAME-kafbat-ui --gvc GVC_NAME`. - - **Cause:** `database.username`, `database.password` or `database.name` differs from `debezium-server.source.database.user`, `.password` or `.name`. The error text still calls the database side `postgres-highly-available.postgres`, the name these keys had before 1.0.4. + + **Cause:** `postgres-highly-available.config.credentialsSecretName`, `debezium-server.credentialsSecretName` and the four `kafka.kafka.env` entries do not all name one secret. + + **Fix:** set all six to the name of your credentials secret, as shown under [Database Credentials](#database-credentials). + + + **Cause:** your values still carry the 1.0.4 credential keys (`database:`, `debezium-server.source.database.password`, `debezium-server.sink.kafka.saslPassword`) or a plain Kafka password. - **Fix:** set both sides to the same values; the install line in [Installation](#installation) passes the password twice for this reason. + **Fix:** follow [Upgrading from 1.0.4](#upgrading-from-1-0-4). **Cause:** `debezium-server.sink.kafka.saslUsername` does not appear in `kafka.kafka.listeners.client.sasl.users`. **Fix:** add the user to the Kafka list, or set `saslUsername` to a user already in it. - - **Cause:** `database.walLevel` was set to something other than `logical`. + + **Cause:** `postgres-highly-available.walLevel` was set to something other than `logical`. **Fix:** leave it at `logical`. - **Cause:** the database credentials secret is named `my-cdc-pipeline-db-credentials` by default, without the release name, and secrets are org-wide. + **Cause:** the credentials secret is named `my-cdc-pipeline-credentials` by default, without the release name, and secrets are org-wide. - **Fix:** give every release in the org its own `postgres-highly-available.config.credentialsSecretName`. + **Fix:** give every release in the org its own credentials secret, and set all six values that name it. **Cause:** a replication slot is not advancing — the connector stopped reading, or a slot was left behind after `debezium-server.source.postgres.slotName` changed. PostgreSQL keeps WAL for an inactive slot indefinitely. @@ -510,11 +672,14 @@ This upgrade path has not been verified end to end. Snapshot the database volume ## Important Notes - In version 1.0.4 the pipeline does not deliver change events with its defaults; see [Troubleshooting](#troubleshooting) and verify events arrive before relying on it. -- Replace every `change-me-cdc-pipeline-` value before installing; they are plain template values and land in the Helm release. -- Pass the database password and the `debezium` Kafka password to both the component that owns them and `debezium-server`; only the database values and the Kafka user name are checked at render. +- Create the credentials secret before installing; a missing secret wedges the deployment silently, and only `status.versions[].message` from `cpln workload get-deployments` names it. +- All six values that name the credentials secret must name the same one; the chart refuses anything else. +- Rotating the database password in HA mode needs a leader switchover first; follow [Rotating the Database Password in HA Mode](#rotating-the-database-password-in-ha-mode). A plain redeploy can promote a member still holding the old password and split the cluster. +- Rotating a Kafka password needs both redeployments; see [Rotating the Kafka Passwords](#rotating-the-kafka-passwords). +- This template runs no Kafka exporter, health sidecar, REST Proxy or Kafka Connect, because each needs a password inline; the chart refuses them. - `kafka.kafka.secrets.kraft_cluster_id` is fixed once the brokers have started. - Kafbat UI is open to the internet by default; set a strong console login or narrow `kafka.kafbat_ui.firewall.external_inboundAllowCIDR`. -- Give each release its own `postgres-highly-available.config.credentialsSecretName`; the default name is shared org-wide. +- Give each release its own credentials secret; the default name is shared org-wide. - Uninstalling deletes every volume set — the source database, the Kafka topics and the Debezium offsets; final snapshots are kept for seven days. - Component versions are pinned in this chart (PostgreSQL Highly Available 2.5.0, Kafka 4.0.1, Debezium Server 1.1.1) and change only with a new CDC Pipeline version. diff --git a/template-catalog/templates/chatwoot.mdx b/template-catalog/templates/chatwoot.mdx index 474f7443..e8647f77 100644 --- a/template-catalog/templates/chatwoot.mdx +++ b/template-catalog/templates/chatwoot.mdx @@ -1,7 +1,7 @@ --- title: Chatwoot icon: /template-catalog/templates/icons/chatwoot.png -description: Deploy Chatwoot, the open-source live chat, shared inbox and omnichannel agent desk (Community Edition) with a background worker, a single-node Redis and a PostgreSQL that is highly available by default or single-instance. Needs an existing GVC and a dictionary secret holding Chatwoot's four keys. +description: Deploy Chatwoot, the open-source live chat, shared inbox and omnichannel agent desk (Community Edition) with a background worker, a single-node Redis and a PostgreSQL that is highly available by default or single-instance. Needs an existing GVC and three dictionary secrets: keys, database and Redis. keywords: ['chat widget', 'help desk', 'customer support', 'intercom alternative', 'zendesk alternative', 'freshdesk alternative', 'crisp alternative', 'sidekiq', 'actioncable', 'websockets', 'pgvector', 'rails', 'chatwoot template'] --- @@ -30,8 +30,6 @@ This template does not create a GVC. It deploys into an existing [GVC](/referenc | Volume Set | `RELEASE_NAME-chatwoot-redis-vs` | Redis append-only file at `/data` | — | | Volume Set | `RELEASE_NAME-postgres-ha-vs`, `RELEASE_NAME-etcd-vs` | Database and etcd data, one volume per replica | `postgresHA.enabled: true` (default) | | Volume Set | `RELEASE_NAME-pg-vs` | Single-instance database data | `postgres.enabled: true` | -| Secret | `my-chatwoot-db-credentials` | Dictionary secret with the database `username`, `password` and `database`, built by this template from `postgres.credentials.*`; the name comes from `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` | — | -| Secret | `RELEASE_NAME-chatwoot-creds` | Dictionary secret holding the bundled Redis password | — | | Secret | `RELEASE_NAME-chatwoot-web-start`, `RELEASE_NAME-chatwoot-worker-start` | Start scripts for the web and worker containers | — | | Secret | `RELEASE_NAME-etcd-startup`, `RELEASE_NAME-patroni-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-wal-g-backup-script` | Start scripts of the highly available database | `postgresHA.enabled: true` (default) | | Secret | `RELEASE_NAME-postgres-config` | Non-sensitive backup settings of the highly available database | `postgresHA.backup.enabled: true` | @@ -40,11 +38,11 @@ This template does not create a GVC. It deploys into an existing [GVC](/referenc | Identity, Policy | `RELEASE_NAME-postgres-ha-identity`, `RELEASE_NAME-postgres-ha-policy`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy` | Database and etcd identities and their secret access | `postgresHA.enabled: true` (default) | | Identity, Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | Single-instance database identity and its secret access | `postgres.enabled: true` | -Your keys secret (see [Prerequisites](#prerequisites)) is **not** created by the template. +Your three prerequisite secrets — the keys, the database credentials and the Redis password (see [Prerequisites](#prerequisites)) — are **not** created by the template. ## Prerequisites -Chatwoot signs its sessions and encrypts sensitive columns with four keys that you supply through a [dictionary secret](/guides/create-secret/dictionary) created **before** installing. The values never pass through Helm values. +Three [dictionary secrets](/guides/create-secret/dictionary) must exist **before** installing: the four keys Chatwoot signs its sessions and encrypts sensitive columns with, the database credentials, and the Redis password. The values never pass through Helm values. @@ -63,13 +61,32 @@ Chatwoot signs its sessions and encrypts sensitive columns with four keys that y Store a copy outside Control Plane. All four keys are write-once for the life of the installation: rotating `SECRET_KEY_BASE` logs out every user, and rotating any `ACTIVE_RECORD_ENCRYPTION_*` key makes stored two-factor secrets undecryptable, locking out every agent who enabled MFA. - - The database password (`postgres.credentials.password`) and the Redis password (`redis.auth.password`) ship as `change-me-…` placeholders that the template uses as-is. Pick strong values and pass them at install time — both seed their component on first boot and are not updated by later value edits. The Redis password may contain only letters, digits, `-` and `_`. + + The bundled PostgreSQL initializes with these values and Chatwoot connects with them, in either database mode: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=chatwoot \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=chatwoot + ``` + + Set its name in the key of the database mode you use: `postgresHA.config.credentialsSecretName` (HA, the default) or `postgres.config.credentialsSecretName` (single-instance). Secret names are org-wide, so give each Chatwoot release its own secret. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password). + + + One key, `password`; the bundled Redis requires it and Chatwoot connects with it: + + ```bash + cpln secret create-dictionary --name REDIS_SECRET_NAME \ + --entry password="$(openssl rand -hex 24)" + ``` + + Set `redis.credentialsSecretName` to this name. The password is part of Chatwoot's Redis URL, so use letters, digits, `-` and `_` only. -**If the keys secret does not exist at install time, the deployment wedges silently.** `cpln logs` returns zero lines — the container never starts, so it has nothing to log. The one place the reason appears is `status.versions[].message`: +**If any of these secrets does not exist at install time, the deployment wedges silently.** `cpln logs` returns zero lines — the container never starts, so it has nothing to log. The one place the reason appears is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-chatwoot --gvc GVC_NAME -o yaml @@ -86,14 +103,14 @@ Everything else works with the defaults. Three optional features need their own ## Installation -Once the keys secret exists, install into your existing GVC with the two bundled passwords set: +Once the three secrets exist, install into your existing GVC with their names set: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/chatwoot \ - --version 1.2.1 --gvc GVC_NAME \ + --version 1.3.0 --gvc GVC_NAME \ --set secrets.name=CHATWOOT_KEYS_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' \ - --set redis.auth.password='YOUR-STRONG-REDIS-PASSWORD' + --set postgresHA.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME \ + --set redis.credentialsSecretName=REDIS_SECRET_NAME ``` To install using another method, follow the instructions for it: @@ -312,7 +329,7 @@ The worker and the bundled Redis are never publicly reachable: the worker accept ### Database -Enable exactly one of `postgresHA` (default) and `postgres` — the chart refuses to render with both or neither. Chatwoot is wired to the active database automatically: the HAProxy leader endpoint in HA mode, or the single instance directly. In **both** modes the database credentials come from `postgres.credentials.*`: this template writes them into a dictionary secret named by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` and hands that name to the bundled database. There is nothing for you to create. +Enable exactly one of `postgresHA` (default) and `postgres` — the chart refuses to render with both or neither. Chatwoot is wired to the active database automatically: the HAProxy leader endpoint in HA mode, or the single instance directly. In **both** modes the database credentials come from your prerequisite dictionary secret (`username`, `password` and `database` — see [Prerequisites](#prerequisites)), named by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName`. ```yaml # Highly available PostgreSQL (default): 3 Patroni replicas (PostgreSQL 17.5), 3 etcd replicas, @@ -320,7 +337,7 @@ Enable exactly one of `postgresHA` (default) and `postgres` — the chart refuse postgresHA: enabled: true config: - credentialsSecretName: my-chatwoot-db-credentials # secret this chart CREATES from postgres.credentials.*; org-wide name — give each release its own + credentialsSecretName: my-chatwoot-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install replicas: 3 resources: # per Patroni replica; the default install runs 3 of these minCpu: 250m @@ -358,12 +375,8 @@ postgresHA: postgres: enabled: false image: pgvector/pgvector:pg18 # MUST carry pgvector — the stock postgres:18 image does NOT (render is refused) - credentials: # bundled database credentials, used in BOTH modes - username: chatwoot - password: change-me-chatwoot-pg # change before install - database: chatwoot config: - credentialsSecretName: my-chatwoot-db-credentials # secret this chart CREATES; org-wide name — give each release its own + credentialsSecretName: my-chatwoot-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 250m maxCpu: 1000m @@ -399,17 +412,12 @@ The other keys of the bundled [postgres-highly-available](/template-catalog/temp Choose the database mode **before** installing — it cannot be switched later. Flipping `postgresHA.enabled` / `postgres.enabled` on a live release points Chatwoot at a different, empty database (a separate volume set and a different PostgreSQL major version), so the app re-runs onboarding and your existing conversations are orphaned rather than migrated. - -Secret names are org-wide, and the credentials secret is created by this template in both modes. If you run more than one release of this template in the same org, give each its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName`). A second release left on the default name is **refused at install** and creates nothing; the first release is unaffected. - - ### Redis ```yaml redis: image: redis:8 - auth: - password: change-me-chatwoot-redis # template-managed; wired into REDIS_URL — letters/digits/-/_ only + credentialsSecretName: my-chatwoot-redis # PREREQUISITE dictionary secret (key: password; letters, digits, - and _) — must exist BEFORE install resources: minCpu: 100m maxCpu: 400m @@ -419,7 +427,7 @@ redis: capacity: 10 # GiB (minimum 10); AOF at /data ``` -Redis is required and runs as a **single node by design**: it carries the pub/sub channel behind every live update, and Redis does not propagate published messages between replicas, so a multi-node topology would silently drop a share of the updates that make the dashboard and widget refresh in real time. Persistence is on and eviction is disabled, so a queued job is never dropped. Change `redis.auth.password` before installing; only letters, digits, `-` and `_` are accepted because the value is embedded in a connection URL. +Redis is required and runs as a **single node by design**: it carries the pub/sub channel behind every live update, and Redis does not propagate published messages between replicas, so a multi-node topology would silently drop a share of the updates that make the dashboard and widget refresh in real time. Persistence is on and eviction is disabled, so a queued job is never dropped. The password comes from the prerequisite secret named by `redis.credentialsSecretName` (key `password`); only letters, digits, `-` and `_` are accepted because the value is embedded in a connection URL. See [Rotating the Redis Password](#rotating-the-redis-password). ## Connecting @@ -432,14 +440,14 @@ Redis is required and runs as a **single node by design**: it carries the pub/su | Login | The account created by the onboarding wizard | See [First Run](#first-run) | | PostgreSQL (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | Credentials in the secret named by `postgresHA.config.credentialsSecretName` | | PostgreSQL (single-instance mode) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Credentials in the secret named by `postgres.config.credentialsSecretName` | -| Redis | `RELEASE_NAME-chatwoot-redis.GVC_NAME.cpln.local:6379` | Password in the `redisPassword` key of `RELEASE_NAME-chatwoot-creds` | +| Redis | `RELEASE_NAME-chatwoot-redis.GVC_NAME.cpln.local:6379` | Password in the `password` key of the secret named by `redis.credentialsSecretName` | Read the public hostname and the stored credentials: ```bash cpln workload get RELEASE_NAME-chatwoot --gvc GVC_NAME -o yaml # status.canonicalEndpoint cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml # the name you set in *.config.credentialsSecretName -cpln secret reveal RELEASE_NAME-chatwoot-creds -o yaml # redisPassword +cpln secret reveal REDIS_SECRET_NAME -o yaml # the name you set in redis.credentialsSecretName ``` To verify the web tier from your machine without going through the public endpoint, forward port `3000` and query the readiness route, which reports the PostgreSQL and Redis status: @@ -568,6 +576,82 @@ Two Chatwoot-specific points: - Attachments are restored separately: copy them back into the bucket (object-storage mode) or onto the `RELEASE_NAME-chatwoot-storage` volume set (local mode). +### Rotating the Database Password + +PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with, so a rotation takes three steps in order (single-instance; in HA mode follow [Rotating the Database Password in HA Mode](#rotating-the-database-password-in-ha-mode) instead): + +1. Change it in the database: `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` (connected as that user, from inside the GVC). +2. Update the secret — export it, change only `password`, and apply it: + + ```bash + cpln secret reveal SECRET_NAME -o yaml > secret.yaml + # edit only the password key in secret.yaml + cpln apply -f secret.yaml + ``` + +3. Right away, redeploy Chatwoot — the worker first, then the web tier — and wait until `get-deployments` shows a higher version number, alone and ready (`ready: true` alone can still describe the old replica): + + ```bash + cpln workload force-redeployment RELEASE_NAME-chatwoot-worker --gvc GVC_NAME + cpln workload force-redeployment RELEASE_NAME-chatwoot --gvc GVC_NAME + ``` + + Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. + +Connections the app already has open keep working after `ALTER ROLE`; new ones fail until its redeploy finishes, so redeploy right away and rotate in a quiet window. + +### Rotating the Database Password in HA Mode + +The HA database (`postgresHA`) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Change BOTH roles on -0 +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +# 4. Right after, put the same password in the secret (edit only the password key in secret.yaml) +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +cpln apply -f secret.yaml +# 5. Restart the members +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 6. Once the restart completes, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 7. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-chatwoot-worker`, then `RELEASE_NAME-chatwoot`, which keep the old password until they restart. The database drops connections briefly at each switchover, and the app cannot open new connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + +### Rotating the Redis Password + +Redis keeps no password on disk: it takes the password from the secret each time it starts, and so do the Chatwoot workloads. + +1. Update the secret — export it, change only `password`, apply it, and check it: + + ```bash + cpln secret reveal SECRET_NAME -o yaml > secret.yaml + # edit only the password key in secret.yaml + cpln apply -f secret.yaml + cpln secret reveal SECRET_NAME -o yaml + ``` + +2. Redeploy Redis, and wait until `get-deployments` shows a higher version number, alone and ready: + + ```bash + cpln workload force-redeployment RELEASE_NAME-chatwoot-redis --gvc GVC_NAME + ``` + +3. Then redeploy `RELEASE_NAME-chatwoot-worker`, then `RELEASE_NAME-chatwoot`, the same way. The web workload must restart after Redis in any case (see [Scaling and Availability](#scaling-and-availability)). + +Chatwoot cannot reach Redis from step 2 until its own redeploys finish, so expect a short interruption and rotate in a quiet window. + This restore path has **not** been exercised against a Chatwoot install. The database procedure is the bundled template's own; follow it with a test release first, and keep the four keys from your prerequisite secret with the backup — a dump restored under different `ACTIVE_RECORD_ENCRYPTION_*` keys cannot decrypt the two-factor secrets it contains. @@ -580,26 +664,53 @@ Version `1.0.1` only updates the bundled highly available database to a release Version `1.1.0` changed the single-instance database path: -- `postgres.config.username`, `postgres.config.password` and `postgres.config.database` moved to `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database`. The new `postgres.config.credentialsSecretName` names the dictionary secret this template now creates from those three values. Carrying the old keys fails the render with `config.username was REMOVED in postgres 3.4.0` — move the three keys and you are done, and ignore that message's advice to create a secret yourself. +- `postgres.config.username`, `postgres.config.password` and `postgres.config.database` were removed. Carrying the old keys fails the render with `config.username was REMOVED in postgres 3.4.0`. Put the values your single-instance database was initialized with into the database credentials secret under [Prerequisites](#prerequisites) (keys `username`, `password` and `database`), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. - `postgres.backup.minio.accessKey` and `postgres.backup.minio.secretKey` were replaced by `postgres.backup.minio.credentialsSecretName`, a prerequisite dictionary secret holding `accessKey` and `secretKey` — see [Backing Up](#backing-up). -Keep `postgres.credentials.*` equal to the values your database was initialised with; the running database still enforces them. +Then move the Redis password as described in [Upgrading From 1.2.1 or Earlier](#upgrading-from-1-2-1-or-earlier). ### Upgrading From 1.1.0 or Earlier Version `1.2.0` moved the highly available database to `postgres-highly-available` 2.5.0, which no longer creates its own credentials secret: - `postgresHA.postgres.username`, `postgresHA.postgres.password` and `postgresHA.postgres.database` were removed. Carrying them fails the render with `the postgres block was REMOVED in 2.5.0`. Delete the block. -- In HA mode the credentials now come from `postgres.credentials.*` — the same keys the single-instance path uses — and this template writes them into the secret named by the new `postgresHA.config.credentialsSecretName`. +- In HA mode the credentials now come from the database credentials secret under [Prerequisites](#prerequisites), named by `postgresHA.config.credentialsSecretName`. -**Use the credentials your cluster already has.** Set `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database` to exactly the values your `postgresHA.postgres.*` block held. The database was initialised with them and still enforces them; putting new values in the secret does not change the running database — it gives Chatwoot a password that no longer works. +**Use the credentials your cluster already has.** Put exactly the values your `postgresHA.postgres.*` block held into the secret's `username`, `password` and `database` keys. The database was initialized with them and still enforces them; putting new values in the secret does not change the running database — it gives Chatwoot a password that no longer works. +Then move the Redis password as described in [Upgrading From 1.2.1 or Earlier](#upgrading-from-1-2-1-or-earlier). + ### Upgrading From 1.2.0 or Earlier Version `1.2.1` replaced `postgresHA.backup.minio.accessKey` and `postgresHA.backup.minio.secretKey` with `postgresHA.backup.minio.credentialsSecretName`, a prerequisite dictionary secret holding `accessKey` and `secretKey`. Only installs backing up the HA database to an S3-compatible server are affected — see [Backing Up](#backing-up) for the secret. +### Upgrading From 1.2.1 or Earlier + +Version `1.3.0` moved the database credentials and the Redis password to prerequisite secrets: + +- `postgres.credentials` was removed: the database credentials are now a prerequisite secret, in both database modes. Carrying the old key forward fails the render. +- `redis.auth.password` was removed: the Redis password is now a prerequisite secret named by `redis.credentialsSecretName`. Carrying it forward fails the render. + +`1.2.x` created both secrets itself, so **this upgrade deletes them**. Before upgrading, create secrets under **new** names holding the values your release uses today — not new ones, or Chatwoot loses its database and Redis connections: + +```bash +cpln secret reveal my-chatwoot-db-credentials -o yaml # default name: use your credentialsSecretName if you changed it +cpln secret create-dictionary --name RELEASE_NAME-chatwoot-db \ + --entry username='YOUR_CURRENT_USERNAME' \ + --entry password='YOUR_CURRENT_PASSWORD' \ + --entry database='YOUR_CURRENT_DATABASE' + +cpln secret reveal RELEASE_NAME-chatwoot-creds -o yaml # redisPassword is the CURRENT Redis password +cpln secret create-dictionary --name RELEASE_NAME-chatwoot-redis \ + --entry password='YOUR_CURRENT_REDIS_PASSWORD' +``` + +The current values are also in `postgres.credentials` and `redis.auth.password` of the `1.2.x` release. Then upgrade with `postgresHA.config.credentialsSecretName` (HA mode) or `postgres.config.credentialsSecretName` (single-instance mode) and `redis.credentialsSecretName` set to the new names, and with `postgres.credentials` and `redis.auth` removed from your values. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Chatwoot can briefly lose its database; it reconnects by itself. The new version of a workload can also wait several minutes with `is not allowed to reveal the secret` in `get-deployments` while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. + ### Scaling and Availability - **Web tier** — `chatwoot.replicas` sets how many web replicas run. Replicas share the database and the bundled Redis, so a message received by one replica reaches a WebSocket client connected to another, and no sticky sessions are needed. Anything above `1` requires `storage.type` `s3` or `s3-compatible`; the chart refuses to render the combination with local storage. @@ -631,7 +742,7 @@ Version `1.2.1` replaced `postgresHA.backup.minio.accessKey` and `postgresHA.bac **Symptom:** `RELEASE_NAME-chatwoot` and `RELEASE_NAME-chatwoot-worker` never reach ready, and `cpln logs` returns zero lines. - **Cause:** a secret referenced by name does not exist — usually the keys secret (`secrets.name`), or the storage or SMTP secret when those features are enabled. The container never starts, so there is nothing to log. + **Cause:** a secret referenced by name does not exist — usually the keys secret (`secrets.name`), the database credentials secret (`*.config.credentialsSecretName`) or the Redis password secret (`redis.credentialsSecretName`), or the storage or SMTP secret when those features are enabled. The container never starts, so there is nothing to log. **Fix:** read `status.versions[].message`, which names the missing secret, then create it. The deployment recovers on its own after several minutes, or force a redeployment to skip the wait. @@ -667,21 +778,14 @@ Version `1.2.1` replaced `postgresHA.backup.minio.accessKey` and `postgresHA.bac **Cause:** your values still carry the `1.0.x` single-instance keys `postgres.config.username`, `postgres.config.password` or `postgres.config.database`. - **Fix:** move the three keys to `postgres.credentials.*` as described in [Upgrading From 1.0.1 or Earlier](#upgrading-from-1-0-1-or-earlier). Do not create a secret yourself — this template creates it. + **Fix:** put the three values into the database credentials secret and remove the keys, as described in [Upgrading From 1.0.1 or Earlier](#upgrading-from-1-0-1-or-earlier). **Symptom:** `helm upgrade` fails before anything is applied, citing `the postgres block was REMOVED in 2.5.0`. **Cause:** your values still carry `postgresHA.postgres.username`, `postgresHA.postgres.password` or `postgresHA.postgres.database` from version `1.1.0` or earlier. - **Fix:** delete the `postgresHA.postgres` block and put the same three values under `postgres.credentials.*` — see [Upgrading From 1.1.0 or Earlier](#upgrading-from-1-1-0-or-earlier). - - - **Symptom:** installing a second Chatwoot release in the same org fails with `cannot be updated because it is being managed by a different release`, and nothing is created. - - **Cause:** both releases use the same `credentialsSecretName`. Secret names are org-wide, and the first release owns that secret. - - **Fix:** give the second release its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName`). The first release is unaffected. + **Fix:** delete the `postgresHA.postgres` block and put the same three values into the database credentials secret — see [Upgrading From 1.1.0 or Earlier](#upgrading-from-1-1-0-or-earlier). **Symptom:** with `storage.type: s3-compatible`, uploads succeed but images and files fail to load for agents or visitors. @@ -708,9 +812,9 @@ Version `1.2.1` replaced `postgresHA.backup.minio.accessKey` and `postgresHA.bac ## Important Notes -- **Create the keys secret before installing** — `secrets.name` must name an existing dictionary secret holding `SECRET_KEY_BASE` and the three `ACTIVE_RECORD_ENCRYPTION_*` keys; a missing secret wedges the deployment silently. +- **Create the three prerequisite secrets before installing** — `secrets.name` (`SECRET_KEY_BASE` and the three `ACTIVE_RECORD_ENCRYPTION_*` keys), the database credentials (`username`, `password`, `database`) and the Redis password (`password`); a missing secret wedges the deployment silently, and `status.versions[].message` from `cpln workload get-deployments RELEASE_NAME-chatwoot --gvc GVC_NAME -o yaml` names it. - **All four keys are write-once** — rotating `SECRET_KEY_BASE` logs out every user, and rotating an `ACTIVE_RECORD_ENCRYPTION_*` key makes stored two-factor secrets undecryptable. Back the values up outside Control Plane. -- **Set `postgres.credentials.password` and `redis.auth.password` at install time** — both ship as placeholders, seed their component on first boot, and are not updated by later value edits. +- **Rotating a password needs a forced redeployment** — the database password also needs `ALTER ROLE` (and, in HA mode, a leader switchover first); see [Rotating the Database Password](#rotating-the-database-password) and [Rotating the Redis Password](#rotating-the-redis-password). - **Complete the onboarding wizard right after install** — the first browser session to reach the endpoint creates the super admin, with no email confirmation. - **Restart the web workload after any Redis restart** — live updates stop silently until it restarts. - **Choose the database mode before installing** — switching `postgresHA` and `postgres` on a live release points Chatwoot at a different, empty database. @@ -720,10 +824,10 @@ Version `1.2.1` replaced `postgresHA.backup.minio.accessKey` and `postgresHA.bac - **AWS S3 is keyless only** — a cloud account plus a bucket-scoped IAM policy; static keys are accepted only for S3-compatible endpoints. - **With SMTP off, no mail is delivered** — configure `smtp.*` before inviting agents. - **Do not scale the worker or Redis** — both are fixed singletons; scale background throughput with `worker.concurrency`. -- **Give each release its own `credentialsSecretName`** — secret names are org-wide and a second release on the default name is refused at install. +- **Give each release its own database credentials secret** — secret names are org-wide. - **Self-serve signup is managed in the app** — turn it on at `/super_admin` under **Settings**; it is off on a fresh install. - **Enterprise features are not included** — the `-ce` image omits SSO/SAML, audit logs, agent capacity management, custom branding, SLA policies and Captain AI. -- **Data survives restarts and upgrades** — conversations live in the database volume sets and local attachments in `RELEASE_NAME-chatwoot-storage`. Uninstalling deletes those volume sets; your keys secret survives an uninstall. +- **Data survives restarts and upgrades** — conversations live in the database volume sets and local attachments in `RELEASE_NAME-chatwoot-storage`. Uninstalling deletes those volume sets; your prerequisite secrets survive an uninstall. ## External References diff --git a/template-catalog/templates/cpln-task-runner.mdx b/template-catalog/templates/cpln-task-runner.mdx index 9868dbaf..5d75e2a2 100644 --- a/template-catalog/templates/cpln-task-runner.mdx +++ b/template-catalog/templates/cpln-task-runner.mdx @@ -1,7 +1,7 @@ --- title: CPLN Task Runner icon: /template-catalog/templates/icons/cpln-task-runner.png -description: CPLN Task Runner is a self-hosted HTTP task queue and scheduler like Google Cloud Tasks. Deploys an API, autoscaling workers and a bundled Redis Sentinel cluster into an existing GVC. An opaque admin API key secret must exist before install. +description: CPLN Task Runner is a self-hosted HTTP task queue and scheduler like Google Cloud Tasks. Deploys an API, autoscaling workers and a bundled Redis Sentinel cluster into an existing GVC. An admin API key secret and Redis and Sentinel password secrets must exist before install. keywords: ['job runner', 'cloud tasks alternative', 'sidekiq alternative', 'celery alternative', 'background worker', 'delayed job', 'work queue', 'job processor', 'webhook delivery', 'priority queue', 'prerequisite secret', 'admin api key', 'redis sentinel', 'ssrf'] --- @@ -27,21 +27,30 @@ This template deploys into an **existing GVC** that you already have. It does no | Workload | `RELEASE_NAME-sentinel` | Redis Sentinel on port 26379, tracks the Redis primary (stateful, 3 replicas) | — | | Volume set | `RELEASE_NAME-redis-vs` | Redis data, daily snapshots kept 7 days | `redis.redis.persistence.enabled: true` (default) | | Volume set | `RELEASE_NAME-sentinel-vs` | Sentinel state | `redis.sentinel.persistence.enabled: true` (default) | -| Secret | `task-runner-secrets` | Dictionary holding the bundled Redis and Sentinel passwords (name set by `secretName`) | `createSecret: true` (default) | | Secret | `RELEASE_NAME-redis-config`, `RELEASE_NAME-sentinel-config` | Redis and Sentinel configuration | — | | Identity | `RELEASE_NAME-task-runner-identity` | Identity used by the API and worker workloads | — | -| Policy | `RELEASE_NAME-task-runner-policy` | Grants that identity `reveal` on the Redis password secret and on your admin key secret, nothing else | — | +| Policy | `RELEASE_NAME-task-runner-policy` | Grants that identity `reveal` on your Redis and Sentinel credentials secrets and on your admin key secret, nothing else | — | | Identity and policy | `RELEASE_NAME-redis-identity`, `RELEASE_NAME-sentinel-identity` and matching `-policy` | Grant Redis and Sentinel `reveal` on their own configuration and password secrets | — | The policies grant only the `reveal` permission on secrets. The template's identities have no cloud access and no permission on any other Control Plane resource. ## Prerequisites -**One secret must exist before you install: the admin API key.** The `/admin/*` endpoints create, edit and delete clients and rate-limit tiers, and they are guarded by the `X-Admin-Key` header. Secrets are org-level, so no GVC flag is involved. +**Three secrets must exist before you install: the Redis password, the Sentinel password and the admin API key.** Secrets are org-level, so no GVC flag is involved. + + Two [dictionary secrets](/guides/create-secret/dictionary), each with the key `password`: one holds the bundled Redis password, the other the Sentinel password. Redis, Sentinel, the API and the worker read both. The names below are the chart defaults for `redis.redis.auth.fromSecret.name` and `redis.sentinel.auth.fromSecret.name`: + + ```bash + cpln secret create-dictionary --name my-cpln-task-runner-redis-credentials --entry password="$(openssl rand -hex 24)" + cpln secret create-dictionary --name my-cpln-task-runner-sentinel-credentials --entry password="$(openssl rand -hex 24)" + ``` + + Upgrading from 1.3.0? Do not generate new passwords; see [Upgrading From 1.3.0](#upgrading-from-1-3-0). + - An [opaque secret](/guides/create-secret/opaque) with encoding `plain` whose payload is the key itself: + The `/admin/*` endpoints create, edit and delete clients and rate-limit tiers, and they are guarded by the `X-Admin-Key` header. The key is an [opaque secret](/guides/create-secret/opaque) with encoding `plain` whose payload is the key itself: ```bash printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name SECRET_NAME --encoding plain -f - @@ -59,7 +68,7 @@ The policies grant only the `reveal` permission on secrets. The template's ident -**Create the secret before installing, or the deployment wedges silently.** An install that names a secret which does not exist still reports success, and the API workload then never starts while the other workloads come up normally. The container never runs, so `cpln logs` returns **zero lines**. The one place the reason appears is `status.versions[].message`: +**Create the secrets before installing, or the deployment wedges silently.** An install that names a secret which does not exist still reports success, and every workload that reads it never starts. The container never runs, so `cpln logs` returns **zero lines**. The one place the reason appears is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-task-runner-api --gvc GVC_NAME -o yaml @@ -70,21 +79,17 @@ The secret SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed. ``` -Use `get-deployments`. Plain `cpln workload get` has no `versions` key. Once the secret exists the workload recovers on its own after several minutes, or run `cpln workload force-redeployment RELEASE_NAME-task-runner-api --gvc GVC_NAME` to skip the wait. +Use `get-deployments`. Plain `cpln workload get` has no `versions` key. Run the same command for `RELEASE_NAME-redis`, `RELEASE_NAME-sentinel` and `RELEASE_NAME-task-runner-worker` when a credentials secret is missing. Once the secret exists the workload recovers on its own after several minutes, or run `cpln workload force-redeployment RELEASE_NAME-task-runner-api --gvc GVC_NAME` to skip the wait. -Nothing else is required. The bundled Redis and Sentinel passwords are ordinary values, since only the API and worker containers use them, but they are used exactly as written, so replace the `change-me-` defaults at install time. - ## Installation -Create the admin key secret first, then install with its name and your own Redis and Sentinel passwords: +Create the three secrets first, then install with the admin key secret's name. If you named the credentials secrets differently from the defaults, also set `redis.redis.auth.fromSecret.name` and `redis.sentinel.auth.fromSecret.name`: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/cpln-task-runner \ - --version 1.3.0 --gvc GVC_NAME \ - --set api.admin.apiKeySecretName=SECRET_NAME \ - --set redis.redisPassword='YOUR-REDIS-PASSWORD' \ - --set redis.sentinelPassword='YOUR-SENTINEL-PASSWORD' + --version 1.4.0 --gvc GVC_NAME \ + --set api.admin.apiKeySecretName=SECRET_NAME ``` For the other install methods, follow the instructions for your preferred tool: @@ -176,41 +181,29 @@ Workers fetch whatever URL a task carries. `allowPrivateUrls: false` is what kee ### Secrets and Redis -```yaml -createSecret: true # false = bring your own secret (see below) -secretName: task-runner-secrets # org-level name of the secret this chart creates +Redis and Sentinel read their passwords from the prerequisite credentials secrets: +```yaml redis: - redisPassword: change-me-cpln-task-runner-redis # used as written when createSecret is true; change it - sentinelPassword: change-me-cpln-task-runner-sentinel # used as written when createSecret is true; change it redis: auth: fromSecret: - enabled: true - name: task-runner-secrets # must equal secretName, or your own secret's name - passwordKey: redis-password + enabled: true # must stay true + name: my-cpln-task-runner-redis-credentials # prerequisite dictionary secret for Redis + passwordKey: password # key holding the Redis password persistence: - enabled: true # false = no volume set, Redis data is not persisted + enabled: true # false = no volume set, Redis data is not persisted sentinel: auth: fromSecret: - enabled: true - name: task-runner-secrets - passwordKey: redis-sentinel-password + enabled: true # must stay true + name: my-cpln-task-runner-sentinel-credentials # prerequisite dictionary secret for Sentinel + passwordKey: password # key holding the Sentinel password persistence: enabled: true ``` -The admin API key is never in this secret. It is always your own opaque secret, whatever `createSecret` is set to. - -- **`secretName` is an org-level name and is not prefixed with the release name.** A second install in the same org needs a different `secretName`, with both `fromSecret.name` values changed to match. -- **To bring your own secret,** set `createSecret: false`, create a [dictionary secret](/guides/create-secret/dictionary) holding the two keys named by the `passwordKey` fields, and point both `fromSecret.name` values at it: - -```bash -cpln secret create-dictionary --name SECRET_NAME \ - --entry redis-password='YOUR-REDIS-PASSWORD' \ - --entry redis-sentinel-password='YOUR-SENTINEL-PASSWORD' -``` +The admin API key is never in these secrets. It is always its own opaque secret, named by `api.admin.apiKeySecretName`. ## Connecting @@ -218,9 +211,9 @@ cpln secret create-dictionary --name SECRET_NAME \ |---|---|---| | API (internal, default) | `RELEASE_NAME-task-runner-api.GVC_NAME.cpln.local:8080` | Reachable from workloads in the same GVC. No credential for `/v1/*`; `X-Admin-Key` for `/admin/*` | | API (public, opt-in) | the workload's canonical endpoint | Only with `api.public.enabled: true`. Read it from `status.canonicalEndpoint` | -| Redis Sentinel | `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` | Master name `mymaster`; used by the API and worker | +| Redis Sentinel | `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` | Master name `mymaster`; used by the API and worker. Password is `password` in your Sentinel credentials secret | | Admin key | your opaque secret | `cpln secret reveal SECRET_NAME -o yaml` | -| Redis passwords | `task-runner-secrets` (the `secretName` value) | `cpln secret reveal task-runner-secrets -o yaml` | +| Redis and Sentinel passwords | your two credentials secrets | `cpln secret reveal my-cpln-task-runner-redis-credentials -o yaml` (and the same for the Sentinel secret) | Read the public endpoint, when enabled: @@ -332,6 +325,63 @@ cpln workload force-redeployment RELEASE_NAME-task-runner-api --gvc GVC_NAME Every admin caller must then send the new key. +### Rotating the Redis and Sentinel Passwords + +Redis reads its password from the secret at every start, but Sentinel keeps the passwords it first started with on its volume, so editing the secrets alone locks the API and worker out. Apply the new passwords live on every node first, then restart the API and worker. `rx` and `sx` run a command on one Redis node or Sentinel (`LOCATION` is your GVC's location; nodes and Sentinels are numbered from 0 up to `redis.redis.replicas` and `redis.sentinel.replicas` minus one). + +1. Accept the new passwords alongside the old ones on every Redis node and Sentinel: + +```bash +rx() { cpln workload exec RELEASE_NAME-redis --gvc GVC_NAME --location LOCATION \ + --container redis --replica "RELEASE_NAME-redis-$1" -- "${@:2}"; } +sx() { cpln workload exec RELEASE_NAME-sentinel --gvc GVC_NAME --location LOCATION \ + --container sentinel --replica "RELEASE_NAME-sentinel-$1" -- "${@:2}"; } +# The loops cover the default 3 Redis nodes and 3 Sentinels; list one number per node if you changed the counts +# On every Redis node: accept the new Redis password alongside the old one, and replicate with it +for n in 0 1 2; do + rx $n redis-cli --no-auth-warning -a OLD_REDIS_PASSWORD ACL SETUSER default ">NEW_REDIS_PASSWORD" + rx $n redis-cli --no-auth-warning -a OLD_REDIS_PASSWORD CONFIG SET masterauth NEW_REDIS_PASSWORD +done +# On every Sentinel: monitor Redis with the new Redis password, and accept the new Sentinel password alongside the old one +for n in 0 1 2; do + sx $n redis-cli -p 26379 --no-auth-warning -a OLD_SENTINEL_PASSWORD SENTINEL SET mymaster auth-pass NEW_REDIS_PASSWORD + sx $n redis-cli -p 26379 --no-auth-warning -a OLD_SENTINEL_PASSWORD ACL SETUSER default ">NEW_SENTINEL_PASSWORD" +done +``` + +2. Put the new passwords in the two secrets (key `password`). For each secret, export it, change only `password`, apply it and read it back: + +```bash +cpln secret reveal my-cpln-task-runner-redis-credentials -o yaml > secret.yaml +# edit only the password key in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal my-cpln-task-runner-redis-credentials -o yaml +``` + +3. Restart the API and worker, which keep the old passwords until they restart: + +```bash +cpln workload force-redeployment RELEASE_NAME-task-runner-api --gvc GVC_NAME +cpln workload force-redeployment RELEASE_NAME-task-runner-worker --gvc GVC_NAME +``` + +4. Once `get-deployments` shows both at a higher version, alone and ready, retire the old passwords and check that the Sentinels can still fail over: + +```bash +for n in 0 1 2; do rx $n redis-cli --no-auth-warning -a NEW_REDIS_PASSWORD ACL SETUSER default " -The bundled Redis and Sentinel passwords keep whatever you already set. Only their defaults changed. +Then follow [Upgrading From 1.3.0](#upgrading-from-1-3-0): the Redis and Sentinel passwords your 1.2.x release runs with (`mypassword` if never changed) go into the two credentials secrets. + +### Upgrading From 1.3.0 + +1.4.0 no longer takes the Redis and Sentinel passwords as values, and no longer creates the `task-runner-secrets` secret. Use the CURRENT passwords: Redis and Sentinel keep the ones they started with, so new ones would lock the API and worker out. To change them, upgrade first, then follow [Rotating the Redis and Sentinel Passwords](#rotating-the-redis-and-sentinel-passwords). + +**With `createSecret: true` (the default)**, the upgrade **deletes** `task-runner-secrets` (or the secret named by your `secretName`, if you changed it), so copy the passwords into two new secrets **before** upgrading: + + + + `redis-password` is the Redis password, `redis-sentinel-password` the Sentinel password. Use the name you set in `secretName` if you changed it: + + ```bash + cpln secret reveal task-runner-secrets -o yaml + ``` + + + ```bash + cpln secret create-dictionary --name my-cpln-task-runner-redis-credentials --entry password='YOUR-CURRENT-REDIS-PASSWORD' + cpln secret create-dictionary --name my-cpln-task-runner-sentinel-credentials --entry password='YOUR-CURRENT-SENTINEL-PASSWORD' + ``` + + + Remove `createSecret`, `secretName`, `redis.redisPassword` and `redis.sentinelPassword`, and point `redis.redis.auth.fromSecret.name` and `redis.sentinel.auth.fromSecret.name` at the new secrets (the defaults above). The chart refuses to render while any of the removed keys is still set, and refuses the name `task-runner-secrets`; do not reuse a custom `secretName` either, because the upgrade deletes that secret too. + + + ```bash + cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/cpln-task-runner \ + --version 1.4.0 --gvc GVC_NAME \ + --set api.admin.apiKeySecretName=SECRET_NAME + ``` + + + +**With `createSecret: false`** and a secret of your own, keep it: remove the same four keys, keep both `fromSecret.name` values pointing at your secret, and set both `passwordKey` values to its key names explicitly (`redis-password` and `redis-sentinel-password` if you kept the 1.3.0 defaults; 1.4.0 defaults both to `password`). Nothing is deleted. ### Public Access and the Open Queue @@ -387,7 +471,7 @@ The API and worker are serverless workloads that autoscale on concurrent request - **Cause:** the secret named by `api.admin.apiKeySecretName` does not exist. `get-deployments` shows `The secret SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` + **Cause:** a prerequisite secret does not exist: the one named by `api.admin.apiKeySecretName`, or a Redis or Sentinel credentials secret. `get-deployments` shows `The secret SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` **Fix:** create the secret as in [Prerequisites](#prerequisites). The workload recovers on its own, or run `cpln workload force-redeployment RELEASE_NAME-task-runner-api --gvc GVC_NAME`. @@ -401,6 +485,11 @@ The API and worker are serverless workloads that autoscale on concurrent request **Fix:** follow [Upgrading From 1.2.x](#upgrading-from-1-2-x). + + **Cause:** your values still carry `createSecret`, `secretName`, `redis.redisPassword` or `redis.sentinelPassword`. + + **Fix:** follow [Upgrading From 1.3.0](#upgrading-from-1-3-0). + **Cause:** the `X-Admin-Key` header is missing or does not match the secret payload, for example because the key was created with `echo` and carries a trailing newline, or the secret was rotated without a redeployment. @@ -415,10 +504,9 @@ The API and worker are serverless workloads that autoscale on concurrent request ## Important Notes -- **Create the admin key secret before installing.** A missing secret wedges the API with no log output; the reason is in `status.versions[].message`. +- **Create the admin key secret and the Redis and Sentinel credentials secrets before installing.** A missing secret wedges the workloads that read it with no log output; the reason is in `status.versions[].message` from `cpln workload get-deployments`. - **Leave `api.public.enabled` at `false` unless an authenticating proxy fronts the API.** `/v1/enqueue` is unauthenticated and auto-registers unknown client IDs. -- **Change the `change-me-` Redis and Sentinel passwords at install time.** Once the volumes are initialised, changing them requires uninstalling (which deletes the volume sets) and reinstalling. -- **`secretName` is org-wide.** Give each install in the same org its own `secretName` and matching `fromSecret.name` values. +- **Never change the Redis or Sentinel password by editing the secrets alone.** Sentinel keeps the passwords it first started with on its volume, so that locks the API and worker out. Follow [Rotating the Redis and Sentinel Passwords](#rotating-the-redis-and-sentinel-passwords). - **Force a redeployment after rotating the admin key.** A running replica keeps the old key. ## External References diff --git a/template-catalog/templates/docmost.mdx b/template-catalog/templates/docmost.mdx index 0623be31..0f332e6c 100644 --- a/template-catalog/templates/docmost.mdx +++ b/template-catalog/templates/docmost.mdx @@ -1,7 +1,7 @@ --- title: Docmost icon: /template-catalog/templates/icons/docmost.png -description: Deploy Docmost, the open-source wiki and knowledge base with real-time collaborative editing, on Control Plane. Single server with bundled PostgreSQL and Redis, optional multi-replica on S3 attachment storage. Needs an existing GVC and a pre-created APP_SECRET opaque secret. +description: Deploy Docmost, the open-source wiki and knowledge base with real-time collaborative editing, on Control Plane. Single server with bundled PostgreSQL and Redis, optional multi-replica on S3 attachment storage. Needs an existing GVC, an APP_SECRET secret and database and Redis credential secrets. keywords: ['docmost', 'wiki', 'knowledge base', 'confluence alternative', 'notion alternative', 'outline alternative', 'bookstack alternative', 'collaborative editing', 'team documentation', 'spaces and permissions', 'self-hosted wiki', 'agpl'] --- @@ -24,18 +24,16 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Volume set | `RELEASE_NAME-docmost-storage` | Local attachment storage, mounted at `/app/data/storage` only when `storage.type` is `local` | — | | Volume set | `RELEASE_NAME-docmost-redis-vs` | Redis append-only file at `/data` | — | | Volume set | `RELEASE_NAME-pg-vs` | PostgreSQL data directory; takes no scheduled snapshots; a final snapshot is taken when the volume set is deleted and kept 7 days | — | -| Secret | `RELEASE_NAME-docmost-creds` | Dictionary secret the chart creates with the PostgreSQL and Redis credentials Docmost uses to assemble `DATABASE_URL` and `REDIS_URL` | — | -| Secret | the name in `postgres.config.credentialsSecretName` | Dictionary secret the chart creates (`username`, `password`, `database`) and hands to the bundled PostgreSQL | — | | Identity | `RELEASE_NAME-docmost-identity` | Bound to the Docmost and Redis workloads; carries your AWS cloud account link in keyless S3 mode | — | -| Policy | `RELEASE_NAME-docmost-policy` | Grants that identity `reveal` on exactly the secrets it reads: the credentials secret, your `APP_SECRET` secret, and your S3-key and SMTP secrets when configured | — | +| Policy | `RELEASE_NAME-docmost-policy` | Grants that identity `reveal` on exactly the secrets it reads: your database credentials, Redis password and `APP_SECRET` secrets, and your S3-key and SMTP secrets when configured | — | | Identity | `RELEASE_NAME-pg-identity` | Bound to the bundled PostgreSQL (and its backup job) | — | | Policy | `RELEASE_NAME-pg-policy` | Grants the PostgreSQL identity `reveal` on its credentials secret only | — | -Your `APP_SECRET` secret is **not** created by the template — see [Prerequisites](#prerequisites). +Your `APP_SECRET`, database credentials and Redis password secrets are **not** created by the template — see [Prerequisites](#prerequisites). ## Prerequisites -Docmost signs sessions, invitation links, and share links with an `APP_SECRET` that you supply through an [opaque secret](/guides/create-secret/opaque) created **before** installing. The value never passes through Helm values. +Three secrets must exist **before** installing: the `APP_SECRET` Docmost signs sessions, invitation links, and share links with, supplied through an [opaque secret](/guides/create-secret/opaque), and two [dictionary secrets](/guides/create-secret/dictionary) holding the database credentials and the Redis password. The values never pass through Helm values. @@ -50,16 +48,38 @@ Docmost signs sessions, invitation links, and share links with an `APP_SECRET` t `APP_SECRET` is effectively write-once: rotating it logs out every user and invalidates every outstanding invitation and share link (stored documents are unaffected). Keep a copy somewhere safe. + + The bundled PostgreSQL initializes with these values and Docmost connects with them: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=docmost \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=docmost + ``` + + Set `postgres.config.credentialsSecretName` to this name. Secret names are org-wide, so give each Docmost release its own secret. The three values are part of Docmost's connection URL, so use letters, digits, `-` and `_` only. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password). + + + One key, `password`; the bundled Redis requires it and Docmost connects with it: + + ```bash + cpln secret create-dictionary --name REDIS_SECRET_NAME \ + --entry password="$(openssl rand -hex 24)" + ``` + + Set `redis.credentialsSecretName` to this name. The password is part of Docmost's Redis URL, so use letters, digits, `-` and `_` only. + -Create the secret **before** installing. Installing without it reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message`: +Create all three secrets **before** installing. Installing without one reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-docmost --gvc GVC_NAME -o yaml ``` -The message reads `The secret APP_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` Creating the secret lets the deployment recover on its own; `cpln workload force-redeployment RELEASE_NAME-docmost --gvc GVC_NAME` skips the wait. +The message reads `The secret APP_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` The message names whichever secret is missing. Creating the secret lets the deployment recover on its own; `cpln workload force-redeployment RELEASE_NAME-docmost --gvc GVC_NAME` skips the wait. Everything else works with the defaults. Three optional features need their own setup first: @@ -70,18 +90,16 @@ Everything else works with the defaults. Three optional features need their own ## Installation -Once your `APP_SECRET` secret exists, install from the marketplace registry, pointing `secrets.name` at it and replacing both bundled-component passwords: +Once your three secrets exist, install from the marketplace registry, pointing the three secret-name keys at them: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/docmost \ - --version 1.1.0 --gvc GVC_NAME \ + --version 1.2.0 --gvc GVC_NAME \ --set secrets.name=APP_SECRET_NAME \ - --set postgres.credentials.password='YOUR-DB-PASSWORD' \ - --set redis.auth.password='YOUR-REDIS-PASSWORD' + --set postgres.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME \ + --set redis.credentialsSecretName=REDIS_SECRET_NAME ``` -Installing a second Docmost release in the same org? Also set `postgres.config.credentialsSecretName` to a name no other release uses — secret names are org-wide. - Other ways to install and manage the template: @@ -196,17 +214,13 @@ A change to either access knob takes a couple of minutes to take effect after th ### Bundled PostgreSQL -The database is the [postgres](/template-catalog/templates/postgres) template as a subchart, so every knob of that template is available under `postgres.*`. The chart builds the credentials secret from `postgres.credentials.*` for you — nothing to create before installing. +The database is the [postgres](/template-catalog/templates/postgres) template as a subchart, so every knob of that template is available under `postgres.*`. The database credentials come from your prerequisite dictionary secret (`username`, `password` and `database` — see [Prerequisites](#prerequisites)), named by `postgres.config.credentialsSecretName`. ```yaml postgres: image: postgres:18 - credentials: # the chart writes these into the secret named below - username: docmost - password: change-me-docmost-pg # change before installing — seeds the database on first boot - database: docmost config: - credentialsSecretName: my-docmost-db-credentials # chart-created; secret names are org-wide, so give each release its own + credentialsSecretName: my-docmost-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 250m maxCpu: 1000m @@ -227,8 +241,7 @@ Redis is required: Docmost's readiness check pings it alongside the database, an ```yaml redis: image: redis:8 - auth: - password: change-me-docmost-redis # change before installing — wired into REDIS_URL + credentialsSecretName: my-docmost-redis # PREREQUISITE dictionary secret (key: password; letters, digits, - and _) — must exist BEFORE install resources: minCpu: 100m maxCpu: 400m @@ -239,7 +252,7 @@ redis: ``` -Both bundled passwords seed their component on first boot and are not updated by later value edits. Uninstalling the release deletes the volume sets; reinstall to reset. +PostgreSQL reads its password from the secret only when its data is first initialized; Redis reads its password each time it starts. To change either, follow [Rotating the Database Password](#rotating-the-database-password) or [Rotating the Redis Password](#rotating-the-redis-password). Uninstalling the release deletes the volume sets; reinstall to reset. ## Connecting @@ -252,7 +265,7 @@ Both bundled passwords seed their component on first boot and are not updated by | Internal (same GVC) | `http://RELEASE_NAME-docmost.GVC_NAME.cpln.local:3000` | Reachable per `internalAccess.type`. | | Login | The account created on first run | Docmost ships no default account. | | PostgreSQL | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | `username` / `password` / `database` from `cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is your `postgres.config.credentialsSecretName`. | -| Redis | `RELEASE_NAME-docmost-redis.GVC_NAME.cpln.local:6379` | Password is the `redisPassword` key of `cpln secret reveal RELEASE_NAME-docmost-creds -o yaml`. | +| Redis | `RELEASE_NAME-docmost-redis.GVC_NAME.cpln.local:6379` | Password is the `password` key of `cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is your `redis.credentialsSecretName`. | With `publicAccess.enabled: false` there is no public endpoint; reach the UI from your machine through a tunnel instead and open `http://localhost:3000`: @@ -482,17 +495,76 @@ Two constraints make this a planned operation: the archive contains `CREATE ROLE This restore path has not been exercised against this template end to end. Rehearse it on a throwaway release before you depend on it. Restoring from a volume set snapshot is likewise unverified here. +### Rotating the Database Password + +PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with, so a rotation takes three steps in order: + +1. Change it in the database: `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` (`DB_USERNAME` is the secret's `username`; connect as that user from inside the GVC). +2. Update the secret — export it, change only `password`, and apply it: + + ```bash + cpln secret reveal SECRET_NAME -o yaml > secret.yaml + # edit only the password key in secret.yaml + cpln apply -f secret.yaml + ``` + +3. Right away, redeploy Docmost, and wait until `get-deployments` shows a higher version number, alone and ready (`ready: true` alone can still describe the old replica): + + ```bash + cpln workload force-redeployment RELEASE_NAME-docmost --gvc GVC_NAME + ``` + + Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. + +Connections the app already has open keep working after `ALTER ROLE`; new ones fail until its redeploy finishes, so redeploy right away and rotate in a quiet window. + +### Rotating the Redis Password + +Redis keeps no password on disk: it takes the password from the secret each time it starts, and so does Docmost. + +1. Update the secret — export it, change only `password`, apply it, and check it: + + ```bash + cpln secret reveal SECRET_NAME -o yaml > secret.yaml + # edit only the password key in secret.yaml + cpln apply -f secret.yaml + cpln secret reveal SECRET_NAME -o yaml + ``` + +2. Redeploy Redis, then Docmost, and wait until `get-deployments` shows a higher version number for each, alone and ready: + + ```bash + cpln workload force-redeployment RELEASE_NAME-docmost-redis --gvc GVC_NAME + cpln workload force-redeployment RELEASE_NAME-docmost --gvc GVC_NAME + ``` + +Docmost cannot reach Redis between the two redeploys, so expect a short interruption and rotate in a quiet window. + ### Upgrading from 1.0.0 -Version 1.1.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. Docmost absorbed that change rather than passing it on — the chart now creates the credentials secret itself — so there is **no new prerequisite**, only a rename of three keys: +`postgres.config.username`, `postgres.config.password` and `postgres.config.database` were removed. Put the values your database was initialized with into the database credentials secret under [Prerequisites](#prerequisites) (keys `username`, `password` and `database`), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. Then move the Redis password as described in [Upgrading from 1.1.0](#upgrading-from-1-1-0) (1.0.0 kept it in the same `RELEASE_NAME-docmost-creds` secret). -| Removed key (1.0.0) | Replacement (1.1.0) | -|---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | +### Upgrading from 1.1.0 -Move the three keys, keep the same values so the running database still matches, and upgrade in place. A values file that still carries an old key fails the render with the **postgres template's** message, which tells you to create a dictionary secret yourself — ignore that advice here; this template creates it. +`postgres.credentials` was removed: the database credentials are now a prerequisite secret, the same one the `postgres` template requires on its own. `redis.auth.password` was removed too: the Redis password is now a prerequisite secret named by `redis.credentialsSecretName`. Carrying either old key forward fails the render. + +1.1.0 created both secrets itself, so **this upgrade deletes them**. Before upgrading, create secrets under **new** names holding the values your release uses today — not new ones, or Docmost loses its database and Redis connections: + +```bash +cpln secret reveal my-docmost-db-credentials -o yaml # default name: use your credentialsSecretName if you changed it +cpln secret create-dictionary --name RELEASE_NAME-docmost-db \ + --entry username='YOUR_CURRENT_USERNAME' \ + --entry password='YOUR_CURRENT_PASSWORD' \ + --entry database='YOUR_CURRENT_DATABASE' + +cpln secret reveal RELEASE_NAME-docmost-creds -o yaml # redisPassword is the CURRENT Redis password +cpln secret create-dictionary --name RELEASE_NAME-docmost-redis \ + --entry password='YOUR_CURRENT_REDIS_PASSWORD' +``` + +The current values are also in `postgres.credentials` and `redis.auth.password` of the 1.1.0 release. Then upgrade with `postgres.config.credentialsSecretName` and `redis.credentialsSecretName` set to the new names, and with `postgres.credentials` and `redis.auth` removed from your values. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Docmost briefly loses its database; it reconnects by itself. The new version of a workload can also wait several minutes with `is not allowed to reveal the secret` in `get-deployments` while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. The first `helm upgrade` after an install can restart the bundled PostgreSQL even when its values are unchanged, so expect a brief interruption. @@ -526,17 +598,10 @@ The bundled PostgreSQL and Redis are single-instance by design: Docmost has no S **Symptom:** `cpln helm install` reported success, `cpln logs` returns nothing, and `cpln workload get-deployments RELEASE_NAME-docmost --gvc GVC_NAME -o yaml` shows `The secret APP_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` under `status.versions[].message`. - **Cause:** The `APP_SECRET` prerequisite secret named by `secrets.name` does not exist. + **Cause:** A prerequisite secret does not exist — the `APP_SECRET` secret named by `secrets.name`, the database credentials secret named by `postgres.config.credentialsSecretName`, or the Redis password secret named by `redis.credentialsSecretName`. The message names the missing one. **Fix:** Create it (see [Prerequisites](#prerequisites)). The deployment recovers on its own once the secret exists; `cpln workload force-redeployment RELEASE_NAME-docmost --gvc GVC_NAME` skips the wait. - - **Symptom:** Installing a second Docmost release fails with `cannot be updated because it is being managed by a different release` and creates nothing. - - **Cause:** Both releases use the same `postgres.config.credentialsSecretName`. Secret names are org-wide, and the first release owns that one. Nothing is shared, overwritten, or deleted. - - **Fix:** Set `postgres.config.credentialsSecretName` to a distinct name for the second release and install again. - **Symptom:** `docmost: docmost.replicas > 1 requires storage.type: s3 — local attachments live on per-replica volumesets and would scatter/404 across replicas`. @@ -556,7 +621,7 @@ The bundled PostgreSQL and Redis are single-instance by design: Docmost has no S **Cause:** Your values file still carries `postgres.config.username`, `postgres.config.password`, or `postgres.config.database`. - **Fix:** Rename them to `postgres.credentials.*` as described in [Upgrading from 1.0.0](#upgrading-from-1-0-0). Do not create a secret yourself — this template creates it. + **Fix:** Put the three values into the database credentials secret and remove the keys, as described in [Upgrading from 1.0.0](#upgrading-from-1-0-0). **Symptom:** An invitation created in the UI reaches no one, and the workload logs contain no invitation link. @@ -576,16 +641,16 @@ The bundled PostgreSQL and Redis are single-instance by design: Docmost has no S ## Important Notes -- **Create the `APP_SECRET` secret before installing** — `secrets.name` must name an existing opaque secret (plain encoding) with a random payload of at least 32 characters. A missing secret wedges the deployment silently. +- **Create the three prerequisite secrets before installing** — `secrets.name` must name an existing opaque secret (plain encoding) with a random payload of at least 32 characters, and `postgres.config.credentialsSecretName` and `redis.credentialsSecretName` existing dictionary secrets. A missing secret wedges the deployment silently; `status.versions[].message` from `cpln workload get-deployments RELEASE_NAME-docmost --gvc GVC_NAME -o yaml` names it. - **Treat `APP_SECRET` as write-once** — rotating it logs out every user and invalidates outstanding invitation and share links. Back the value up outside Control Plane. - **Claim the workspace right after install** — the first visitor to complete the setup form becomes the workspace owner, and the form is reachable by anyone who can reach the endpoint. - **With SMTP off, member invitations cannot be delivered** — configure `smtp.*` before inviting anyone. - **`docmost.replicas` above `1` requires `storage.type: s3`** — local attachments are per-replica. - **AWS S3 is keyless only** — a cloud account plus a bucket-scoped IAM policy; static keys are accepted only with an S3-compatible `storage.s3.endpoint`. - **Uploads larger than `storage.fileUploadSizeLimit` are truncated, not rejected** — raise the limit before importing large attachments. -- **Change `postgres.credentials.password` and `redis.auth.password` before installing** — both seed their component on first boot and are not updated by later value edits. -- **Give each release its own `postgres.config.credentialsSecretName`** — secret names are org-wide, and a second release on the same name is refused at install. -- **Uninstalling deletes the volume sets** and every page and local attachment in them; your `APP_SECRET` secret is yours and survives. +- **Rotating a password needs a forced redeployment** — the database password also needs `ALTER ROLE` first; see [Rotating the Database Password](#rotating-the-database-password) and [Rotating the Redis Password](#rotating-the-redis-password). +- **Give each release its own database credentials secret** — secret names are org-wide. +- **Uninstalling deletes the volume sets** and every page and local attachment in them; your prerequisite secrets are yours and survive. - **Docmost is licensed under the AGPL** — this template deploys the open-source edition. ## External References diff --git a/template-catalog/templates/fusionauth.mdx b/template-catalog/templates/fusionauth.mdx index e3dc683f..5be562be 100644 --- a/template-catalog/templates/fusionauth.mdx +++ b/template-catalog/templates/fusionauth.mdx @@ -1,13 +1,13 @@ --- title: FusionAuth icon: /template-catalog/templates/icons/fusionauth.png -description: Self-hosted FusionAuth identity provider (OAuth2, OIDC, SAML, hosted login pages) with a bundled single-instance PostgreSQL 18 and optional scheduled S3 or GCS backups. Public by default. Deploys into an existing GVC; the only install-time requirement is choosing the bundled database password. +description: Self-hosted FusionAuth identity provider (OAuth2, OIDC, SAML, hosted login pages) with a bundled single-instance PostgreSQL 18 and optional scheduled S3 or GCS backups. Public by default. Deploys into an existing GVC; needs a dictionary secret holding the database credentials. keywords: ['auth server', 'sso', 'jwt', 'login page', 'auth0 alternative', 'keycloak alternative', 'okta alternative', 'mfa', 'user database', 'idp', 'setup wizard', 'tenant issuer', 'pg_dumpall'] --- ## Overview -FusionAuth is a self-hosted identity and access management platform: a user store with hosted login and registration pages, OAuth2 and OpenID Connect token issuance, SAML, MFA, and federation to external identity providers, managed through a browser admin UI. This template deploys one FusionAuth node together with a bundled PostgreSQL database that it creates, wires up and waits for — there is no database or secret to prepare before installing. +FusionAuth is a self-hosted identity and access management platform: a user store with hosted login and registration pages, OAuth2 and OpenID Connect token issuance, SAML, MFA, and federation to external identity providers, managed through a browser admin UI. This template deploys one FusionAuth node together with a bundled PostgreSQL database that it creates, wires up and waits for — the only thing to prepare before installing is the secret holding the database credentials. The FusionAuth workload is publicly reachable by default so that applications can redirect users to it, and it keeps no state of its own: every user, tenant and application lives in the bundled database. @@ -22,32 +22,51 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-fusionauth` | FusionAuth app, login pages and admin UI (serverless, HTTP `9011`) | — | | Workload | `RELEASE_NAME-postgres` | Bundled PostgreSQL 18 server (stateful, 1 replica, TCP `5432`) | — | | Volume Set | `RELEASE_NAME-pg-vs` | PostgreSQL data (`ext4`, 10 GiB; a final snapshot is kept for 7 days when it is deleted) | — | -| Secret | the name in `postgres.config.credentialsSecretName` | Dictionary secret the chart creates from `postgres.credentials.*` — `username`, `password`, `database` | — | | Secret | `RELEASE_NAME-fusionauth-startup` | Opaque start script that waits for PostgreSQL before launching FusionAuth | — | | Identity | `RELEASE_NAME-fusionauth-identity` | Identity of the FusionAuth workload | — | -| Policy | `RELEASE_NAME-fusionauth-policy` | `reveal` on the two secrets above, nothing else | — | +| Policy | `RELEASE_NAME-fusionauth-policy` | `reveal` on the startup secret and your database credentials secret, nothing else | — | | Identity | `RELEASE_NAME-pg-identity` | Identity of the PostgreSQL and backup workloads; carries the cloud account binding when backups are enabled | — | | Policy | `RELEASE_NAME-pg-policy` | `reveal` on the database credentials secret | — | | Workload | `RELEASE_NAME-postgres-backup` | Scheduled `pg_dumpall` of the database to S3 or GCS (cron) | `postgres.backup.enabled: true` | ## Prerequisites -None for a default install. The bundled database's credentials secret is created by the chart from your values, so there is nothing to create beforehand. +**One [dictionary secret](/guides/create-secret/dictionary) must exist before you install** — the database credentials. The bundled PostgreSQL initializes with them and FusionAuth connects with them, and the password never passes through Helm values: + +```bash +cpln secret create-dictionary --name SECRET_NAME \ + --entry username=fusionauth \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=fusionauth +``` + +Set `postgres.config.credentialsSecretName` to that name. Secret names are org-wide, so give each FusionAuth release its own secret. + +- **`database` must be URL-safe** (letters, digits, `_`): FusionAuth's JDBC URL is assembled from it at container start. The snippet's hex password is URL-safe too. +- **PostgreSQL reads the secret only when its data directory is first initialized** — to change the password later, see [Rotating the Database Password](#rotating-the-database-password). + + +**If the secret does not exist at install time, the deployment wedges silently.** `cpln logs` returns zero lines — the containers never start. The only diagnostic is `status.versions[].message` (plain `cpln workload get` has no `versions` key): + +```bash +cpln workload get-deployments RELEASE_NAME-fusionauth --gvc GVC_NAME -o yaml +``` + +Create the missing secret and the deployment recovers on its own after several minutes, or skip the wait with `cpln workload force-redeployment RELEASE_NAME-fusionauth --gvc GVC_NAME`. + Backups are optional and need a bucket plus a Control Plane [cloud account](/guides/create-cloud-account) — see [Backing Up](#backing-up). ## Installation -Install the chart from the marketplace registry, choosing the bundled database password — it is used exactly as given and is baked into the database the first time it starts: +Once the database credentials secret exists, install the chart from the marketplace registry with its name set: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/fusionauth \ - --version 2.4.1 --gvc GVC_NAME \ - --set postgres.credentials.password='YOUR-STRONG-PASSWORD' + --version 2.5.0 --gvc GVC_NAME \ + --set postgres.config.credentialsSecretName=SECRET_NAME ``` -Secret names are org-wide, so when you install a **second** FusionAuth release in the same org also set `--set postgres.config.credentialsSecretName=` to a name of your own; see [Configuration](#bundled-postgresql). - Or follow the instructions for your preferred method: @@ -117,17 +136,13 @@ A firewall change can take a few minutes to take effect — re-test before concl ### Bundled PostgreSQL -The database is the [PostgreSQL](/template-catalog/templates/postgres) template deployed as a dependency. Unlike a standalone PostgreSQL install, **this chart creates the credentials secret for you** from the three values under `postgres.credentials` and hands the database its name, so the keys below are values rather than a secret you create. +The database is the [PostgreSQL](/template-catalog/templates/postgres) template deployed as a dependency. It reads the credentials secret you created under [Prerequisites](#prerequisites), named by `postgres.config.credentialsSecretName`. ```yaml postgres: image: postgres:18 - credentials: # the chart writes these three into the secret named below — nothing for you to create - username: username - password: change-me-fusionauth-db # used EXACTLY as given — change it before installing - database: test config: - credentialsSecretName: my-fusionauth-db-credentials # secret names are org-wide — give each release its own + credentialsSecretName: my-fusionauth-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 200m minMemory: 128Mi @@ -148,8 +163,7 @@ postgres: #- //gvc/GVC_NAME/workload/WORKLOAD_NAME ``` -- `postgres.credentials.*` — applied only when the volume is first initialized. Changing them afterwards changes the secret but not the database; see [Rotating the Database Password](#rotating-the-database-password). `username` and `database` ship as `username` and `test` and are deliberately left alone across upgrades because an existing data directory was initialized with them; `database` also forms the JDBC URL FusionAuth connects with. -- `postgres.config.credentialsSecretName` — the name of the dictionary secret the chart creates. Secret names are org-wide, so a second FusionAuth release left on the default is refused at install (the secret is owned by the first release) and creates nothing — the first release is untouched. +- `postgres.config.credentialsSecretName` — the name of your prerequisite dictionary secret (`username`, `password`, `database`). Its values are applied only when the volume is first initialized; changing the secret afterwards does not change the database — see [Rotating the Database Password](#rotating-the-database-password). `database` also forms the JDBC URL FusionAuth connects with. - `postgres.internalAccess.type` — which workloads can reach port `5432`. The default `same-gvc` is what the FusionAuth workload needs; `none` would cut it off from its own database. - `postgres.resources` — on a stateful workload `maxCpu` may not exceed four times `minCpu`; raising one without the other is rejected when the workload is applied. @@ -159,7 +173,7 @@ There is no high-availability database option in this template: the dependency i ### PostgreSQL Backup -A cron workload runs `pg_dumpall` on the schedule and uploads a gzip-compressed SQL dump of the database to your bucket, authenticating with the `username` and `password` the chart wrote into the credentials secret. Provider setup steps are under [Backing Up](#backing-up). +A cron workload runs `pg_dumpall` on the schedule and uploads a gzip-compressed SQL dump of the database to your bucket, authenticating with the `username` and `password` from your credentials secret. Provider setup steps are under [Backing Up](#backing-up). ```yaml postgres: @@ -354,7 +368,7 @@ cpln port-forward RELEASE_NAME-postgres 5432:5432 --gvc GVC_NAME ### Rotating the Database Password -The database reads `postgres.credentials.password` only when its volume is first initialized, and FusionAuth resolves the secret only when a replica starts. So rotate in three moves: change it inside PostgreSQL, change the value so the chart rewrites the secret, then restart FusionAuth. +The database reads the password from your credentials secret only when its volume is first initialized, and FusionAuth resolves the secret only when a replica starts. So rotate in three moves: change it inside PostgreSQL, update the secret, then right away restart FusionAuth. @@ -364,11 +378,17 @@ The database reads `postgres.credentials.password` only when its volume is first ALTER ROLE USERNAME WITH PASSWORD 'NEW-STRONG-PASSWORD'; ``` - - Set `postgres.credentials.password` to the new password in your values and run `cpln helm upgrade` with your usual arguments. The chart rewrites the credentials secret; nothing else changes. + + Export it, change only `password`, and apply it: + + ```bash + cpln secret reveal SECRET_NAME -o yaml > secret.yaml + # edit only the password key in secret.yaml + cpln apply -f secret.yaml + ``` - A running replica keeps the password it started with, so force a redeployment: + A running replica keeps the password it started with, so force a redeployment right away: ```bash cpln workload force-redeployment RELEASE_NAME-fusionauth --gvc GVC_NAME @@ -376,35 +396,30 @@ The database reads `postgres.credentials.password` only when its volume is first +Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. Connections FusionAuth already has open keep working after `ALTER ROLE`; new ones fail until its redeploy finishes, so redeploy right away and rotate in a quiet window. + ### Upgrading From 2.3.x -The bundled PostgreSQL moved to the `postgres` 3.4.1 template, which no longer takes database credentials as values. This template absorbed that change rather than passing it on — **there is no new prerequisite**, only a rename: +`postgres.config.username`, `postgres.config.password` and `postgres.config.database` were removed. A values file that still carries an old key fails the render. Put the values your database was initialized with into the database credentials secret under [Prerequisites](#prerequisites) (keys `username`, `password` and `database`), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. -| Removed key | Replacement | -|---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | +### Upgrading From 2.4.0 -Carry the **same three values** across — the existing database was initialized with them and the rename does not change it. A values file that still carries an old key fails the render with the PostgreSQL template's message telling you to create a dictionary secret yourself. Ignore that advice here: this template creates the secret. Move the three keys and upgrade. +`postgres.credentials` was removed in 2.5.0: the database credentials are now a prerequisite secret, the same one the `postgres` template requires on its own. Carrying the old key forward fails the render. This applies to upgrades from 2.4.0 and 2.4.1. -### Upgrading From 2.4.0 +2.4.x created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or FusionAuth loses its database connection: -One thing changed: the default `postgres.credentials.password` is `change-me-fusionauth-db` instead of `password`. The database initialized its data directory with whatever password was in force at first boot, and a new default does not change it, so: +```bash +cpln secret create-dictionary --name RELEASE_NAME-fusionauth-db \ + --entry username=username \ + --entry password='YOUR_CURRENT_PASSWORD' \ + --entry database=test +``` -- **If you already set your own password**, nothing changes — upgrade normally. -- **If you never overrode it**, your database's password is literally `password`. Do one of these before upgrading, or FusionAuth fails to authenticate after the upgrade: +(`username` and `test` were the 2.4.x defaults; use your own values if you overrode them.) Your current values are in `postgres.credentials` of the installed release, and also in the old secret (`cpln secret reveal my-fusionauth-db-credentials -o yaml`, or the name your release set in `credentialsSecretName`) until the upgrade deletes it. **If you installed 2.4.0 and never overrode the password, it is literally `password`** — put that in the new secret, and rotate it afterwards as described in [Rotating the Database Password](#rotating-the-database-password). - - - Set `postgres.credentials.password: password` explicitly in your values. The upgrade is then a no-op for the database. - - - `password` was a published default, so treat it as compromised. Follow [Rotating the Database Password](#rotating-the-database-password), using the new password as the value you upgrade with. - - +Then upgrade with `postgres.config.credentialsSecretName` set to the new name and `postgres.credentials` removed. -`postgres.credentials.username` and `postgres.credentials.database` are deliberately unchanged for the same reason — they are baked into the initialized data directory, and `database` also forms the JDBC URL. +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so FusionAuth can briefly lose its database; it reconnects by itself. The new version of a workload can also wait several minutes with `is not allowed to reveal the secret` in `get-deployments` while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. ### Scaling and Availability @@ -428,30 +443,31 @@ One thing changed: the default `postgres.credentials.password` is `change-me-fus **Fix:** usually none — wait for `Starting FusionAuth...`. If the script instead exits with `PostgreSQL did not become ready after 300s. Exiting.`, the database itself is the problem: check `cpln workload get-deployments RELEASE_NAME-postgres --gvc GVC_NAME -o yaml` and its logs with `cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres"}' --limit 50 --since 10m`. - - **Symptom:** installing a second FusionAuth release in the same org fails with: + + **Symptom:** `RELEASE_NAME-fusionauth` and `RELEASE_NAME-postgres` never reach ready, and `cpln logs` returns zero lines. - ```text - cannot be updated because it is being managed by a different release - ``` + **Cause:** the database credentials secret named by `postgres.config.credentialsSecretName` does not exist. The containers never start, so there is nothing to log. - **Cause:** both releases render the credentials secret under the default name `my-fusionauth-db-credentials`, and secret names are org-wide. The second install is refused and creates nothing; the first release is untouched. + **Fix:** read `status.versions[].message`, which names the missing secret, then create it as in [Prerequisites](#prerequisites). The deployment recovers on its own after several minutes, or force a redeployment to skip the wait. - **Fix:** install the second release with `--set postgres.config.credentialsSecretName=` set to a name of its own. + ```bash + cpln workload get-deployments RELEASE_NAME-fusionauth --gvc GVC_NAME -o yaml + cpln workload force-redeployment RELEASE_NAME-fusionauth --gvc GVC_NAME + ``` **Symptom:** `cpln helm upgrade` from 2.3.x stops at render with a message from the PostgreSQL template saying that `config.username` (or `config.password`, `config.database`) was removed and that you should create a dictionary secret. - **Cause:** your values still carry `postgres.config.username`, `postgres.config.password` or `postgres.config.database`, which were renamed in 2.4.0. + **Cause:** your values still carry `postgres.config.username`, `postgres.config.password` or `postgres.config.database`, which were removed. - **Fix:** do not create a secret — this template creates it. Rename the keys as shown in [Upgrading From 2.3.x](#upgrading-from-2-3-x) and upgrade again. The running release is unaffected until the upgrade succeeds. + **Fix:** create the database credentials secret with the values your database was initialized with, as shown in [Upgrading From 2.3.x](#upgrading-from-2-3-x), and upgrade again. The running release is unaffected until the upgrade succeeds. - + **Symptom:** after the upgrade FusionAuth logs database authentication failures and never becomes ready. - **Cause:** the release never overrode `postgres.credentials.password`, so the database was initialized with the old default `password` while the new chart wrote `change-me-fusionauth-db` into the secret. + **Cause:** the new credentials secret holds a password other than the one the database was initialized with — for a 2.4.0 install that never overrode it, that password is literally `password`. - **Fix:** follow [Upgrading From 2.4.0](#upgrading-from-2-4-0) — either pin the old value explicitly or rotate the password inside PostgreSQL first. + **Fix:** put the password the database was initialized with into the secret, as in [Upgrading From 2.4.0](#upgrading-from-2-4-0), then force a redeployment of `RELEASE_NAME-fusionauth`. **Symptom:** clicking a Google, Microsoft or SAML login button returns to the login page or errors, and outbound requests from FusionAuth never arrive at the provider. @@ -472,12 +488,12 @@ One thing changed: the default `postgres.credentials.password` is `change-me-fus ## Important Notes - **Complete the setup wizard immediately after installing.** Inbound is open to `0.0.0.0/0` by default and there is no administrator until the wizard is finished — whoever gets there first creates that account. -- **Change `postgres.credentials.password` before the first install.** It is used exactly as given and applied only when the volume is first initialized; afterwards rotate with `ALTER ROLE` as in [Rotating the Database Password](#rotating-the-database-password). -- **Give each release its own `postgres.config.credentialsSecretName`.** Secret names are org-wide; a second release left on the default is refused at install. +- **Create the database credentials secret before installing.** A missing secret leaves the workloads waiting with zero log lines; `status.versions[].message` from `cpln workload get-deployments RELEASE_NAME-fusionauth --gvc GVC_NAME -o yaml` names it. +- **Rotating the database password needs `ALTER ROLE` plus a forced redeployment** — PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with; see [Rotating the Database Password](#rotating-the-database-password). - **Egress is closed by default.** External identity providers, outbound email and internet webhooks need `firewall.external.outboundAllowCIDR` opened. - **Set the tenant issuer before applications validate tokens** — it is the `iss` claim in every token, and changing it later invalidates the expectation in every client. - **`cpln helm uninstall` deletes the volume set** and with it every tenant, application and user. Take a backup first; the volume set keeps only a final snapshot for seven days. -- **Upgrading from 2.3.x or 2.4.0 needs a values change** — see [Upgrading From 2.3.x](#upgrading-from-2-3-x) and [Upgrading From 2.4.0](#upgrading-from-2-4-0). +- **Upgrading from 2.3.x or 2.4.x needs a new secret and a values change** — see [Upgrading From 2.3.x](#upgrading-from-2-3-x) and [Upgrading From 2.4.0](#upgrading-from-2-4-0). ## External References diff --git a/template-catalog/templates/ghost.mdx b/template-catalog/templates/ghost.mdx index c68461b3..a4c643d8 100644 --- a/template-catalog/templates/ghost.mdx +++ b/template-catalog/templates/ghost.mdx @@ -1,7 +1,7 @@ --- title: Ghost icon: /template-catalog/templates/icons/ghost.png -description: Deploy Ghost on Control Plane. Open-source publishing platform for blogs, newsletters and paid memberships, as a single-replica Ghost 6 workload with a bundled MySQL 8 database, persistent content volume, public HTTPS endpoint, optional SMTP and scheduled database backups. Needs an existing GVC. +description: Deploy Ghost on Control Plane. Open-source publishing platform for blogs, newsletters and paid memberships, as a single-replica Ghost 6 workload with a bundled MySQL 8 database, persistent content volume, public HTTPS endpoint, optional SMTP and backups. Needs an existing GVC and database secrets. keywords: ['blogging platform', 'cms', 'wordpress alternative', 'substack alternative', 'medium alternative', 'headless cms', 'content api', 'admin api', 'self-hosted publishing', 'mysqldump', 'staff 2fa', 'custom domain'] --- @@ -20,17 +20,41 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-ghost` | Ghost server — site, editor and Content/Admin APIs on port `2368` (stateful, 1 replica, container `ghost`) | — | | Volume Set | `RELEASE_NAME-ghost-vs` | Content directory at `/var/lib/ghost/content`: uploaded images, themes, logs, adapters | — | | Secret | `RELEASE_NAME-ghost-start` | Startup script that sets Ghost's `url` and hands off to the image entrypoint | — | -| Identity & Policy | `RELEASE_NAME-ghost-identity`, `RELEASE_NAME-ghost-policy` | `reveal` on exactly the startup script, the MySQL credentials secret, and your SMTP secret when `mail.secretName` is set | — | +| Identity & Policy | `RELEASE_NAME-ghost-identity`, `RELEASE_NAME-ghost-policy` | `reveal` on exactly the startup script, your database credentials secret, and your SMTP secret when `mail.secretName` is set | — | | Workload | `RELEASE_NAME-mysql` | MySQL 8 database (stateful, 1 replica, container `mysql`, port `3306`) from the bundled [mysql](/template-catalog/templates/mysql) subchart | — | | Volume Set | `RELEASE_NAME-mysql-vs` | MySQL data directory at `/var/lib/mysql` | — | -| Secret | `RELEASE_NAME-mysql-config` | Dictionary secret the chart writes from `mysql.config` — keys `database`, `user`, `password`, `rootPassword` | — | -| Identity & Policy | `RELEASE_NAME-mysql-identity`, `RELEASE_NAME-mysql-policy` | `reveal` on the MySQL credentials secret; carries the cloud-account binding when backups are on | — | +| Identity & Policy | `RELEASE_NAME-mysql-identity`, `RELEASE_NAME-mysql-policy` | `reveal` on your two database secrets; carries the cloud-account binding when backups are on | — | | Workload | `RELEASE_NAME-mysql-backup` | Cron job running a scheduled `mysqldump` to S3 or GCS | `mysql.backup.enabled: true` | -| Workload | `RELEASE_NAME-mysql-phpmyadmin` | phpMyAdmin console, published to the internet with the root password in its environment | `mysql.enablePhpMyAdmin: true` | ## Prerequisites -None for a default install — it deploys with a working MySQL 8 backend and an auto-assigned HTTPS endpoint. The two bundled database passwords ship as `change-me-…` placeholders and **must** be changed at install time; see [Installation](#installation). +Two database secrets must exist **before** you install; the chart never creates them. They hold credentials, so they are prerequisite secrets rather than values — a value would sit in plaintext in the Helm release for the life of the install. Secrets are org-level, so no GVC flag is involved. + + + + A [dictionary secret](/guides/create-secret/dictionary) with keys `username`, `password` and `database`. MySQL is initialized with it on first boot, and Ghost connects with the same keys: + + ```bash + cpln secret create-dictionary --name my-ghost-db-credentials \ + --entry username=ghost --entry password='YOUR-STRONG-PASSWORD' --entry database=ghost + ``` + + Set `mysql.credentialsSecretName` to this name. + + + An [opaque secret](/guides/create-secret/opaque) holding the MySQL root password, kept separate so Ghost never reads it: + + ```bash + printf '%s' 'YOUR-STRONG-ROOT-PASSWORD' | cpln secret create-opaque --name my-ghost-mysql-root-password --encoding plain -f - + ``` + + Set `mysql.rootPasswordSecretName` to this name. The two names must differ. + + + + +If either database secret is missing, the install reports success and then waits silently: `cpln logs` shows nothing, and the missing secret is named only in `status.versions[].message` of `cpln workload get-deployments RELEASE_NAME-mysql --gvc GVC_NAME -o yaml`. The workload recovers on its own once the secret exists, or immediately after `cpln workload force-redeployment RELEASE_NAME-mysql --gvc GVC_NAME`. + Two optional features need setup before you install: @@ -57,13 +81,13 @@ Create the SMTP secret **before** installing with `mail.secretName` set. A name ## Installation -A default install needs only the two database passwords — leave neither on its `change-me-…` placeholder: +A default install needs only the names of the two database secrets: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/ghost \ - --version 1.2.0 --gvc GVC_NAME \ - --set mysql.config.password='YOUR-STRONG-PASSWORD' \ - --set mysql.config.rootPassword='YOUR-STRONG-ROOT-PASSWORD' + --version 1.3.0 --gvc GVC_NAME \ + --set mysql.credentialsSecretName=my-ghost-db-credentials \ + --set mysql.rootPasswordSecretName=my-ghost-mysql-root-password ``` Or install using your preferred method: @@ -186,12 +210,13 @@ internalAccess: # Ghost supports ONLY MySQL 8 (not MySQL 9, not MariaDB) — image pinned to mysql:8. mysql: image: mysql:8 - enablePhpMyAdmin: false # keep the footprint to Ghost + its DB - config: - db: ghost - user: ghost - password: change-me-ghost-db # change before installing - rootPassword: change-me-mysql-root # change before installing + # BOTH SECRETS MUST EXIST BEFORE YOU INSTALL — the deployment waits silently on a + # secret that does not exist. See Prerequisites for the exact commands. + # `dictionary` secret with keys `username`, `password`, `database`: MySQL is + # initialized with it on first boot, and Ghost connects with the same keys. + credentialsSecretName: my-ghost-db-credentials + # `opaque` secret (encoding: plain) holding the MySQL root password. + rootPasswordSecretName: my-ghost-mysql-root-password resources: minCpu: 150m # keep cpu:minCpu ratio under the platform's 4:1 (500/150 = 3.3:1) maxCpu: 500m @@ -224,14 +249,13 @@ mysql: ``` - `mysql.image` — Pinned to `mysql:8`. Ghost supports **only MySQL 8** — not MySQL 9 and not MariaDB. Do not change it. -- `mysql.config.db` / `user` / `password` / `rootPassword` — The database, user and passwords the chart writes into the `RELEASE_NAME-mysql-config` secret and MySQL applies on first boot. **Both passwords must be changed before installing** — the `change-me-…` defaults are published in the public template repository, and an install that keeps them runs with a known password. They are internal to the release: Ghost reads them from the secret, and nothing outside the GVC can reach the database. +- `mysql.credentialsSecretName` / `mysql.rootPasswordSecretName` — The names of your two prerequisite database secrets (see [Prerequisites](#prerequisites)): a dictionary with `username`, `password` and `database`, and an opaque secret with the root password. MySQL applies them on first boot; Ghost reads only the first. Both must exist before installing and must be two different secrets. - `mysql.resources` / `mysql.volumeset.capacity` — CPU and memory bounds and the initial data volume size in GiB (minimum 10). The reservation-to-limit ratio is tuned to the platform's cap for stateful workloads; keep it when you raise the limits. - `mysql.internalAccess.type` — Which workloads can reach MySQL directly on port `3306`, using the same four types as Ghost's `internalAccess`. -- `mysql.enablePhpMyAdmin` — Off by default. When `true`, the subchart adds a `RELEASE_NAME-mysql-phpmyadmin` workload that is **open to the whole internet** and carries the root password in its environment. Leave it off on a public site. - `mysql.backup.*` — Scheduled dumps to S3 or GCS, off by default. See [Backing Up](#backing-up) for the bucket, Cloud Account and IAM setup. -MySQL applies `mysql.config.*` only on first startup, when the data directory is empty. Changing a password in values after the initial deployment does not change the running database — change it inside MySQL with `ALTER USER` and upgrade the release so the secret matches. To fully reset, `cpln helm uninstall RELEASE_NAME --gvc GVC_NAME` (which deletes both volume sets and all data) and reinstall, or install under a new release name. +MySQL applies the database secrets only on first startup, when the data directory is empty. Editing a secret after the initial deployment does not change the running database — change the password inside MySQL with `ALTER USER` first, then update the secret's `password` key (`cpln secret edit SECRET_NAME`) and force a redeployment of `RELEASE_NAME-mysql` and `RELEASE_NAME-ghost` (a secret change is not picked up until the workloads redeploy). To fully reset, `cpln helm uninstall RELEASE_NAME --gvc GVC_NAME` (which deletes both volume sets and all data) and reinstall, or install under a new release name. ## Connecting @@ -241,7 +265,7 @@ MySQL applies `mysql.config.*` only on first startup, when the data directory is | Public site | `https://CANONICAL_ENDPOINT` | Assigned automatically when `publicAccess.enabled: true`. Read it from `status.canonicalEndpoint` in `cpln workload get RELEASE_NAME-ghost --gvc GVC_NAME -o yaml`. | | Admin panel | `https://CANONICAL_ENDPOINT/ghost` | Editor, settings, members and staff. The owner account is created here on first visit — see [First Run](#first-run). | | Internal (same GVC) | `http://RELEASE_NAME-ghost.GVC_NAME.cpln.local:2368` | Subject to `internalAccess.type`. With public access on, Ghost redirects plain-HTTP requests to its HTTPS `url` unless the request carries `X-Forwarded-Proto: https`. | -| Database | `RELEASE_NAME-mysql.GVC_NAME.cpln.local:3306` | Subject to `mysql.internalAccess.type`. Credentials: `cpln secret reveal RELEASE_NAME-mysql-config -o yaml` (keys `database`, `user`, `password`, `rootPassword`). | +| Database | `RELEASE_NAME-mysql.GVC_NAME.cpln.local:3306` | Subject to `mysql.internalAccess.type`. Credentials: the secret named by `mysql.credentialsSecretName` — `cpln secret reveal SECRET_NAME -o yaml` (keys `username`, `password`, `database`). | Verify the site is serving once the workload reports ready — the default theme answers `200` before any owner exists: @@ -278,7 +302,7 @@ Ghost ships no bootstrap credentials. The first visitor to `/ghost` creates the Two things hold your data: the MySQL database (posts, pages, members, settings, staff accounts) and the content volume set (uploaded images, themes). A database dump does not include media. -**Scheduled database dumps** are off by default. When `mysql.backup.enabled: true`, the `RELEASE_NAME-mysql-backup` cron workload runs `mysqldump --databases` of the database named in `mysql.config.db` on `mysql.backup.schedule` and uploads a gzipped archive named `mysql-TIMESTAMP.sql.gz` under `mysql.backup.*.prefix` in your bucket. Pick a `mysql.backup.provider` and complete the storage setup **before** enabling it: +**Scheduled database dumps** are off by default. When `mysql.backup.enabled: true`, the `RELEASE_NAME-mysql-backup` cron workload runs `mysqldump --databases` of the database named in the `database` key of your `mysql.credentialsSecretName` secret on `mysql.backup.schedule` and uploads a gzipped archive named `mysql-TIMESTAMP.sql.gz` under `mysql.backup.*.prefix` in your bucket. Pick a `mysql.backup.provider` and complete the storage setup **before** enabling it: @@ -339,7 +363,7 @@ This ad-hoc command has **not been run against a live install of this template** This restore procedure was written against the chart's configuration and the backup image's dump format. It has **not been exercised against a live install of this template** — rehearse it on a throwaway release before you need it, and treat any step that errors as a stop. -Scheduled dumps are gzipped `mysqldump --databases` output of the database named in `mysql.config.db`, created with `--set-gtid-purged=OFF`, so the archive recreates the database itself (`CREATE DATABASE` and `USE` are embedded) and carries no GTID statement to strip. Download the archive from your bucket (`aws s3 cp` or `gsutil cp`), then stream it into the running MySQL through `cpln workload exec`. Stdin is off by default on `exec`, so pass `--stdin`, and base64-wrap the stream so nothing is altered in transit: +Scheduled dumps are gzipped `mysqldump --databases` output of the database named in the `database` key of your `mysql.credentialsSecretName` secret, created with `--set-gtid-purged=OFF`, so the archive recreates the database itself (`CREATE DATABASE` and `USE` are embedded) and carries no GTID statement to strip. Download the archive from your bucket (`aws s3 cp` or `gsutil cp`), then stream it into the running MySQL through `cpln workload exec`. Stdin is off by default on `exec`, so pass `--stdin`, and base64-wrap the stream so nothing is altered in transit: ```bash gunzip -c mysql-TIMESTAMP.sql.gz \ @@ -384,6 +408,26 @@ Template version 1.2.0 adds `staffDeviceVerification` (default `false`), which t The first `cpln helm upgrade` after an install can restart the bundled MySQL even when no database value changed, so run it during a quiet period. +### Upgrading from 1.2.0 + +Template version 1.3.0 removed `mysql.config` (database name, user, password and root password): the database credentials are now the two prerequisite secrets under [Prerequisites](#prerequisites), the same ones the `mysql` template requires on its own. An upgrade that still carries `mysql.config` is refused at render, before anything changes; `mysql.enablePhpMyAdmin` is refused the same way, so remove it too. + +1.2.0 created the secret `RELEASE_NAME-mysql-config` itself, and **this upgrade deletes it**. Create the new secrets under **different** names, holding the values your database was initialized with (not new ones, or Ghost and MySQL lose their login). Read the current values first: + +```bash +cpln secret reveal RELEASE_NAME-mysql-config -o yaml +``` + +Then create the two secrets from them (`user` becomes `username`; `rootPassword` gets its own secret). Use your current user and database names too, not `ghost`, if you changed them: + +```bash +cpln secret create-dictionary --name my-ghost-db-credentials \ + --entry username='YOUR_CURRENT_USER' --entry password='YOUR_CURRENT_PASSWORD' --entry database='YOUR_CURRENT_DATABASE' +printf '%s' 'YOUR_CURRENT_ROOT_PASSWORD' | cpln secret create-opaque --name my-ghost-mysql-root-password --encoding plain -f - +``` + +Upgrade with `mysql.credentialsSecretName` and `mysql.rootPasswordSecretName` set to those names, and `mysql.config` and `mysql.enablePhpMyAdmin` removed. The upgrade restarts MySQL and Ghost, so the site is briefly unavailable. The new version of a workload can wait for several minutes, with `get-deployments` reporting that it is not allowed to reveal the secret, while access to the new secrets propagates; the previous version keeps serving meanwhile, and it clears by itself. + ### Scaling and Availability Ghost is single-replica by design and this template has no `replicas` knob: Ghost holds scheduler, job-queue and cache state in-process and has no upstream support for running more than one instance against the same database. Both workloads are pinned to one replica. @@ -428,25 +472,39 @@ Durability comes from the two volume sets and, optionally, scheduled database du **Fix:** Set `publicUrl` to the full `https://` address visitors use, upgrade the release, then run `cpln workload force-redeployment RELEASE_NAME-ghost --gvc GVC_NAME`. - - **Symptom:** After `cpln helm upgrade` with a new `mysql.config.password` or `rootPassword`, Ghost fails to authenticate with `Access denied for user`. + + **Symptom:** After editing the `password` key of your database credentials secret and redeploying, Ghost fails to authenticate with `Access denied for user`. + + **Cause:** MySQL applies the database secrets only when the data directory is first initialized. Ghost now sends a password the database never learned. + + **Fix:** Either change the password inside MySQL to match the new value (`ALTER USER 'ghost'@'%' IDENTIFIED BY '...'` via `cpln workload exec RELEASE_NAME-mysql --gvc GVC_NAME --container mysql -- mysql -u root -p ...`), or put the old value back in the secret and force a redeployment of `RELEASE_NAME-ghost`. A clean reset is uninstall (deletes both volume sets) and reinstall. + + + **Symptom:** `RELEASE_NAME-mysql` stays not ready after install, and `cpln logs` returns no lines for it. + + **Cause:** A secret named by `mysql.credentialsSecretName` or `mysql.rootPasswordSecretName` does not exist, so the container never starts. + + **Fix:** Read `status.versions[].message` in `cpln workload get-deployments RELEASE_NAME-mysql --gvc GVC_NAME -o yaml` — it names the missing secret. Create it as shown in [Prerequisites](#prerequisites); the workload recovers on its own, or immediately after `cpln workload force-redeployment RELEASE_NAME-mysql --gvc GVC_NAME`. + + + **Symptom:** `cpln helm upgrade` stops at render with `ghost: mysql.config was removed in 1.3.0` (or `mysql.enablePhpMyAdmin was removed in 1.3.0`). Nothing is changed on the platform. - **Cause:** MySQL applies `mysql.config.*` only when the data directory is first initialized. The upgrade changed the secret, so Ghost now sends a password the database never learned. + **Cause:** Your values still carry the 1.2.0 plaintext database passwords. - **Fix:** Either change the password inside MySQL to match the new value (`ALTER USER 'ghost'@'%' IDENTIFIED BY '...'` via `cpln workload exec RELEASE_NAME-mysql --gvc GVC_NAME --container mysql -- mysql -u root -p ...`), or put the old value back in values and upgrade again. A clean reset is uninstall (deletes both volume sets) and reinstall. + **Fix:** Create the two database secrets holding your current values and remove the old keys — see [Upgrading from 1.2.0](#upgrading-from-1-2-0). ## Important Notes -- **Change both `mysql.config` passwords at install time** — the `change-me-…` defaults are public, and they seed the database on first boot; later value edits do not change the stored passwords. +- **Create both database secrets before installing** — a missing one wedges `RELEASE_NAME-mysql` silently; see [Prerequisites](#prerequisites). +- **The database secrets seed MySQL on first boot only** — editing them later does not change the stored passwords: change the password inside MySQL first, then update the secret and force a redeployment of `RELEASE_NAME-mysql` and `RELEASE_NAME-ghost`. - **Create the owner account right after install** — until someone completes the form at `/ghost`, the public site has no admin and anyone who reaches it can claim ownership. - **MySQL 8 only** — keep `mysql.image: mysql:8`; Ghost does not support MySQL 9 or MariaDB. - **SMTP is a prerequisite secret, not a value** — create the dictionary secret (keys `user`, `password`) before installing with `mail.secretName` set; a missing secret wedges the deployment silently. - **Set `publicUrl` when you attach a custom domain**, then force a redeployment of `RELEASE_NAME-ghost` — links and emails use whatever `url` the container started with. - **`staffDeviceVerification: true` requires working SMTP** — the chart refuses it otherwise, and a mis-delivering mail setup locks staff out of `/ghost` from new devices. - **Single replica by design** — there is no `replicas` knob; expect a brief interruption on restart or upgrade, and the first upgrade after install can restart the bundled MySQL. -- **Leave `mysql.enablePhpMyAdmin` off on a public site** — the console it adds is open to the internet and holds the root password. - **Data survives redeploys under the same release name** — content in `RELEASE_NAME-ghost-vs`, the database in `RELEASE_NAME-mysql-vs`. `cpln helm uninstall` deletes both; a database dump does not include uploaded media. ## External References diff --git a/template-catalog/templates/gitea.mdx b/template-catalog/templates/gitea.mdx index af327ace..ad62c71f 100644 --- a/template-catalog/templates/gitea.mdx +++ b/template-catalog/templates/gitea.mdx @@ -1,7 +1,7 @@ --- title: Gitea icon: /template-catalog/templates/icons/gitea.png -description: Single-replica Gitea Git server with web UI, Git-over-HTTPS, package registry and optional Git-over-SSH, backed by a bundled PostgreSQL. Deploys into an existing GVC and needs one dictionary auth secret (admin login plus signing keys) created before install. +description: Single-replica Gitea Git server with web UI, Git-over-HTTPS, package registry and optional Git-over-SSH, backed by a bundled PostgreSQL. Deploys into an existing GVC and needs two dictionary secrets created before install: auth (admin login plus signing keys) and database credentials. keywords: ['self-hosted git', 'git server', 'github alternative', 'gitlab alternative', 'gogs', 'source code management', 'pull requests', 'git clone', 'git lfs', 'container registry', 'jwt secret', 'secret key', 'rootless', 'developer tools'] --- @@ -9,7 +9,7 @@ keywords: ['self-hosted git', 'git server', 'github alternative', 'gitlab altern Gitea is a lightweight self-hosted Git service — repositories, pull requests, issues and a built-in package registry. This template deploys one Gitea server (the rootless image) on a persistent volume, a bundled PostgreSQL database from the [PostgreSQL](/template-catalog/templates/postgres) template, a public HTTPS endpoint serving the web UI, Git-over-HTTPS and the package registry, and optional Git-over-SSH. -The site administrator's login and Gitea's three long-lived signing keys are **not** template values. They come from one [dictionary secret](/guides/create-secret/dictionary) you create before installing, so they never pass through Helm and never land in the release. The bundled database's credentials are internal plumbing: the template builds that secret for you from plain values. +The site administrator's login and Gitea's three long-lived signing keys are **not** template values. They come from one [dictionary secret](/guides/create-secret/dictionary) you create before installing, so they never pass through Helm and never land in the release. The bundled database's credentials come from a second dictionary secret you create before installing. This template deploys into an existing [GVC](/reference/gvc) that you already have. It does not create or manage a GVC. @@ -24,20 +24,31 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Volume Set | `RELEASE_NAME-gitea-vs` | `/var/lib/gitea` — repositories, LFS objects, attachments and SSH host keys on an `ext4` general-purpose SSD volume; a final snapshot is taken when the volume set is deleted and kept 7 days | — | | Volume Set | `RELEASE_NAME-pg-vs` | PostgreSQL data | — | | Secret | `RELEASE_NAME-gitea-bootstrap` | Opaque secret holding the admin-bootstrap script mounted into the Gitea container | — | -| Secret | `my-gitea-db-credentials` | Dictionary secret with the bundled database's `username`, `password` and `database`, built from `postgres.credentials.*`; its name is `postgres.config.credentialsSecretName` | — | | Identity | `RELEASE_NAME-gitea-identity` | Bound to the Gitea workload | — | -| Policy | `RELEASE_NAME-gitea-policy` | Grants the Gitea identity `reveal` on exactly three secrets: the bootstrap script, the database credentials and the auth secret you created | — | +| Policy | `RELEASE_NAME-gitea-policy` | Grants the Gitea identity `reveal` on exactly three secrets: the bootstrap script, and the database credentials and auth secrets you created | — | | Identity | `RELEASE_NAME-pg-identity` | Bound to the database and backup workloads; carries the cloud account binding when backups use AWS or GCP | — | | Policy | `RELEASE_NAME-pg-policy` | Grants the database identity `reveal` on the database credentials secret | — | | Workload | `RELEASE_NAME-postgres-backup` | Scheduled `pg_dumpall` of the bundled database to S3, GCS or MinIO (cron) | `postgres.backup.enabled: true` | -The template creates **no credential secret for Gitea itself** — the admin login and signing keys live only in the auth secret you create. +The template creates **no credential secret** — the admin login and signing keys live only in the auth secret you create, and the database credentials only in the database credentials secret you create. ## Prerequisites -One secret must exist before you install. Secrets are org-level, so no GVC flag is involved. +Two secrets must exist before you install. Secrets are org-level, so no GVC flag is involved. + + A [dictionary secret](/guides/create-secret/dictionary) the bundled PostgreSQL initializes with and Gitea connects with: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=gitea \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=gitea + ``` + + Set `postgres.config.credentialsSecretName` to this name. Secret names are org-wide, so give each Gitea release its own. PostgreSQL reads it only when its data directory is first initialized — see [Bundled Database](#bundled-database) for changing the password later. + A [dictionary secret](/guides/create-secret/dictionary) holding exactly six keys. The three `admin*` keys create the first site-admin account on first boot; the other three are Gitea's signing keys: @@ -67,7 +78,7 @@ One secret must exist before you install. Secrets are org-level, so no GVC flag -**Create the secret before installing, or the deployment wedges silently.** The chart refuses to render when `gitea.auth.secretName` is blank, but a name pointing at a secret that does not exist installs successfully and then never starts. The container never runs, so `cpln logs` returns zero lines. The one place the reason appears is `status.versions[].message`: +**Create both secrets before installing, or the deployment wedges silently.** The chart refuses to render when `gitea.auth.secretName` is blank, but a name pointing at a secret that does not exist installs successfully and then never starts. The container never runs, so `cpln logs` returns zero lines. The one place the reason appears is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-gitea --gvc GVC_NAME -o yaml @@ -81,21 +92,19 @@ secret is added or the reference to the secret removed. Use `get-deployments` — plain `cpln workload get` has no `versions` key. Creating the missing secret repairs the deployment on its own within several minutes, or run `cpln workload force-redeployment RELEASE_NAME-gitea --gvc GVC_NAME` to skip the wait. -The bundled database needs nothing created in advance: the template writes its credentials secret from `postgres.credentials.*`. Pick a database password at install time (see [Installation](#installation)). Database backups are optional and need a bucket plus, for AWS or GCP, a Control Plane [cloud account](/guides/create-cloud-account) — see [Backing Up](#backing-up). +Database backups are optional and need a bucket plus, for AWS or GCP, a Control Plane [cloud account](/guides/create-cloud-account) — see [Backing Up](#backing-up). ## Installation -Install the chart from the marketplace registry, pointing it at the auth secret and replacing the published database-password placeholder: +Install the chart from the marketplace registry, pointing it at your two secrets: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/gitea \ - --version 1.2.0 --gvc GVC_NAME \ + --version 1.3.0 --gvc GVC_NAME \ --set gitea.auth.secretName=GITEA_AUTH_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' + --set postgres.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME ``` -For a second Gitea release in the same org, also set `postgres.config.credentialsSecretName` to a name unique to that release — secret names are org-wide, and a second release left on the default is refused at install (see [Troubleshooting](#troubleshooting)). - Or follow the instructions for your preferred method: @@ -219,17 +228,13 @@ internalAccess: # internal firewall scope ### Bundled Database -The database is internal plumbing: it serves Gitea only, is unreachable from outside the GVC, and no human types its password into a client — so the credentials stay plain values and the template writes them into the dictionary secret named by `postgres.config.credentialsSecretName`. The password is used as-is, so change it before installing. +The database serves Gitea only and is unreachable from outside the GVC. Its credentials come from your prerequisite dictionary secret (`username`, `password` and `database` — see [Prerequisites](#prerequisites)), named by `postgres.config.credentialsSecretName`. ```yaml postgres: image: postgres:18 - credentials: - username: gitea - password: change-me-gitea-db - database: gitea config: - credentialsSecretName: my-gitea-db-credentials + credentialsSecretName: my-gitea-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 200m minMemory: 256Mi @@ -247,12 +252,12 @@ postgres: ``` -PostgreSQL applies `postgres.credentials.*` only when its volume is first initialized. Changing them afterwards rewrites the secret but not the running database's password — rotate inside PostgreSQL first, as described in the [PostgreSQL template](/template-catalog/templates/postgres#rotating-the-password), then update the value and force a redeployment of `RELEASE_NAME-gitea` so it reconnects with the new password. +PostgreSQL reads the credentials secret only when its volume is first initialized, and a running workload keeps the values it started with. To change the password later, run `ALTER ROLE` first, then update the secret (export it with `cpln secret reveal SECRET_NAME -o yaml > secret.yaml`, change only `password`, and `cpln apply -f secret.yaml`), and right away force a redeployment of `RELEASE_NAME-gitea` with `cpln workload force-redeployment RELEASE_NAME-gitea --gvc GVC_NAME`. Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. ### Database Backup -Every knob of the PostgreSQL template is available under `postgres.*`, including its scheduled backup. It is off by default; enabling it adds one cron workload that runs `pg_dumpall` and uploads a gzip-compressed dump to your bucket. The job authenticates with the chart-created database credentials secret, and the database identity picks up the bucket-scoped policy or cloud account automatically. +Every knob of the PostgreSQL template is available under `postgres.*`, including its scheduled backup. It is off by default; enabling it adds one cron workload that runs `pg_dumpall` and uploads a gzip-compressed dump to your bucket. The job authenticates with your database credentials secret, and the database identity picks up the bucket-scoped policy or cloud account automatically. ```yaml postgres: @@ -428,19 +433,24 @@ If your install still carries the published 1.0.0 defaults, everything it has en ### Upgrading from 1.1.0 -Version 1.2.0 adopted the PostgreSQL template 3.4.1 as its bundled database. That template no longer takes credentials as values, and Gitea absorbed the change rather than passing it on: **there is no new prerequisite**, only a rename. The database keeps the credentials it was initialized with, so carry the **same three values** over under the new keys. +`postgres.config.username`, `postgres.config.password` and `postgres.config.database` were removed. The database keeps the credentials it was initialized with, so put the **same three values** into the database credentials secret under [Prerequisites](#prerequisites) (keys `username`, `password` and `database`), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. The auth secret is unchanged. -| Removed key | Replacement | -|---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | +### Upgrading from 1.2.0 -The chart now writes those values into a dictionary secret named by `postgres.config.credentialsSecretName` (default `my-gitea-db-credentials`) and hands only the name to the database. Secret names are org-wide, so give each Gitea release its own. +`postgres.credentials` was removed in 1.3.0: the database credentials are now a prerequisite secret, the same one the `postgres` template requires on its own. Carrying the old key forward fails the render. - -**A values file that still carries the old keys fails with the PostgreSQL template's message, not this template's.** It tells you to create a dictionary secret yourself — ignore that advice here, because this template creates the secret for you. Move the three keys and the upgrade proceeds; the auth secret is unchanged. - +1.2.0 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Gitea loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-gitea-db \ + --entry username=gitea \ + --entry password='YOUR_CURRENT_PASSWORD' \ + --entry database=gitea +``` + +Then upgrade with `postgres.config.credentialsSecretName` set to that name and `postgres.credentials` removed. Your current values are in `postgres.credentials` of the 1.2.0 release, and also in the old secret (`cpln secret reveal my-gitea-db-credentials -o yaml`, or the name your release set in `credentialsSecretName`) until the upgrade deletes it. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Gitea can briefly lose its database; it reconnects by itself. The new version of a workload can also wait several minutes with `is not allowed to reveal the secret` in `get-deployments` while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. ### Scaling and Availability @@ -499,16 +509,9 @@ The chart now writes those values into a dictionary secret named by `postgres.co config.credentialsSecretName to its name. ... ``` - **Cause:** your values still set `postgres.config.username`, `postgres.config.password` or `postgres.config.database`, which moved to `postgres.credentials.*` in 1.2.0. - - **Fix:** rename the three keys as in [Upgrading from 1.1.0](#upgrading-from-1-1-0). Do not create a database secret yourself — this template creates it. - - - **Symptom:** installing a second Gitea release in the same org fails and creates nothing. - - **Cause:** both releases render the same database credentials secret name, `postgres.config.credentialsSecretName` (default `my-gitea-db-credentials`), and secret names are org-wide. The first release owns it; nothing is shared, overwritten or deleted. + **Cause:** your values still set `postgres.config.username`, `postgres.config.password` or `postgres.config.database`, which were removed. - **Fix:** install the second release with `--set postgres.config.credentialsSecretName=UNIQUE_NAME`. + **Fix:** create the database credentials secret with those three values and remove the keys, as in [Upgrading from 1.1.0](#upgrading-from-1-1-0). **Symptom:** you updated `adminPassword` in the auth secret, but Gitea still accepts the previous password. @@ -528,15 +531,15 @@ The chart now writes those values into a dictionary secret named by `postgres.co ## Important Notes -- **Create the auth secret before installing.** A reference to a secret that does not exist wedges the workload with no log output; see [Prerequisites](#prerequisites) for the one command that shows the reason. +- **Create both secrets before installing.** A reference to a secret that does not exist wedges the workload with no log output; see [Prerequisites](#prerequisites) for the one command that shows the reason. +- **Rotating the database password needs `ALTER ROLE` plus a forced redeployment of `RELEASE_NAME-gitea`** — a running workload keeps the values it started with; see [Bundled Database](#bundled-database). - **Never rotate `secretKey` or `jwtSecret` after install.** The first makes stored 2FA secrets, tokens and mirror credentials permanently unreadable; the second invalidates issued OAuth2 tokens. - **`jwtSecret` must be base64url of exactly 32 bytes** — use the command in [Prerequisites](#prerequisites). -- **Change `postgres.credentials.password` before installing.** The shipped `change-me-gitea-db` is a published placeholder used as-is. -- **Give each Gitea release its own `postgres.config.credentialsSecretName`.** Secret names are org-wide; a second release on the default name is refused at install. +- **Give each Gitea release its own database credentials secret.** Secret names are org-wide. - **Public SSH and the public web UI cannot share the canonical endpoint** — SSH is off by default. See [Git-over-SSH](#git-over-ssh). - **Do not scale past one replica.** Replicas would each get a separate repository volume. - **Editing the auth secret after first boot does not change the administrator account.** Change the password in Gitea — see [Rotating the Admin Password](#rotating-the-admin-password). -- **`cpln helm uninstall` deletes both volume sets** — repositories and database alike. Your auth secret is yours and is left alone. +- **`cpln helm uninstall` deletes both volume sets** — repositories and database alike. Your auth and database credentials secrets are yours and are left alone. ## External References diff --git a/template-catalog/templates/glitchtip.mdx b/template-catalog/templates/glitchtip.mdx index b2b6da88..5f224156 100644 --- a/template-catalog/templates/glitchtip.mdx +++ b/template-catalog/templates/glitchtip.mdx @@ -1,7 +1,7 @@ --- title: GlitchTip icon: /template-catalog/templates/icons/glitchtip.png -description: Deploy GlitchTip, open-source Sentry-compatible error tracking, on Control Plane. A stateless web tier and a background worker on bundled Redis Sentinel and PostgreSQL — highly available by default, single-instance by one flag. Needs an existing GVC and a pre-created auth dictionary secret. +description: Deploy GlitchTip, open-source Sentry-compatible error tracking. A stateless web tier and a background worker on bundled Redis Sentinel and PostgreSQL — highly available by default, single-instance by one flag. Needs an existing GVC and pre-created auth, database and Redis dictionary secrets. keywords: ['error tracking', 'error monitoring', 'exception tracking', 'sentry alternative', 'self-hosted sentry', 'crash reporting', 'application monitoring', 'observability', 'mit licensed', 'dsn', 'event ingest', 'django secret key', 'uptime monitoring'] --- @@ -30,20 +30,40 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Volume set | `RELEASE_NAME-postgres-ha-vs`, `RELEASE_NAME-etcd-vs` | HA database and etcd data, 10 GiB per replica; takes no scheduled snapshots; a final snapshot is taken when the volume set is deleted and kept 7 days | `postgresHA.enabled: true` (default) | | Volume set | `RELEASE_NAME-pg-vs` | Single-instance database data, 10 GiB; takes no scheduled snapshots; a final snapshot is taken when the volume set is deleted and kept 7 days | `postgres.enabled: true` | | Volume set | `RELEASE_NAME-redis-vs`, `RELEASE_NAME-sentinel-vs` | Redis and Sentinel persistence, 10 GiB each with 7-day snapshots | `redis.enabled: true` (default) | -| Secret | the name in `postgresHA.config.credentialsSecretName` (HA) or `postgres.config.credentialsSecretName` (single) | Dictionary secret the chart creates from `postgres.credentials.*` (`username`, `password`, `database`) and hands to whichever database is enabled | — | | Secret | `RELEASE_NAME-glitchtip-web-start`, `RELEASE_NAME-glitchtip-worker-start` | Start scripts that finalize the public URL and the Sentinel connection URL at boot | — | -| Secret | `RELEASE_NAME-redis-auth-password` | Redis password, created by the redis subchart from `redis.redis.auth.password.value` | `redis.enabled: true` (default) | | Identity | `RELEASE_NAME-glitchtip-identity` | Shared by the web and worker workloads | — | -| Policy | `RELEASE_NAME-glitchtip-policy` | Grants that identity `reveal` on exactly the secrets it reads: the start scripts, the database and Redis secrets, your auth secret, and your email secret when configured | — | +| Policy | `RELEASE_NAME-glitchtip-policy` | Grants that identity `reveal` on exactly the secrets it reads: the start scripts, your database credentials, Redis password and auth secrets, and your email secret when configured | — | | Identity / Policy | `RELEASE_NAME-postgres-ha-identity`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-pg-identity`, `RELEASE_NAME-redis-identity`, `RELEASE_NAME-sentinel-identity` and their matching `-policy` | Per-component identities and least-privilege policies from the bundled templates | Follow the component | -The web and worker tiers have no volumes of their own: every issue, event, and user lives in PostgreSQL. Your auth secret is **not** created by the template — see [Prerequisites](#prerequisites). +The web and worker tiers have no volumes of their own: every issue, event, and user lives in PostgreSQL. Your auth, database credentials and Redis password secrets are **not** created by the template — see [Prerequisites](#prerequisites). ## Prerequisites -The Django signing key and the initial superuser login guard a login form on a public endpoint, so they are a [dictionary secret](/guides/create-secret/dictionary) you create **before** installing rather than Helm values. Secrets are org-level, so no GVC flag is involved. +The database credentials, the Redis password, and the Django signing key and initial superuser login (which guard a login form on a public endpoint) are [dictionary secrets](/guides/create-secret/dictionary) you create **before** installing rather than Helm values. Secrets are org-level, so no GVC flag is involved. + + The bundled PostgreSQL initializes with these values and GlitchTip connects with them, in either database mode: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=glitchtip \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=glitchtip + ``` + + Set its name in the key of the database mode you use: `postgresHA.config.credentialsSecretName` (HA, the default) or `postgres.config.credentialsSecretName` (single-instance). Secret names are org-wide, so give each GlitchTip release its own. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password). + + + Required while Redis is enabled (the default). Redis requires the password, and Sentinel and GlitchTip connect with it. No spaces, backslashes or double quotes: Redis writes it into its config file and will not start on them (the hex value below is safe): + + ```bash + cpln secret create-dictionary --name REDIS_SECRET_NAME \ + --entry password="$(openssl rand -hex 24)" + ``` + + Set `redis.redis.auth.fromSecret.name` to this name. To change the password later, see [Rotating the Redis Password](#rotating-the-redis-password). + A dictionary secret holding exactly three keys: @@ -78,28 +98,28 @@ The Django signing key and the initial superuser login guard a login form on a p -Create the auth secret **before** installing. Installing without it reports success, but **both** workloads then wedge silently — `cpln logs` returns nothing because no container ever starts, and the only place the missing secret is named is `status.versions[].message`: +Create the secrets **before** installing. Installing without one reports success, but **both** workloads then wedge silently — `cpln logs` returns nothing because no container ever starts, and the only place the missing secret is named is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-glitchtip --gvc GVC_NAME -o yaml ``` -The message reads `The secret AUTH_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` The command is `get-deployments` — plain `cpln workload get` has no `versions` key. Creating the secret lets the deployment recover on its own; `cpln workload force-redeployment RELEASE_NAME-glitchtip --gvc GVC_NAME` skips the wait, and the worker needs the same treatment. +The message reads `The secret AUTH_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` The command is `get-deployments` — plain `cpln workload get` has no `versions` key. Creating the secret lets the deployment recover on its own; `cpln workload force-redeployment RELEASE_NAME-glitchtip --gvc GVC_NAME` skips the wait, and the worker needs the same treatment. A missing Redis password secret also wedges `RELEASE_NAME-redis` and `RELEASE_NAME-sentinel`. ## Installation -Once the auth secret exists, install from the marketplace registry, pointing `auth.secretName` at it and replacing both bundled-component passwords: +Once the three secrets exist, install from the marketplace registry, pointing the secret-name keys at them: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/glitchtip \ - --version 1.3.1 --gvc GVC_NAME \ + --version 1.4.0 --gvc GVC_NAME \ --set auth.secretName=AUTH_SECRET_NAME \ - --set postgres.credentials.password='YOUR-DB-PASSWORD' \ - --set redis.redis.auth.password.value='YOUR-REDIS-PASSWORD' + --set postgresHA.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME \ + --set redis.redis.auth.fromSecret.name=REDIS_SECRET_NAME ``` -`postgres.credentials.password` seeds the bundled database in **both** database modes. Installing a second GlitchTip release in the same org? Also set `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName` in single-instance mode) to a name no other release uses — secret names are org-wide. +In single-instance mode set `postgres.config.credentialsSecretName` instead of `postgresHA.config.credentialsSecretName`. Other ways to install and manage the template: @@ -206,9 +226,9 @@ redis: redis: replicas: 2 auth: - password: - enabled: true - value: change-me-glitchtip-redis # change before installing (any characters OK — the boot script percent-encodes it) + fromSecret: + enabled: true # required when redis is enabled (the chart enforces this) + name: my-glitchtip-redis # PREREQUISITE dictionary secret (key: password) — must exist BEFORE install firewall: internal_inboundAllowType: same-gvc persistence: @@ -221,19 +241,11 @@ redis: enabled: true ``` -When Redis is enabled the chart requires `redis.redis.auth.password.enabled: true` with a value, and requires Sentinel authentication to stay off — GlitchTip does not send a Sentinel password, so the same-GVC firewall is the boundary there. Every other knob of the redis template is available under `redis.*`. +When Redis is enabled the chart requires `redis.redis.auth.fromSecret.enabled: true` with `redis.redis.auth.fromSecret.name` naming your Redis password secret, and requires Sentinel authentication to stay off — GlitchTip does not send a Sentinel password, so the same-GVC firewall is the boundary there. Every other knob of the redis template is available under `redis.*`. ### Database -Enable exactly one database — the chart refuses to render with both or neither. The highly available cluster is the default; the single-instance template is one flag away for development and lightweight installs. In both modes the chart builds the database credentials secret from `postgres.credentials.*` — there is nothing for you to create — and GlitchTip is wired to the active database automatically: the HAProxy leader endpoint in HA mode, the single instance directly otherwise. - -```yaml -postgres: - credentials: # used in BOTH database modes — the chart writes these into the credentials secret - username: glitchtip - password: change-me-glitchtip-db # change before installing - database: glitchtip -``` +Enable exactly one database — the chart refuses to render with both or neither. The highly available cluster is the default; the single-instance template is one flag away for development and lightweight installs. In both modes the database credentials come from your prerequisite dictionary secret (`username`, `password` and `database` — see [Prerequisites](#prerequisites)), and GlitchTip is wired to the active database automatically: the HAProxy leader endpoint in HA mode, the single instance directly otherwise. @@ -243,7 +255,7 @@ postgres: postgresHA: enabled: true config: - credentialsSecretName: my-glitchtip-db-credentials # chart-created from postgres.credentials.*; secret names are org-wide, so give each release its own + credentialsSecretName: my-glitchtip-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install replicas: 3 volumeset: capacity: 10 # initial capacity in GiB per replica (minimum is 10) @@ -263,7 +275,7 @@ postgres: postgres: enabled: true config: - credentialsSecretName: my-glitchtip-db-credentials # chart-created from postgres.credentials.*; secret names are org-wide, so give each release its own + credentialsSecretName: my-glitchtip-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) @@ -275,7 +287,7 @@ postgres: -`postgres.credentials.password` and `redis.redis.auth.password.value` seed their component on first boot and are used as-is — change both before installing. Later value edits do not update a database that has already initialized; uninstalling the release deletes the volume sets, and a reinstall starts fresh. +PostgreSQL reads its credentials secret only when its data is first initialized, and the Sentinels keep the Redis password they started with, so editing a secret alone changes neither — follow [Rotating the Database Password](#rotating-the-database-password) or [Rotating the Redis Password](#rotating-the-redis-password). Uninstalling the release deletes the volume sets, and a reinstall starts fresh. ## Connecting @@ -290,7 +302,7 @@ postgres: | Internal (same GVC) | `http://RELEASE_NAME-glitchtip.GVC_NAME.cpln.local:8000` | Reachable per `internalAccess.type`; in-GVC SDKs can report here. | | PostgreSQL (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | `username` / `password` / `database` from `cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is your `postgresHA.config.credentialsSecretName`. | | PostgreSQL (single mode) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Same keys, from the secret named by `postgres.config.credentialsSecretName`. | -| Redis Sentinel | `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` | Master name `mymaster`; password in `cpln secret reveal RELEASE_NAME-redis-auth-password -o yaml`. | +| Redis Sentinel | `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` | Master name `mymaster`; password in the `password` key of `cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is your `redis.redis.auth.fromSecret.name`. | With `publicAccess.enabled: false` there is no public endpoint; reach the UI from your machine through a tunnel instead and open `http://localhost:8000`: @@ -479,31 +491,131 @@ Template version 1.1.0 moved the Django signing key and the superuser login out If your install still carries the published 1.0.x default signing key, its sessions and tokens are signed with a value printed in a public repository. Rotating it is the fix, and the cost is exactly that one forced sign-out — plan the upgrade for a quiet window rather than skipping it, and change the admin password in the UI at the same time. +### Rotating the Database Password + +PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with, so in single-instance mode a rotation takes three steps in order (in HA mode follow [Rotating the Database Password in HA Mode](#rotating-the-database-password-in-ha-mode) instead): + +1. Change it in the database: `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` (`DB_USERNAME` is the secret's `username`; connect as that user from inside the GVC). +2. Update the secret — export it, change only `password`, and apply it: + + ```bash + cpln secret reveal SECRET_NAME -o yaml > secret.yaml + # edit only the password key in secret.yaml + cpln apply -f secret.yaml + ``` + +3. Right away, redeploy GlitchTip: + + ```bash + cpln workload force-redeployment RELEASE_NAME-glitchtip --gvc GVC_NAME + cpln workload force-redeployment RELEASE_NAME-glitchtip-worker --gvc GVC_NAME + ``` + +Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. Connections the app already has open keep working after `ALTER ROLE`; new ones fail until its redeploy finishes, so rotate in a quiet window. + +### Rotating the Database Password in HA Mode + +The HA database (`postgresHA`) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Change BOTH roles on -0 +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +# 4. Right after, put the same password in the secret (edit only the password key in secret.yaml) +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +cpln apply -f secret.yaml +# 5. Restart the members +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 6. Once the restart completes, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 7. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-glitchtip-worker` and `RELEASE_NAME-glitchtip`, which keep the old password until they restart. The database drops connections briefly at each switchover, and the app cannot open new connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + +### Rotating the Redis Password + +Redis keeps no password on disk, but the Sentinels do, so a new password is applied live in three places before anything restarts. Each Redis node first accepts both passwords, which keeps replication and every connected client working throughout. `rx` and `sx` run a command on one Redis node or Sentinel (`LOCATION` is your GVC's location; nodes and Sentinels are numbered from 0 up to `redis.redis.replicas` and `redis.sentinel.replicas` minus one): + +```bash +rx() { cpln workload exec RELEASE_NAME-redis --gvc GVC_NAME --location LOCATION \ + --container redis --replica "RELEASE_NAME-redis-$1" -- "${@:2}"; } +sx() { cpln workload exec RELEASE_NAME-sentinel --gvc GVC_NAME --location LOCATION \ + --container sentinel --replica "RELEASE_NAME-sentinel-$1" -- "${@:2}"; } +# 1. On EVERY Redis node: accept the new password alongside the old one, and replicate with it +rx 0 redis-cli --no-auth-warning -a OLD_PASSWORD ACL SETUSER default ">NEW_PASSWORD" +rx 0 redis-cli --no-auth-warning -a OLD_PASSWORD CONFIG SET masterauth NEW_PASSWORD +# 2. On EVERY Sentinel +sx 0 redis-cli -p 26379 SENTINEL SET mymaster auth-pass NEW_PASSWORD +# 3. Put the new password in the secret (edit only the password key in secret.yaml) and read it back +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +# 4. Restart GlitchTip, which keeps the old password until it restarts +cpln workload force-redeployment RELEASE_NAME-glitchtip --gvc GVC_NAME +cpln workload force-redeployment RELEASE_NAME-glitchtip-worker --gvc GVC_NAME +# 5. Once get-deployments shows a higher version, alone and ready, for both: retire the old password on EVERY Redis node +rx 0 redis-cli --no-auth-warning -a NEW_PASSWORD ACL SETUSER default " The first Helm upgrade after an install can restart the bundled database and Redis even when their values are unchanged, so expect a brief interruption. @@ -527,7 +639,7 @@ The default database is the 3-replica Patroni cluster with automatic leader fail **Symptom:** `cpln helm install` reported success, `cpln logs` returns nothing for either workload, and `cpln workload get-deployments RELEASE_NAME-glitchtip --gvc GVC_NAME -o yaml` shows `The secret AUTH_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` under `status.versions[].message`. - **Cause:** The prerequisite auth secret named by `auth.secretName` does not exist (or the optional email secret named by `email.secretName` does not). + **Cause:** A prerequisite secret does not exist — the auth secret named by `auth.secretName`, the database credentials secret named by `*.config.credentialsSecretName`, the Redis password secret named by `redis.redis.auth.fromSecret.name`, or the optional email secret named by `email.secretName`. The message names the missing one. **Fix:** Create it (see [Prerequisites](#prerequisites)). The deployment recovers on its own once the secret exists; `cpln workload force-redeployment RELEASE_NAME-glitchtip --gvc GVC_NAME` skips the wait — run it for `RELEASE_NAME-glitchtip-worker` too. @@ -546,25 +658,18 @@ The default database is the 3-replica Patroni cluster with automatic leader fail **Fix:** Set exactly one of them to `true` — see [Database](#database). - **Symptom:** `glitchtip: redis.redis.auth.password.enabled must be true with a value` or `redis.redis.auth.password.value is required when redis is enabled`. - - **Cause:** Redis is enabled without a password value. GlitchTip's Sentinel connection URL embeds the password, so the chart requires one; the redis template's `fromSecret` method is not supported here. - - **Fix:** Set `redis.redis.auth.password.enabled: true` and `redis.redis.auth.password.value`, or set `redis.enabled: false` to run the queue on PostgreSQL. - - - **Symptom:** Installing a second GlitchTip release fails with `cannot be updated because it is being managed by a different release` and creates nothing. + **Symptom:** `glitchtip: redis.redis.auth.fromSecret.enabled must be true when redis is enabled` or `redis.redis.auth.fromSecret.name is required when redis is enabled`. - **Cause:** Both releases use the same credentials secret name (`postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName`). Secret names are org-wide, and the first release owns that one. Nothing is shared, overwritten, or deleted. + **Cause:** Redis is enabled without the Redis password secret. GlitchTip's Sentinel connection URL carries the password, read from the prerequisite secret named by `redis.redis.auth.fromSecret.name`. - **Fix:** Set a distinct secret name for the second release and install again. + **Fix:** Create the Redis password secret (see [Prerequisites](#prerequisites)) and set `redis.redis.auth.fromSecret.enabled: true` and `redis.redis.auth.fromSecret.name`, or set `redis.enabled: false` to run the queue on PostgreSQL. - **Symptom:** The render fails with `glitchtip: ... was removed in 1.1.0`, `config.username was REMOVED in postgres 3.4.0`, or `postgres-highly-available: the postgres block was REMOVED in 2.5.0`. + **Symptom:** The render fails with `glitchtip: ... was removed in 1.1.0`, `config.username was REMOVED in postgres 3.4.0`, `postgres-highly-available: the postgres block was REMOVED in 2.5.0`, `postgres.credentials was REMOVED in 1.4.0`, or `redis.redis.auth.password was REMOVED in 1.4.0`. **Cause:** Your values file still carries a key from an earlier template version. - **Fix:** Follow the table for your version under [Upgrading from 1.0.x](#upgrading-from-1-0-x), [Upgrading from 1.1.0](#upgrading-from-1-1-0), or [Upgrading from 1.2.x](#upgrading-from-1-2-x). Do not create a database credentials secret yourself — this template creates it. + **Fix:** Follow the section for your version under [Upgrading from 1.0.x](#upgrading-from-1-0-x), [Upgrading from 1.1.0](#upgrading-from-1-1-0), [Upgrading from 1.2.x](#upgrading-from-1-2-x), or [Upgrading from 1.3.1](#upgrading-from-1-3-1). **Symptom:** An invite or alert is created in the UI and no email is delivered. @@ -584,17 +689,17 @@ The default database is the 3-replica Patroni cluster with automatic leader fail ## Important Notes -- **Create the auth secret before installing** — `auth.secretName` must name an existing dictionary secret with `secretKey`, `adminEmail`, and `adminPassword`. A missing secret wedges both workloads silently. +- **Create the prerequisite secrets before installing** — `auth.secretName` must name an existing dictionary secret with `secretKey`, `adminEmail`, and `adminPassword`, plus the database credentials secret and, with Redis enabled, the Redis password secret. A missing secret wedges the workloads silently; `status.versions[].message` from `cpln workload get-deployments RELEASE_NAME-glitchtip --gvc GVC_NAME -o yaml` names it. - **Keep `secretKey` for the life of the install** — rotating it logs every user out and invalidates password-reset links in flight. Nothing is corrupted. - **The auth secret seeds the superuser once** — editing it after first boot does not change the existing account. Change the password in the UI. - **Set `domain` to the URL your users actually open** — DSNs and emailed links embed it; after a change, upgrade and update the DSNs in your applications. -- **Change `postgres.credentials.password` and `redis.redis.auth.password.value` before installing** — both seed their component on first boot and are used as-is. -- **Give each release its own credentials secret name** — secret names are org-wide, and a second release on the same name is refused at install. +- **Changing a password takes more than a secret edit** — the database password needs `ALTER ROLE` (and a leader switchover first in HA mode), the Redis password must be applied live on Redis and the Sentinels, and both need a forced redeployment of GlitchTip; see [Rotating the Database Password](#rotating-the-database-password) and [Rotating the Redis Password](#rotating-the-redis-password). +- **Give each release its own database credentials secret** — secret names are org-wide. - **Do not expect to scale the worker** — it is a fixed singleton; `replicas` scales the web tier. Ingest keeps accepting events while the worker is down. - **With registration closed, invites only work for existing accounts** — create them at `/admin/` first; invite and alert emails require `email.secretName`. - **Pick the database mode before the first install** — there is no in-place switch between `postgresHA` and `postgres`. - **Source-map and artifact uploads are ephemeral** — kept on local disk, lost on restart, and not shared across web replicas. Error ingest itself is unaffected. -- **Uninstalling deletes the volume sets** and every issue, event, and user in them; your auth secret survives. Enable backups if the data matters. +- **Uninstalling deletes the volume sets** and every issue, event, and user in them; your prerequisite secrets survive. Enable backups if the data matters. ## External References diff --git a/template-catalog/templates/grafana.mdx b/template-catalog/templates/grafana.mdx index 80ef1650..bac37fce 100644 --- a/template-catalog/templates/grafana.mdx +++ b/template-catalog/templates/grafana.mdx @@ -1,7 +1,7 @@ --- title: Grafana icon: /template-catalog/templates/icons/grafana.png -description: Grafana OSS dashboards and alerting over your own datasources, deployed into an existing GVC with a bundled PostgreSQL app database (highly available by default, single-instance optional) and optional Redis-coordinated multi-replica alerting. Two opaque prerequisite secrets must exist first. +description: Grafana OSS dashboards and alerting over your own datasources, deployed into an existing GVC with a bundled PostgreSQL app database (highly available by default, single-instance optional) and optional Redis-coordinated multi-replica alerting. Three prerequisite secrets must exist first. keywords: ['promql', 'datasource provisioning', 'alert rules', 'alertmanager', 'kibana alternative', 'chronograf alternative', 'metrics visualization', 'time series dashboards', 'redis sentinel', 'prometheus datasource', 'thanos datasource', 'patroni', 'self-hosted', 'observability'] --- @@ -33,7 +33,6 @@ This template deploys into a GVC **you already have** — it does not create, pr | Workload | `RELEASE_NAME-redis`, `RELEASE_NAME-sentinel` | Alerting-HA coordination tier (stateful, 3 replicas each) | `redis.enabled: true` | | Volume set | `RELEASE_NAME-postgres-ha-vs`, `RELEASE_NAME-etcd-vs` | Highly available database and consensus-store data (10 GiB per replica by default) | Default; `postgresHA.enabled: true` | | Volume set | `RELEASE_NAME-pg-vs` | Single-instance database data (10 GiB by default) | `postgres.enabled: true` | -| Secret | The name in `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` (default `my-grafana-db-credentials`) | App-database credentials, **created by this chart** from `postgres.credentials.*` and read by whichever database is enabled | — | | Secret | `RELEASE_NAME-grafana-datasources` | Rendered datasource provisioning file, mounted by the Grafana workload | `datasources.definitions` non-empty | | Secret | `RELEASE_NAME-etcd-startup`, `RELEASE_NAME-patroni-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-wal-g-backup-script` | Startup scripts of the highly available database tier | Default; `postgresHA.enabled: true` | | Secret | `RELEASE_NAME-postgres-config` | Non-sensitive backup settings of the highly available database (bucket, region, endpoint) | `postgresHA.backup.enabled: true` | @@ -43,13 +42,25 @@ This template deploys into a GVC **you already have** — it does not create, pr | Identity / Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | The single-instance database's own identity and policy | `postgres.enabled: true` | | Identity / Policy | `RELEASE_NAME-redis-identity`, `RELEASE_NAME-redis-policy`, `RELEASE_NAME-sentinel-identity`, `RELEASE_NAME-sentinel-policy` | The coordination tier's own identities and policies | `redis.enabled: true` | -The admin password and the encryption key are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. The app-database credentials secret is the opposite: the chart creates it from `postgres.credentials.*`, so there is nothing for you to create for the database. +The app-database credentials, the admin password and the encryption key are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. ## Prerequisites -Two [opaque secrets](/guides/create-secret/opaque) must exist before you install. Grafana's admin account is a human-facing login, and `publicAccess.enabled` defaults to `true`, so that account is reachable from the internet — neither credential is a Helm value, and neither ever lands in the release. Secrets are org-level, so no `--gvc` flag is involved. +Three secrets must exist before you install: a [dictionary secret](/guides/create-secret/dictionary) with the app-database credentials, and two [opaque secrets](/guides/create-secret/opaque) for the admin password and the encryption key. Grafana's admin account is a human-facing login, and `publicAccess.enabled` defaults to `true`, so that account is reachable from the internet — none of these credentials is a Helm value, and none ever lands in the release. Secrets are org-level, so no `--gvc` flag is involved. + + The bundled PostgreSQL initializes with these values and Grafana connects with them, in either database mode: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=grafana \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=grafana + ``` + + Set `postgresHA.config.credentialsSecretName` (HA, the default) or `postgres.config.credentialsSecretName` (single-instance) to this name. Secret names are org-wide, so give each release its own. PostgreSQL reads it only when its data directory is first initialized — see [Rotating the Database Password](#rotating-the-database-password). + Choose your own strong password for the admin login and store it as the secret's payload with encoding `plain`. Grafana reads it only when the admin account is first created: @@ -68,9 +79,6 @@ Two [opaque secrets](/guides/create-secret/opaque) must exist before you install Set `admin.secretKeySecretName` to this name. - - The bundled database's credentials are internal plumbing: this chart writes `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database` into a dictionary secret it creates, and the database reads that secret. Nothing to create — but the password ships as the placeholder `change-me-grafana-db-password`, so set `postgres.credentials.password` to a value of your own before installing. This applies in both database modes. - @@ -83,7 +91,7 @@ cpln workload get-deployments RELEASE_NAME-grafana --gvc GVC_NAME -o yaml The message reads `The secret no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` Create the missing secret and the deployment recovers on its own after several minutes, or run `cpln workload force-redeployment RELEASE_NAME-grafana --gvc GVC_NAME` to skip the wait. -The two secrets have **different lifecycles** — read [Admin Credentials](#admin-credentials) before you delete or rotate either one. +The admin password and encryption key secrets have **different lifecycles** — read [Admin Credentials](#admin-credentials) before you delete or rotate either one. Optional features each need something created **before** you install: @@ -95,17 +103,17 @@ Optional features each need something created **before** you install: ## Installation -Once both secrets exist, install the template into your GVC, naming the secrets and replacing the database password placeholder: +Once the three secrets exist, install the template into your GVC, naming the secrets: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/grafana \ - --version 1.4.1 --gvc GVC_NAME \ + --version 1.5.0 --gvc GVC_NAME \ + --set postgresHA.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME \ --set admin.passwordSecretName=ADMIN_PASSWORD_SECRET_NAME \ - --set admin.secretKeySecretName=SECRET_KEY_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' + --set admin.secretKeySecretName=SECRET_KEY_SECRET_NAME ``` -Prefer a values file (`--values`) over `--set` for the database password if you do not want it in your shell history. If you run more than one release of this template in the same org, also give each its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName` in single-instance mode) — see [App Database](#app-database). +In single-instance mode set `postgres.config.credentialsSecretName` instead — see [App Database](#app-database). @@ -347,7 +355,7 @@ Exactly one of the two database modes must be enabled — the chart enforces thi postgresHA: # default: highly available PostgreSQL app database enabled: true config: - credentialsSecretName: my-grafana-db-credentials # dictionary secret this chart CREATES from postgres.credentials.*; org-wide name — give each release its own + credentialsSecretName: my-grafana-db-credentials # PREREQUISITE dictionary secret (username, password, database), must exist BEFORE install replicas: 3 volumeset: capacity: 10 # initial capacity in GiB per replica (minimum is 10) @@ -358,19 +366,14 @@ postgresHA: enabled: false # set postgresHA.enabled: true (the default) for automatic database failover postgres: # dev/lightweight: single-instance PostgreSQL enabled: true - credentials: # the chart writes these into the secret named below — nothing for you to create - username: grafana - password: change-me-grafana-db-password # change before installing - database: grafana config: - credentialsSecretName: my-grafana-db-credentials # secret names are org-wide — give each release its own + credentialsSecretName: my-grafana-db-credentials # PREREQUISITE dictionary secret (username, password, database), must exist BEFORE install volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) ``` -In **both** modes the database credentials come from `postgres.credentials.*` — the chart writes them into the dictionary secret named by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` (whichever database is enabled), and the database reads that secret. Change `postgres.credentials.password` before installing even when `postgres.enabled` is `false`. Grafana is wired to the active database automatically: the HAProxy leader endpoint in HA mode, or the single instance directly. In HA mode the proxy must stay enabled — that endpoint is Grafana's stable database address, and the chart refuses to render with it disabled. +In **both** modes the database credentials come from your prerequisite dictionary secret (`username`, `password` and `database` — see [Prerequisites](#prerequisites)), named by the `credentialsSecretName` of the database you enable. Grafana is wired to the active database automatically: the HAProxy leader endpoint in HA mode, or the single instance directly. In HA mode the proxy must stay enabled — that endpoint is Grafana's stable database address, and the chart refuses to render with it disabled. -Secret names are org-wide, so a second release of this template left on the default `credentialsSecretName` is **refused at install** with `cannot be updated because it is being managed by a different release` and creates nothing — the first release is unaffected. Give each release its own name. Everything else the bundled databases can do — volume autoscaling, PgBouncer pooling, etcd tuning — is documented on the [postgres-highly-available](/template-catalog/templates/postgres-highly-available) and [postgres](/template-catalog/templates/postgres) pages and is set under the same `postgresHA.*` / `postgres.*` prefixes. Their backup settings are covered under [Backing Up](#backing-up). @@ -590,7 +593,7 @@ In HA mode `postgresHA.backup.mode` selects `logical` (a scheduled `pg_dump` run ### Restoring a Backup -A restore goes into the **database**, not into Grafana: everything that makes up your Grafana lives there, so restoring it brings back dashboards, users, datasources and alert rules together. The restore procedures belong to the bundled database — follow [Restoring a Backup](/template-catalog/templates/postgres-highly-available#restoring-a-backup) on the `postgres-highly-available` page in HA mode, or [Restoring a Backup](/template-catalog/templates/postgres#restoring-a-backup) on the `postgres` page in single-instance mode, substituting this release's database address (`RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local` or `RELEASE_NAME-postgres.GVC_NAME.cpln.local`, port `5432`) and the credentials from the chart-created secret: +A restore goes into the **database**, not into Grafana: everything that makes up your Grafana lives there, so restoring it brings back dashboards, users, datasources and alert rules together. The restore procedures belong to the bundled database — follow [Restoring a Backup](/template-catalog/templates/postgres-highly-available#restoring-a-backup) on the `postgres-highly-available` page in HA mode, or [Restoring a Backup](/template-catalog/templates/postgres#restoring-a-backup) on the `postgres` page in single-instance mode, substituting this release's database address (`RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local` or `RELEASE_NAME-postgres.GVC_NAME.cpln.local`, port `5432`) and the credentials from your database credentials secret: ```bash cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml @@ -602,6 +605,56 @@ cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml Restore into a release that uses the **same encryption key**: `admin.secretKeySecretName` decrypts the datasource credentials stored in the dump, and a different key makes every saved datasource credential unreadable. +### Rotating the Database Password + +PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with, so in single-instance mode a rotation takes three steps in order (in HA mode follow [Rotating the Database Password in HA Mode](#rotating-the-database-password-in-ha-mode) instead): + +1. Change it in the database: `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` (`DB_USERNAME` is the secret's `username`; connect as that user from inside the GVC). +2. Update the secret — export it, change only `password`, and apply it: + + ```bash + cpln secret reveal SECRET_NAME -o yaml > secret.yaml + # edit only the password key in secret.yaml + cpln apply -f secret.yaml + ``` + +3. Right away, redeploy Grafana: + + ```bash + cpln workload force-redeployment RELEASE_NAME-grafana --gvc GVC_NAME + ``` + +Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. A new value in the secret without `ALTER ROLE` breaks Grafana's login to the database. + +### Rotating the Database Password in HA Mode + +The HA database (`postgresHA`) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Change BOTH roles on -0 +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +# 4. Right after, put the same password in the secret (edit only the password key in secret.yaml) +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +cpln apply -f secret.yaml +# 5. Restart the members +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 6. Once the restart completes, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 7. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-grafana`, which keeps the old password until it restarts. The database drops connections briefly at each switchover, and Grafana cannot open new connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + ### Upgrading from 1.1.0 or Earlier Version 1.2.0 moved the admin password and the encryption key out of Helm values (`admin.password`, `admin.secretKey`) into the two prerequisite [opaque secrets](/guides/create-secret/opaque) described under [Prerequisites](#prerequisites), and stopped creating the chart-owned admin secret those values used to feed. Version 1.1.0 had already renamed `resources.cpu` and `resources.memory` to `resources.maxCpu` and `resources.maxMemory`. The old keys are not rejected at render — they are simply ignored — so remove them deliberately. @@ -630,7 +683,7 @@ Template versions before 1.2.1 did not compact the etcd cluster inside the bundl ### Upgrading from 1.2.1 or Earlier -Version 1.3.0 moved the single-instance database credentials from `postgres.config.username`, `postgres.config.password` and `postgres.config.database` to `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database`, named by the new `postgres.config.credentialsSecretName`. `postgres.backup.minio.accessKey` and `postgres.backup.minio.secretKey` were replaced by `postgres.backup.minio.credentialsSecretName` at the same time. Carrying the old keys stops the upgrade at render, before anything is applied: +Version 1.3.0 removed the single-instance database credential keys `postgres.config.username`, `postgres.config.password` and `postgres.config.database`. `postgres.backup.minio.accessKey` and `postgres.backup.minio.secretKey` were replaced by `postgres.backup.minio.credentialsSecretName` at the same time. Carrying the old keys stops the upgrade at render, before anything is applied: ```text Error: ... config.username was REMOVED in postgres 3.4.0. Database credentials are no longer values: @@ -638,11 +691,11 @@ create a `dictionary` secret holding the keys `username`, `password` and `databa config.credentialsSecretName to its name. See Prerequisites in the README. ``` -Move the three keys to `postgres.credentials.*` **using the values the database was initialised with** — PostgreSQL enforces the credentials written when the volume was first created, and different values just leave Grafana unable to authenticate. **Ignore the message's advice to create the secret yourself:** this template creates it from `postgres.credentials.*`. If you back up to MinIO, create the dictionary secret described under [Backing Up](#backing-up) and set `postgres.backup.minio.credentialsSecretName`. The HA path, Redis and the admin and encryption-key secrets are unchanged by this step. +Put the three values into the app-database credentials secret under [Prerequisites](#prerequisites) **using the values the database was initialized with** — PostgreSQL enforces the credentials written when the volume was first created, and different values just leave Grafana unable to authenticate — set `postgres.config.credentialsSecretName` to its name, and remove the three keys. If you back up to MinIO, create the dictionary secret described under [Backing Up](#backing-up) and set `postgres.backup.minio.credentialsSecretName`. The HA path, Redis and the admin and encryption-key secrets are unchanged by this step. ### Upgrading from 1.3.0 -Version 1.4.0 made the same change on the HA path: `postgresHA.postgres.username`, `postgresHA.postgres.password` and `postgresHA.postgres.database` were removed, and the bundled highly available database now reads the dictionary secret named by `postgresHA.config.credentialsSecretName` — which this chart creates from `postgres.credentials.*`, exactly as in single-instance mode. Carrying the old block stops the upgrade at render: +Version 1.4.0 made the same change on the HA path: `postgresHA.postgres.username`, `postgresHA.postgres.password` and `postgresHA.postgres.database` were removed, and the bundled highly available database now reads the dictionary secret named by `postgresHA.config.credentialsSecretName`. Carrying the old block stops the upgrade at render: ```text Error: ... postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres.username`, @@ -651,7 +704,7 @@ Error: ... postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres:` block from your values. See Prerequisites in the README. ``` -Delete `postgresHA.postgres.*`, and set `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database` to the **same values the cluster already has** (the message's instruction to create the secret yourself does not apply — this template creates it). Leaving `postgres.enabled: false` is correct; the `postgres.credentials.*` values feed both database modes. If you run several releases in one org, give each its own `postgresHA.config.credentialsSecretName`. +Delete `postgresHA.postgres.*`, put the **same values the cluster already has** into the app-database credentials secret under [Prerequisites](#prerequisites), and set `postgresHA.config.credentialsSecretName` to its name. If you run several releases in one org, give each its own secret. ### Upgrading from 1.4.0 @@ -665,6 +718,23 @@ backup.minio.credentialsSecretName to its name. See Backup setup in the README. Create the dictionary secret described under [Backing Up](#backing-up) with the same access key and secret key, set `postgresHA.backup.minio.credentialsSecretName` to its name, remove the two old keys, and upgrade. +### Upgrading from 1.4.1 + +Version 1.5.0 removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the bundled database template requires on its own. Carrying the old key forward fails the render. + +1.4.1 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with, not new ones, or Grafana loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-grafana-db \ + --entry username=grafana \ + --entry password='YOUR_CURRENT_PASSWORD' \ + --entry database=grafana +``` + +Then upgrade with `postgres.credentials` removed and the `credentialsSecretName` of the mode you use (`postgresHA.config.credentialsSecretName` by default) set to that name. Your current values are in `postgres.credentials` of the 1.4.1 release, and also in the old secret (`cpln secret reveal my-grafana-db-credentials -o yaml`, or the name your release set in `credentialsSecretName`) until the upgrade deletes it. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Grafana can briefly lose its database; it reconnects by itself. The new version of a workload can also wait several minutes with `is not allowed to reveal the secret` in `get-deployments` while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. + ### Scaling and Availability - **`replicas`** scales the Grafana tier; `2` or more requires `redis.enabled: true` — see [Alerting High Availability](#alerting-high-availability). Database replicas are set separately with `postgresHA.replicas`; the single-instance database must stay at one replica. @@ -679,7 +749,7 @@ Create the dictionary secret described under [Backing Up](#backing-up) with the **Symptom:** `cpln helm install` succeeded, but `RELEASE_NAME-grafana` shows no containers and `cpln logs` returns nothing. - **Cause:** A prerequisite secret named in your values does not exist — the admin password, the encryption key, a datasource-credential secret or the SMTP password. The deployment pauses on the unresolved reference. + **Cause:** A prerequisite secret named in your values does not exist — the app-database credentials, the admin password, the encryption key, a datasource-credential secret or the SMTP password. The deployment pauses on the unresolved reference. **Fix:** Read the message naming the secret, then create it: @@ -710,13 +780,6 @@ Create the dictionary secret described under [Backing Up](#backing-up) with the **Fix:** Enable exactly one — see [App Database](#app-database). To use the single-instance database, set `postgresHA.enabled: false` together with `postgres.enabled: true`. - - **Symptom:** Installing a second release of this template in the same org fails with `cannot be updated because it is being managed by a different release`, naming the database credentials secret, and creates nothing. - - **Cause:** Secret names are org-wide, and both releases are trying to create the same `credentialsSecretName` (default `my-grafana-db-credentials`). The first release owns it. - - **Fix:** Give the new release its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName` in single-instance mode) and install again. The first release is unaffected. - **Symptom:** Datasources that worked before now fail their connection test, and Grafana cannot read the credentials it stored for them. @@ -732,23 +795,24 @@ Create the dictionary secret described under [Backing Up](#backing-up) with the **Fix:** Set `publicAccess.enabled: true` and upgrade, or use contact points that do not rely on absolute links. - **Symptom:** `helm upgrade` stops at render with `config.username was REMOVED in postgres 3.4.0`, `the postgres block was REMOVED in 2.5.0`, or `backup.minio.accessKey and backup.minio.secretKey were REMOVED`. + **Symptom:** `helm upgrade` stops at render with `config.username was REMOVED in postgres 3.4.0`, `the postgres block was REMOVED in 2.5.0`, `backup.minio.accessKey and backup.minio.secretKey were REMOVED`, or a message that `postgres.credentials` was removed. - **Cause:** Your values still carry database or MinIO credential keys that moved in template versions 1.3.0, 1.4.0 or 1.4.1. The existing release is left untouched and running. + **Cause:** Your values still carry database or MinIO credential keys that moved in template versions 1.3.0, 1.4.0, 1.4.1 or 1.5.0. The existing release is left untouched and running. - **Fix:** Follow the matching migration note above — [Upgrading from 1.2.1 or Earlier](#upgrading-from-1-2-1-or-earlier), [Upgrading from 1.3.0](#upgrading-from-1-3-0) or [Upgrading from 1.4.0](#upgrading-from-1-4-0) — keeping the credentials the database already uses. + **Fix:** Follow the matching migration note above — [Upgrading from 1.2.1 or Earlier](#upgrading-from-1-2-1-or-earlier), [Upgrading from 1.3.0](#upgrading-from-1-3-0), [Upgrading from 1.4.0](#upgrading-from-1-4-0) or [Upgrading from 1.4.1](#upgrading-from-1-4-1) — keeping the credentials the database already uses. ## Important Notes -- **Create the admin-password and encryption-key secrets before installing**, and change `postgres.credentials.password` from its `change-me` placeholder. A missing prerequisite secret does not fail the install — the workload sits at zero replicas waiting on it. +- **Create the app-database credentials, admin-password and encryption-key secrets before installing.** A missing prerequisite secret does not fail the install — the workload sits at zero replicas waiting on it, and `status.versions[].message` from `cpln workload get-deployments RELEASE_NAME-grafana --gvc GVC_NAME -o yaml` names it. +- **Rotating the database password needs `ALTER ROLE` plus a forced redeployment of Grafana** (and a leader switchover first in HA mode) — see [Rotating the Database Password](#rotating-the-database-password). - **The encryption-key secret is permanent: never delete it, never rotate it.** Back the key up outside Control Plane. - **The admin password applies on first boot only.** Change it in the Grafana UI; then set `admin.applyPassword: false`, upgrade, and delete the password secret — see [First Run](#first-run). - **This template ships no dashboards and no datasources.** Control Plane's console already provides built-in workload-metrics dashboards; this is the pane for data you own. - **`replicas >= 2` requires `redis.enabled: true`** — the chart refuses to render multi-replica without the Sentinel tier that keeps one firing alert from producing one notification per replica. - **Dashboards, users and alert rules live in the app database** and survive Grafana restarts and replica replacement. **Uninstall deletes the database volume sets** — enable backups if the data matters. -- **Give each release its own `credentialsSecretName`** — secret names are org-wide, and a second release on the default name is refused at install. +- **Give each release its own app-database credentials secret** — secret names are org-wide. - **Prerequisite secrets you create are not owned by the release** — they survive `helm uninstall` and must be deleted manually if you no longer need them. - **Keep `publicAccess.enabled: true` if you rely on links in alert emails** — an internal-only instance has no derived `root_url` and links point at localhost. - **This template ships Grafana OSS only** — Enterprise features such as fine-grained RBAC, reporting and query caching are not available. diff --git a/template-catalog/templates/guacamole.mdx b/template-catalog/templates/guacamole.mdx index 76e682f1..a4f7ca60 100644 --- a/template-catalog/templates/guacamole.mdx +++ b/template-catalog/templates/guacamole.mdx @@ -1,7 +1,7 @@ --- title: Guacamole icon: /template-catalog/templates/icons/guacamole.png -description: Deploy Apache Guacamole on Control Plane. A clientless remote desktop gateway that opens RDP, VNC, SSH and telnet sessions to internal machines in a browser tab. Single replica with a bundled PostgreSQL, private by default. Needs an existing GVC and a pre-created admin dictionary secret. +description: Deploy Apache Guacamole on Control Plane. A clientless remote desktop gateway that opens RDP, VNC, SSH and telnet sessions to internal machines in a browser tab. Single replica with a bundled PostgreSQL, private by default. Needs an existing GVC and pre-created admin and database dictionary secrets. keywords: ['guacd', 'browser rdp', 'rdp gateway', 'vnc gateway', 'ssh in browser', 'html5 remote access', 'bastion host', 'jump host', 'vpn alternative', 'audited remote access', 'teleport alternative', 'remote desktop'] --- @@ -9,7 +9,7 @@ keywords: ['guacd', 'browser rdp', 'rdp gateway', 'vnc gateway', 'ssh in browser Apache Guacamole is a clientless remote desktop gateway: users open RDP, VNC, SSH, and telnet sessions to internal machines in a plain browser tab, with no client, plugin, or VPN to install. This template deploys the Guacamole web application together with the `guacd` protocol daemon in one workload, backed by a bundled PostgreSQL that holds users, connections, permissions, and session history. The gateway runs as a single replica and is **private by default** — you reach the UI over a port forward and opt in to publishing it. -The administrator login is not a template value. It comes from a dictionary secret you create before installing, so it never passes through Helm and never lands in the release. +The administrator login and the database credentials are not template values. They come from dictionary secrets you create before installing, so they never pass through Helm and never land in the release. This template deploys into an existing [GVC](/reference/gvc) that you already have — it does not create or manage a GVC. @@ -23,15 +23,14 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-postgres` | Bundled PostgreSQL from the [postgres](/template-catalog/templates/postgres) template — users, connections, connection parameters, permissions, and session history (stateful, 1 replica) | — | | Workload | `RELEASE_NAME-postgres-backup` | Cron job that dumps the database to object storage on a schedule | `postgres.backup.enabled: true` | | Volume set | `RELEASE_NAME-pg-vs` | PostgreSQL data directory; takes no scheduled snapshots; a final snapshot is taken when the volume set is deleted and kept 7 days. The gateway workload itself mounts no volume set | — | -| Secret | the name in `postgres.config.credentialsSecretName` | Dictionary secret the chart creates (`username`, `password`, `database`) from `postgres.credentials.*` and hands to the bundled PostgreSQL | — | | Secret | `RELEASE_NAME-guacamole-start` | Opaque secret holding the start script of the `guacamole` container; contains no credential | — | | Secret | `RELEASE_NAME-guacamole-init` | Opaque secret holding the schema bootstrap script of the `schema-init` container; contains no credential | — | | Identity | `RELEASE_NAME-guacamole-identity` | Bound to the gateway workload; no cloud bindings | — | -| Policy | `RELEASE_NAME-guacamole-policy` | Grants that identity `reveal` on exactly four secrets: your admin secret, the database credentials secret, and the two script secrets | — | +| Policy | `RELEASE_NAME-guacamole-policy` | Grants that identity `reveal` on exactly four secrets: your admin and database credentials secrets, and the two script secrets | — | | Identity | `RELEASE_NAME-pg-identity` | Bound to the bundled PostgreSQL (and its backup job) | — | | Policy | `RELEASE_NAME-pg-policy` | Grants the PostgreSQL identity `reveal` on the database credentials secret only | — | -Your admin secret is **not** created by the template — see [Prerequisites](#prerequisites). +Your admin and database credentials secrets are **not** created by the template — see [Prerequisites](#prerequisites). `guacd` listens on port `4822` and that port is deliberately **not published**. The protocol daemon is unauthenticated — anything that can reach it can drive an arbitrary RDP, VNC, or SSH session — so only the web application running beside it in the same workload can reach it. Do not "fix" this by declaring the port. @@ -39,9 +38,21 @@ Your admin secret is **not** created by the template — see [Prerequisites](#pr ## Prerequisites -**One secret must exist before you install.** It holds the Guacamole administrator login. The values never pass through Helm, so they do not land in the release. Secrets are org-level, so no GVC flag is involved. +**Two secrets must exist before you install.** They hold the database credentials and the Guacamole administrator login. The values never pass through Helm, so they do not land in the release. Secrets are org-level, so no GVC flag is involved. + + A [dictionary secret](/guides/create-secret/dictionary) holding `username`, `password` and `database`. The bundled PostgreSQL initializes with them, and Guacamole and its `schema-init` container connect with them: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=guacamole \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=guacamole + ``` + + Set `postgres.config.credentialsSecretName` to this name. Secret names are org-wide, so give each release its own. PostgreSQL reads it only when its data directory is first initialized — see [Bundled PostgreSQL](#bundled-postgresql) for changing the password later. + A [dictionary secret](/guides/create-secret/dictionary) holding exactly two keys — `username` and `password`. They become the Guacamole administrator account, replacing the well-known stock `guacadmin` account before the web application ever serves a request: @@ -63,7 +74,7 @@ Your admin secret is **not** created by the template — see [Prerequisites](#pr -**Create the secret before installing, or the deployment wedges silently.** The template refuses to render when `admin.secretName` is blank, but a name that points at a secret which does not exist installs "successfully" and then never starts. No container runs, so `cpln logs` returns **zero lines**. The one place the reason appears is `status.versions[].message`: +**Create both secrets before installing, or the deployment wedges silently.** The template refuses to render when `admin.secretName` is blank, but a name that points at a secret which does not exist installs "successfully" and then never starts. A missing database credentials secret also wedges `RELEASE_NAME-postgres`. No container runs, so `cpln logs` returns **zero lines**. The one place the reason appears is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-guacamole --gvc GVC_NAME -o yaml @@ -81,21 +92,19 @@ Use `get-deployments` — plain `cpln workload get` has no `versions` key. Creat **The administrator credentials are applied on first boot only.** They seed the database once. Rotating the secret afterwards does not change your login — change the password in the Guacamole UI instead (see [First Run](#first-run)). -Nothing else is required for a default install: the bundled database password is internal plumbing that this template turns into a secret for you. Scheduled database backups need a bucket and, for AWS or GCP, a Control Plane cloud account — see [Backing Up](#backing-up). +Nothing else is required for a default install. Scheduled database backups need a bucket and, for AWS or GCP, a Control Plane cloud account — see [Backing Up](#backing-up). ## Installation -Once your admin secret exists, install from the marketplace registry, pointing `admin.secretName` at it and replacing the bundled database password: +Once both secrets exist, install from the marketplace registry, pointing `admin.secretName` and `postgres.config.credentialsSecretName` at them: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/guacamole \ - --version 1.0.0 --gvc GVC_NAME \ + --version 1.1.0 --gvc GVC_NAME \ --set admin.secretName=ADMIN_SECRET_NAME \ - --set postgres.credentials.password='YOUR-DB-PASSWORD' + --set postgres.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME ``` -Installing a second Guacamole release in the same org? Also set `postgres.config.credentialsSecretName` to a name no other release uses — secret names are org-wide. - Other ways to install and manage the template: @@ -188,17 +197,13 @@ These knobs govern **inbound** traffic to Guacamole. They do not restrict which ### Bundled PostgreSQL -The database is the [postgres](/template-catalog/templates/postgres) template as a subchart, so every knob of that template is available under `postgres.*`. The chart builds the credentials secret from `postgres.credentials.*` for you — nothing to create before installing. `postgres.image` also sets the image of the `schema-init` container, so `psql` always matches the server version. +The database is the [postgres](/template-catalog/templates/postgres) template as a subchart, so every knob of that template is available under `postgres.*`. The database credentials come from your prerequisite dictionary secret (`username`, `password` and `database` — see [Prerequisites](#prerequisites)), named by `postgres.config.credentialsSecretName`. `postgres.image` also sets the image of the `schema-init` container, so `psql` always matches the server version. ```yaml postgres: image: postgres:18 # also used by the schema-init container, so psql matches the server - credentials: # the chart writes these into the secret named below - username: guacamole - password: change-me-guacamole-db # change before installing — seeds the database on first boot - database: guacamole config: - credentialsSecretName: my-guacamole-db-credentials # chart-created; secret names are org-wide, so give each release its own + credentialsSecretName: my-guacamole-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 250m maxCpu: 1000m @@ -213,7 +218,7 @@ postgres: ``` -`postgres.credentials.password` seeds the database on first boot and is not updated by later value edits. Uninstalling the release deletes the volume set; reinstall to reset. +PostgreSQL reads the credentials secret only when its data directory is first initialized, and a running workload keeps the values it started with. To change the password later, run `ALTER ROLE` first, then update the secret (export it with `cpln secret reveal SECRET_NAME -o yaml > secret.yaml`, change only `password`, and `cpln apply -f secret.yaml`), and right away force a redeployment of the gateway with `cpln workload force-redeployment RELEASE_NAME-guacamole --gvc GVC_NAME`. Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. Uninstalling the release deletes the volume set; reinstall to reset. ## Connecting @@ -411,6 +416,23 @@ Two constraints make this a planned operation: the archive contains `CREATE ROLE This restore path has not been exercised against this template end to end. Rehearse it on a throwaway release before you depend on it. Restoring from a volume set snapshot is likewise unverified here. +### Upgrading from 1.0.0 + +`postgres.credentials` was removed in 1.1.0: the database credentials are now a prerequisite secret, the same one the `postgres` template requires on its own. Carrying the old key forward fails the render. + +1.0.0 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Guacamole loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-guacamole-db \ + --entry username=guacamole \ + --entry password='YOUR_CURRENT_PASSWORD' \ + --entry database=guacamole +``` + +Then upgrade with `postgres.config.credentialsSecretName` set to that name and `postgres.credentials` removed. Your current values are in `postgres.credentials` of the 1.0.0 release, and also in the old secret (`cpln secret reveal my-guacamole-db-credentials -o yaml`, or the name you gave it) until the upgrade deletes it. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so expect a short database outage and dropped sessions; Guacamole reconnects by itself. The new version of a workload can also wait several minutes with `is not allowed to reveal the secret` in `get-deployments` while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. The admin secret is unchanged. + ### Scaling and Availability This template runs **exactly one replica**, and there is no `replicas` knob. Guacamole keeps authentication tokens in each web application's memory with no cross-instance sharing, so a second replica would randomly sign users out. @@ -429,7 +451,7 @@ Rotating a secret the workload references does not restart it; the old value sta **Symptom:** `cpln helm install` reported success, `cpln logs` returns nothing, and `cpln workload get-deployments RELEASE_NAME-guacamole --gvc GVC_NAME -o yaml` shows `The secret ADMIN_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` under `status.versions[].message`. - **Cause:** The prerequisite admin secret named by `admin.secretName` does not exist. + **Cause:** A prerequisite secret does not exist — the admin secret named by `admin.secretName` or the database credentials secret named by `postgres.config.credentialsSecretName`. The message names the missing one. **Fix:** Create it (see [Prerequisites](#prerequisites)). The deployment recovers on its own once the secret exists; `cpln workload force-redeployment RELEASE_NAME-guacamole --gvc GVC_NAME` skips the wait. @@ -444,13 +466,6 @@ Rotating a secret the workload references does not restart it; the old value sta cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-guacamole"} |= "schema-init"' --limit 50 --since 10m ``` - - **Symptom:** Installing a second Guacamole release fails with `cannot be updated because it is being managed by a different release` and creates nothing. - - **Cause:** Both releases use the same `postgres.config.credentialsSecretName`. Secret names are org-wide, and the first release owns that one. Nothing is shared, overwritten, or deleted. - - **Fix:** Set `postgres.config.credentialsSecretName` to a distinct name for the second release and install again. - **Symptom:** You changed `username` or `password` in the admin secret (and even forced a redeployment), yet the original credentials still sign in and the new ones are rejected. @@ -490,7 +505,7 @@ Rotating a secret the workload references does not restart it; the old value sta ## Important Notes -- **Create the admin secret before installing** — `admin.secretName` must name an existing dictionary secret with `username` and `password`. A missing secret wedges the deployment silently; see [Prerequisites](#prerequisites) for the one command that shows the reason. +- **Create both secrets before installing** — `admin.secretName` must name an existing dictionary secret with `username` and `password`, and `postgres.config.credentialsSecretName` one with `username`, `password` and `database`. A missing secret wedges the deployment silently; see [Prerequisites](#prerequisites) for the one command that shows the reason. - **The admin credentials are first-boot only.** Rotating the secret afterwards does not change your login. Change the password in the Guacamole UI instead. - **`guacadmin` / `guacadmin` is never valid here** — the stock account is replaced by yours before the web application starts. - **Rotating any referenced secret does not restart the workload.** The old value stays in force until you run `cpln workload force-redeployment RELEASE_NAME-guacamole --gvc GVC_NAME`. @@ -500,8 +515,8 @@ Rotating a secret the workload references does not restart it; the old value sta - **Only RDP, VNC, SSH, and telnet are available** in `guacd:1.6.0`. The Kubernetes protocol is not compiled in. - **Use fully qualified internal names** — `WORKLOAD_NAME.GVC_NAME.cpln.local` — both for reaching Guacamole from other workloads and for the connection targets you configure in it. - **Keep `guacamole.image` and `guacd.image` on the same tag** — the web application and the protocol daemon are versioned together upstream. -- **Change `postgres.credentials.password` before installing** and give each release its own `postgres.config.credentialsSecretName` — secret names are org-wide, and a second release on the same name is refused at install. -- **Uninstalling deletes the volume set** and every user, connection, and history row in it; your admin secret is yours and survives. +- **Rotating the database password needs `ALTER ROLE` first and a forced redeployment of `RELEASE_NAME-guacamole`**, not the database — see [Bundled PostgreSQL](#bundled-postgresql). Give each release its own database credentials secret; secret names are org-wide. +- **Uninstalling deletes the volume set** and every user, connection, and history row in it; your prerequisite secrets are yours and survive. ## External References diff --git a/template-catalog/templates/infisical.mdx b/template-catalog/templates/infisical.mdx index c733ae58..3fe0d58c 100644 --- a/template-catalog/templates/infisical.mdx +++ b/template-catalog/templates/infisical.mdx @@ -1,7 +1,7 @@ --- title: Infisical icon: /template-catalog/templates/icons/infisical.png -description: Deploy Infisical, the open-source secrets-management platform with a web UI and REST API, on Control Plane. Single or multi-replica server over bundled PostgreSQL and Sentinel-backed HA Redis. Needs an existing GVC and a pre-created ENCRYPTION_KEY + AUTH_SECRET dictionary secret. +description: Deploy Infisical, the open-source secrets-management platform with a web UI and REST API, on Control Plane. Single or multi-replica server over bundled PostgreSQL and Sentinel-backed HA Redis. Needs an existing GVC plus pre-created root-key and database-credentials secrets. keywords: ['secret manager', 'vault alternative', 'hashicorp vault alternative', 'doppler alternative', 'secrets vault', 'env vars', 'api keys', 'self-hosted secrets', 'secret versioning', 'redis sentinel', 'server admin console', 'signup mode'] --- @@ -23,22 +23,33 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-sentinel` | Redis Sentinel monitoring master `mymaster`, so the server follows a Redis failover without reconfiguration (stateful, 3 replicas) | — | | Workload | `RELEASE_NAME-postgres-backup` | Cron job that dumps the database to object storage on a schedule | `postgres.backup.enabled: true` | | Volume set | `RELEASE_NAME-pg-vs` | PostgreSQL data directory; takes no scheduled snapshots; a final snapshot is taken when the volume set is deleted and kept 7 days | — | -| Secret | `RELEASE_NAME-infisical-db` | Dictionary secret the chart creates with the PostgreSQL username and password the server uses to assemble `DB_CONNECTION_URI` | — | -| Secret | the name in `postgres.config.credentialsSecretName` | Dictionary secret the chart creates (`username`, `password`, `database`) and hands to the bundled PostgreSQL | — | | Secret | `RELEASE_NAME-redis-config`, `RELEASE_NAME-sentinel-config` | Redis and Sentinel configuration files for the bundled cache | — | -| Secret | `RELEASE_NAME-redis-auth-password` | Redis password, created by the bundled redis template and wired into the server as `REDIS_PASSWORD` | `redis.redis.auth.password.enabled: true` | | Identity | `RELEASE_NAME-infisical-identity` | Bound to the server workload | — | -| Policy | `RELEASE_NAME-infisical-policy` | Grants that identity `reveal` on exactly the secrets the server reads: its database secret, your root-of-trust secret, and the SMTP and Redis password secrets when configured | — | +| Policy | `RELEASE_NAME-infisical-policy` | Grants that identity `reveal` on exactly the secrets the server reads: your database credentials secret, your root-of-trust secret, and the SMTP and Redis password secrets when configured | — | | Identity | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-redis-identity`, `RELEASE_NAME-sentinel-identity` | Bound to the bundled PostgreSQL (and its backup job), Redis, and Sentinel workloads | — | | Policy | `RELEASE_NAME-pg-policy`, `RELEASE_NAME-redis-policy`, `RELEASE_NAME-sentinel-policy` | Grant each bundled identity `reveal` on its own configuration or credentials secrets only | — | -Your root-of-trust secret holding `ENCRYPTION_KEY` and `AUTH_SECRET` is **not** created by the template — see [Prerequisites](#prerequisites). +Your root-of-trust secret holding `ENCRYPTION_KEY` and `AUTH_SECRET`, your database credentials secret and, when Redis auth is on, your Redis password secret are **not** created by the template — see [Prerequisites](#prerequisites). ## Prerequisites -Infisical encrypts every stored secret with an `ENCRYPTION_KEY` and signs every session and token with an `AUTH_SECRET`. Both are long-lived root-of-trust values, so you supply them through a [dictionary secret](/guides/create-secret/dictionary) created **before** installing; neither value ever passes through Helm values. +Two [dictionary secrets](/guides/create-secret/dictionary) must exist **before** installing. They hold credentials and key material, so they are prerequisite secrets rather than values — a value would sit in plaintext in the Helm release for the life of the install. + +Infisical encrypts every stored secret with an `ENCRYPTION_KEY` and signs every session and token with an `AUTH_SECRET`. Both are long-lived root-of-trust values; neither ever passes through Helm values. The bundled PostgreSQL takes its credentials from the second secret. + + A dictionary secret with exactly three keys. The bundled PostgreSQL initializes with these values and Infisical connects with them: + + ```bash + cpln secret create-dictionary --name my-infisical-db-credentials \ + --entry username=infisical \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=infisical + ``` + + Set `postgres.config.credentialsSecretName` to this name. Keep the password and database name URL-safe (letters, digits, `-`, `_`) — Infisical places them in a connection URL as-is. PostgreSQL reads the secret only when its data directory is first initialized; see [Important Notes](#important-notes) for how to change the password later. + A dictionary secret with exactly two keys: `ENCRYPTION_KEY`, a 16-byte hex string, and `AUTH_SECRET`, a 32-byte base64 string. @@ -56,7 +67,7 @@ Infisical encrypts every stored secret with an `ENCRYPTION_KEY` and signs every -Create the secret **before** installing. Installing without it reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message`: +Create the secrets **before** installing. Installing without one reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message` (also check `RELEASE_NAME-postgres` for the database secret, and `RELEASE_NAME-redis` and `RELEASE_NAME-sentinel` for the Redis password secret): ```bash cpln workload get-deployments RELEASE_NAME-infisical --gvc GVC_NAME -o yaml @@ -65,23 +76,24 @@ cpln workload get-deployments RELEASE_NAME-infisical --gvc GVC_NAME -o yaml The message reads `The secret INFISICAL_KEYS_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` Creating the secret lets the deployment recover on its own; `cpln workload force-redeployment RELEASE_NAME-infisical --gvc GVC_NAME` skips the wait. -Everything else works with the defaults. Two optional features need their own setup first: +Everything else works with the defaults. Three optional features need their own setup first: - **Authenticated SMTP** — a dictionary secret holding `SMTP_USERNAME` and `SMTP_PASSWORD`. See [Email](#email). +- **Redis authentication** — a dictionary secret holding the key `password`. See [Bundled Redis](#bundled-redis). - **Scheduled database backups** — a bucket and cloud account (or S3-compatible credentials) for the bundled PostgreSQL. See [Backing Up](#backing-up). ## Installation -Once the root-of-trust secret exists, install from the marketplace registry, pointing `secrets.name` at it and replacing the bundled database password: +Once both prerequisite secrets exist, install from the marketplace registry, pointing `secrets.name` and `postgres.config.credentialsSecretName` at them: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/infisical \ - --version 1.1.0 --gvc GVC_NAME \ + --version 1.2.1 --gvc GVC_NAME \ --set secrets.name=INFISICAL_KEYS_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' + --set postgres.config.credentialsSecretName=my-infisical-db-credentials ``` -Installing a second Infisical release in the same org? Also set `postgres.config.credentialsSecretName` to a name no other release uses — secret names are org-wide. +Installing a second Infisical release in the same org? Create a separate database credentials secret for it — secret names are org-wide. Other ways to install and manage the template: @@ -177,17 +189,13 @@ A change to either access knob takes a couple of minutes to take effect after th ### Bundled PostgreSQL -The database is the [postgres](/template-catalog/templates/postgres) template as a subchart, so every knob of that template is available under `postgres.*`. The chart builds the credentials secret from `postgres.credentials.*` for you — nothing to create before installing. +The database is the [postgres](/template-catalog/templates/postgres) template as a subchart, so every knob of that template is available under `postgres.*`. Its credentials come from the prerequisite database credentials secret you created (see [Prerequisites](#prerequisites)). ```yaml postgres: image: postgres:18 - credentials: # the chart writes these into the secret named below - username: infisical - password: change-me-infisical-pg # change before installing — seeds the database on first boot - database: infisical config: - credentialsSecretName: my-infisical-db-credentials # chart-created; secret names are org-wide, so give each release its own + credentialsSecretName: my-infisical-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 250m maxCpu: 1000m @@ -211,9 +219,9 @@ redis: image: redis:7.4 replicas: 3 auth: - password: - enabled: false # true = require AUTH on the data nodes; the password is wired into Infisical as REDIS_PASSWORD - value: change-me-infisical-redis # used only when enabled is true — change it then + fromSecret: + enabled: false # true = require AUTH, with the password from the secret below (wired into Infisical as REDIS_PASSWORD) + name: "" # dictionary secret with the key `password`, e.g. my-infisical-redis — must exist BEFORE install resources: cpu: 200m memory: 256Mi @@ -221,7 +229,7 @@ redis: minMemory: 128Mi sentinel: image: redis:7.4 - replicas: 3 + replicas: 3 # no `auth:` block — Sentinel auth (incl. a plaintext `auth.password`) is refused at render resources: cpu: 200m memory: 256Mi @@ -229,7 +237,14 @@ redis: minMemory: 128Mi ``` -Leave Sentinel authentication off: the server authenticates only to the data nodes, and the chart refuses to render if `redis.sentinel.auth` is enabled. +To require a Redis password, create a dictionary secret holding the key `password` **before** you install or upgrade, then set `redis.redis.auth.fromSecret.enabled: true` and `redis.redis.auth.fromSecret.name` to its name. Redis, Sentinel and the Infisical server all read it. Leave auth off for the authless default behind the same-GVC firewall. + +```bash +cpln secret create-dictionary --name my-infisical-redis \ + --entry password="$(openssl rand -hex 24)" +``` + +Leave Sentinel authentication off: the server authenticates only to the data nodes, and the chart refuses to render with any `redis.sentinel.auth` setting, including a plaintext `redis.sentinel.auth.password`. To change the Redis password later, see [Rotating the Redis Password](#rotating-the-redis-password). ## Connecting @@ -240,7 +255,7 @@ Leave Sentinel authentication off: the server authenticates only to the data nod | Internal API (same GVC) | `http://RELEASE_NAME-infisical.GVC_NAME.cpln.local:8080` | For other workloads using the Infisical SDK, CLI, or agent; reachable per `internalAccess.type`. | | Login | The account created on first run | Infisical ships no default account. | | PostgreSQL | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | `username` / `password` / `database` from `cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is your `postgres.config.credentialsSecretName`. | -| Redis Sentinel | `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` | Master name `mymaster`; data nodes are unauthenticated unless `redis.redis.auth.password.enabled` is `true`. | +| Redis Sentinel | `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` | Master name `mymaster`; data nodes are unauthenticated unless `redis.redis.auth.fromSecret.enabled` is `true`, with the password in the `password` key of the secret named by `redis.redis.auth.fromSecret.name`. | With `publicAccess.enabled: false` there is no public endpoint; reach the UI from your machine through a tunnel instead and open `http://localhost:8080`: @@ -405,20 +420,68 @@ This restore path has not been exercised against this template end to end. Rehea ### Upgrading from 1.0.0 -Version 1.1.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. Infisical absorbed that change rather than passing it on — the chart now creates the credentials secret itself — so there is **no new prerequisite**, only a rename of three keys: +Version 1.1.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. Three keys were removed: -| Removed key (1.0.0) | Replacement (1.1.0) | +| Removed key (1.0.0) | Replacement (1.2.0 and later) | |---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | +| `postgres.config.username` | `username` key in the database credentials secret named by `postgres.config.credentialsSecretName` | +| `postgres.config.password` | `password` key in the same secret | +| `postgres.config.database` | `database` key in the same secret | -Move the three keys, keep the same values so the running database still matches, and upgrade in place. A values file that still carries an old key fails the render with the **postgres template's** message, which tells you to create a dictionary secret yourself — ignore that advice here; this template creates it. +Put the values your database was initialized with into the database credentials secret under [Prerequisites](#prerequisites), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. A values file that still carries an old key fails the render with the **postgres template's** message. The keys secret is unchanged. The first `helm upgrade` after an install can restart the bundled PostgreSQL even when its values are unchanged, so expect a brief interruption. +### Upgrading from 1.1.0 + +Version 1.2.0 removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the `postgres` template requires on its own. Carrying the old key forward fails the render. + +1.1.0 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Infisical loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-postgres-credentials \ + --entry username=infisical --entry password='YOUR_CURRENT_PASSWORD' --entry database=infisical +``` + +Then upgrade with `postgres.config.credentialsSecretName` set to that name and `postgres.credentials` removed. Your current values are in `postgres.credentials` of the 1.1.0 release, and also in the old secret (`cpln secret reveal my-infisical-db-credentials -o yaml`, or the name your release set in `credentialsSecretName`) until the upgrade deletes it. + +`redis.redis.auth.password` was removed the same way. With Redis auth off (the default) just delete the block. With it on, this upgrade deletes the `RELEASE_NAME-redis-auth-password` secret 1.1.0 had the redis chart create, so first create one under a **new** name holding your **current** Redis password: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-infisical-redis --entry password='YOUR_CURRENT_REDIS_PASSWORD' +``` + +Your current password is `redis.redis.auth.password.value` of the 1.1.0 release, and also in `cpln secret reveal RELEASE_NAME-redis-auth-password -o yaml` until the upgrade deletes it. Upgrade with `redis.redis.auth.fromSecret.enabled: true`, `redis.redis.auth.fromSecret.name` set to the new name, and the `redis.redis.auth.password` block removed. Version 1.2.1 also refuses any `redis.sentinel.auth` setting, so remove that block too if your values carry one. + +The upgrade restarts the bundled PostgreSQL (and, with auth on, Redis) too, since they now read the new secrets, so Infisical briefly loses its database; it reconnects by itself. The new version of a workload can also wait for several minutes, with `get-deployments` reporting that it is not allowed to reveal the secret, while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating the Redis Password + +Only with Redis auth on. Redis keeps no password on disk, but a running Sentinel holds the one it started with, so a new password is applied live in three places before Infisical restarts. Each Redis node first accepts both passwords, which keeps replication and every connected client working throughout. `rx` and `sx` run a command on one Redis node or Sentinel (`LOCATION` is your GVC's location; nodes and Sentinels are numbered from 0 up to `redis.redis.replicas` and `redis.sentinel.replicas` minus one): + +```bash +rx() { cpln workload exec RELEASE_NAME-redis --gvc GVC_NAME --location LOCATION \ + --container redis --replica "RELEASE_NAME-redis-$1" -- "${@:2}"; } +sx() { cpln workload exec RELEASE_NAME-sentinel --gvc GVC_NAME --location LOCATION \ + --container sentinel --replica "RELEASE_NAME-sentinel-$1" -- "${@:2}"; } +# 1. On EVERY Redis node: accept the new password alongside the old one, and replicate with it +rx 0 redis-cli --no-auth-warning -a OLD_PASSWORD ACL SETUSER default ">NEW_PASSWORD" +rx 0 redis-cli --no-auth-warning -a OLD_PASSWORD CONFIG SET masterauth NEW_PASSWORD +# 2. On EVERY Sentinel +sx 0 redis-cli -p 26379 SENTINEL SET mymaster auth-pass NEW_PASSWORD +# 3. Put the new password in the secret and read it back +cpln secret edit SECRET_NAME +cpln secret reveal SECRET_NAME -o yaml +# 4. Restart Infisical, which keeps the old password until it restarts +cpln workload force-redeployment RELEASE_NAME-infisical --gvc GVC_NAME +# 5. Once get-deployments shows a higher version, alone and ready: retire the old password on EVERY Redis node +rx 0 redis-cli --no-auth-warning -a NEW_PASSWORD ACL SETUSER default " **Symptom:** `cpln helm install` reported success, `cpln logs` returns nothing, and `cpln workload get-deployments RELEASE_NAME-infisical --gvc GVC_NAME -o yaml` shows `The secret INFISICAL_KEYS_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` under `status.versions[].message`. - **Cause:** The root-of-trust secret named by `secrets.name` does not exist (or, with SMTP auth on, the secret named by `smtp.auth.secretName` does not). + **Cause:** The root-of-trust secret named by `secrets.name` or the database credentials secret named by `postgres.config.credentialsSecretName` does not exist (or, with SMTP or Redis auth on, the secret named by `smtp.auth.secretName` or `redis.redis.auth.fromSecret.name` does not). **Fix:** Create it (see [Prerequisites](#prerequisites)). The deployment recovers on its own once the secret exists; `cpln workload force-redeployment RELEASE_NAME-infisical --gvc GVC_NAME` skips the wait. @@ -449,26 +512,26 @@ The bundled Redis is already highly available (three data nodes, three Sentinels **Fix:** Nothing — the retries stop once the database accepts connections and the workload becomes ready. If the lines keep coming, check `RELEASE_NAME-postgres` with `cpln workload get-deployments RELEASE_NAME-postgres --gvc GVC_NAME -o yaml`. - - **Symptom:** Installing a second Infisical release fails with `cannot be updated because it is being managed by a different release` and creates nothing. - - **Cause:** Both releases use the same `postgres.config.credentialsSecretName`. Secret names are org-wide, and the first release owns that one. Nothing is shared, overwritten, or deleted. - - **Fix:** Set `postgres.config.credentialsSecretName` to a distinct name for the second release and install again. - **Symptom:** The render fails with `config.username was REMOVED in postgres 3.4.0` (or the same message for `config.password` / `config.database`) and tells you to create a dictionary secret. **Cause:** Your values file still carries `postgres.config.username`, `postgres.config.password`, or `postgres.config.database`. - **Fix:** Rename them to `postgres.credentials.*` as described in [Upgrading from 1.0.0](#upgrading-from-1-0-0). Do not create a secret yourself — this template creates it. + **Fix:** Put their values into the database credentials secret and remove the keys, as described in [Upgrading from 1.0.0](#upgrading-from-1-0-0). + + + **Symptom:** The render fails with `postgres.credentials was removed in 1.2.0` or `infisical: redis.redis.auth.password was REMOVED in 1.2.0`. + + **Cause:** Your values file still carries the 1.1.0 plaintext database credentials or Redis password. + + **Fix:** Create the prerequisite secrets holding your current values and remove the old keys, as described in [Upgrading from 1.1.0](#upgrading-from-1-1-0). - **Symptom:** `infisical: redis.sentinel.auth must stay disabled — Infisical authenticates only the data nodes (REDIS_PASSWORD), not the sentinels`. + **Symptom:** `infisical: redis.sentinel.auth must stay disabled — Infisical authenticates only the Redis nodes (REDIS_PASSWORD), not the Sentinels`. - **Cause:** `redis.sentinel.auth.password.enabled` or `redis.sentinel.auth.fromSecret.enabled` is `true`. + **Cause:** Your values carry a `redis.sentinel.auth` setting — `redis.sentinel.auth.password.enabled`, a plaintext `redis.sentinel.auth.password.value`, or `redis.sentinel.auth.fromSecret.enabled`. - **Fix:** Turn Sentinel authentication off. To require a password on the data nodes, set `redis.redis.auth.password.enabled: true` and a value for `redis.redis.auth.password.value` instead. + **Fix:** Remove `redis.sentinel.auth` from your values. To require a password on the data nodes, use `redis.redis.auth.fromSecret` instead — see [Bundled Redis](#bundled-redis). **Symptom:** Signing in returns you to the login page, or emails and UI links use a host other than the one you browse to. @@ -481,15 +544,16 @@ The bundled Redis is already highly available (three data nodes, three Sentinels ## Important Notes -- **Create the root-of-trust secret before installing** — `secrets.name` must name an existing dictionary secret with `ENCRYPTION_KEY` and `AUTH_SECRET`. A missing secret wedges the deployment silently. +- **Create both prerequisite secrets before installing** — `secrets.name` must name an existing dictionary secret with `ENCRYPTION_KEY` and `AUTH_SECRET`, and `postgres.config.credentialsSecretName` one with `username`, `password` and `database`; with Redis auth on, the Redis password secret too. A missing secret wedges the deployment silently. - **`ENCRYPTION_KEY` and `AUTH_SECRET` are write-once — never rotate them.** A changed `ENCRYPTION_KEY` makes every stored secret unreadable; a changed `AUTH_SECRET` invalidates every session. Back both up outside Control Plane. - **Sign up right after install** — the first account becomes the server administrator and the form is open to anyone who can reach the endpoint. Then set **Signup mode** to **Disabled** in the Server Console. - **Redis is required, not optional** — the server does not boot without it, and Sentinel authentication must stay off. - **Keep `infisical.resources.maxMemory` at `2Gi`** — a cold boot with `infisical.replicas` of 2 or more runs out of memory at `1Gi`. -- **Change `postgres.credentials.password` before installing** — it seeds the database on first boot and is not updated by later value edits. -- **Give each release its own `postgres.config.credentialsSecretName`** — secret names are org-wide, and a second release on the same name is refused at install. +- **Changing the database password is a database change first.** Editing the secret alone does nothing to the database, and a running workload keeps the values it started with. Run `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` in PostgreSQL, then `cpln secret edit SECRET_NAME`, then right away `cpln workload force-redeployment RELEASE_NAME-infisical --gvc GVC_NAME`, and wait until `get-deployments` shows a higher version number, alone and ready (right after the command the status may not show the restart yet, and `ready: true` alone can still describe the old replica). Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. +- **Rotating the Redis password needs live changes on every Redis node and Sentinel before Infisical restarts** — see [Rotating the Redis Password](#rotating-the-redis-password). +- **Give each release its own database credentials secret** — secret names are org-wide. - **A database backup is only as good as the key** — a restored dump is unreadable without the `ENCRYPTION_KEY` it was written under. -- **Uninstalling deletes the PostgreSQL volume set** and every stored secret in it; your root-of-trust secret is yours and survives. +- **Uninstalling deletes the PostgreSQL volume set** and every stored secret in it; your prerequisite secrets are yours and survive. - **Set `smtp.requireTls: false` only for a plaintext mail catcher** — real providers on port `587` need it left `true`. ## External References diff --git a/template-catalog/templates/kafka.mdx b/template-catalog/templates/kafka.mdx index 6a75bc22..840760e4 100644 --- a/template-catalog/templates/kafka.mdx +++ b/template-catalog/templates/kafka.mdx @@ -21,11 +21,11 @@ This template deploys into an **existing** [GVC](/reference/gvc) — the one you | Volume Set | `RELEASE_NAME-logs-0`, `RELEASE_NAME-logs-1` | One per path in `kafka.logDirs`; ext4 SSD, autoscaling, daily snapshots | `kafka.volumes.logs.externalVolumeSets` mounts existing volume sets instead | | Secret | `RELEASE_NAME-controller-configuration` | Broker `server.properties` (listeners, SASL, ACL, replication defaults) | — | | Secret | `RELEASE_NAME-init` | Broker startup script | — | -| Secret | `RELEASE_NAME-secrets` | KRaft cluster ID, controller, inter-broker and listener admin passwords | — | +| Secret | `RELEASE_NAME-secrets` | KRaft cluster ID, controller, inter-broker and listener admin passwords (references to your credentials secret when `kafka.credentialsSecretName` is set) | — | | Secret | `RELEASE_NAME-jmx-exporter-conf` | JMX exporter rules | with `jmx_exporter` | -| Identity and Policy | `RELEASE_NAME` | `reveal` on the broker secrets | — | +| Identity and Policy | `RELEASE_NAME` | `reveal` on the broker secrets, and on your credentials secret when `kafka.credentialsSecretName` is set | — | | Workload | `RELEASE_NAME-kafbat-ui` | Kafbat UI web console (standard), with an identity and policy of the same name reading your configuration secret | `kafbat_ui.enabled: true` (default) | -| Workload | `RELEASE_NAME-rest-proxy` | Confluent REST Proxy (standard) on port 8082, with identity `RELEASE_NAME-rest-proxy-identity`, policy `RELEASE_NAME-rest-proxy-policy` and secrets `RELEASE_NAME-rest-properties`, `RELEASE_NAME-rest-jaas-conf` (plus `RELEASE_NAME-rest-password-properties` when BASIC-auth users are set) | `kafka_rest_proxy.enabled: true` (default) | +| Workload | `RELEASE_NAME-rest-proxy` | Confluent REST Proxy (standard) on port 8082, with identity `RELEASE_NAME-rest-proxy-identity`, policy `RELEASE_NAME-rest-proxy-policy` and secrets `RELEASE_NAME-rest-properties`, `RELEASE_NAME-rest-jaas-conf` (not created when `kafka_rest_proxy.jaas_conf_secret` names your own secret) and `RELEASE_NAME-rest-password-properties` (only when BASIC-auth users are set inline) | `kafka_rest_proxy.enabled: true` (default) | | Workload | `RELEASE_NAME-client` | Kafka CLI tools container for testing from inside the GVC | `kafka_client` block (default on; set to `null` to omit) | | Workload | `RELEASE_NAME-connect-` followed by the connector `name` | Kafka Connect worker (stateful). Containers: `kafka-connect` and `plugins-downloader`. Comes with a plugins volume set, identity and policy of the same name, and secrets with the suffixes `-props`, `-init` and `-download` | `kafka_connectors` list | | Domain | the hostname you configure | TLS entry point for the public listener, Kafbat UI or REST Proxy | `kafka.listeners..directReplicaRouting.publicAddress`, `kafbat_ui.domain`, `kafka_rest_proxy.domain` | @@ -34,7 +34,7 @@ This template deploys into an **existing** [GVC](/reference/gvc) — the one you - Kafbat UI is on by default and reads its whole configuration from an **opaque** secret mounted as a file. The secret must exist **before** you install. Replace `ADMIN_PASSWORD` with the value you will pass as `kafka.listeners.client.sasl.admin.password`, and `UI_PASSWORD` with the login you want for the console. The full file format is in the [Kafbat configuration reference](https://ui.docs.kafbat.io/configuration/configuration-file). + Kafbat UI is on by default and reads its whole configuration from an **opaque** secret mounted as a file. The secret must exist **before** you install. Replace `ADMIN_PASSWORD` with the client admin password (`client-admin-password` in your credentials secret, or `kafka.listeners.client.sasl.admin.password` if you set it inline), and `UI_PASSWORD` with the login you want for the console. The full file format is in the [Kafbat configuration reference](https://ui.docs.kafbat.io/configuration/configuration-file). ```bash cat > kafbat-config.yaml <<'EOF' @@ -69,14 +69,30 @@ This template deploys into an **existing** [GVC](/reference/gvc) — the one you Create the secret and the deployment recovers on its own after a few minutes, or force it with `cpln workload force-redeployment RELEASE_NAME-kafbat-ui --gvc GVC_NAME`. - - The KRaft cluster ID and the SASL passwords are template values, set at install time. Generate a cluster ID with the Kafka tooling (the same image the template runs): + + Every broker password can come from one [dictionary secret](/guides/create-secret/dictionary) named by `kafka.credentialsSecretName`, which keeps the passwords out of the Helm release. It holds `inter-broker-password`, `controller-password`, and one `--password` key for each SASL admin and user, where `` is the listener's `name` in lower case. For the default listeners: + + ```bash + cpln secret create-dictionary --name my-kafka-credentials \ + --entry inter-broker-password="$(openssl rand -hex 24)" \ + --entry controller-password="$(openssl rand -hex 24)" \ + --entry client-admin-password="$(openssl rand -hex 24)" \ + --entry client-user-password="$(openssl rand -hex 24)" + ``` + + Create it before you install: while it is missing the brokers wait and `cpln logs` shows nothing, and `status.versions[].message` in `cpln workload get-deployments RELEASE_NAME-cluster --gvc GVC_NAME -o yaml` names it. Without this secret, the passwords are the inline values `kafka.secrets.inter_broker_password`, `kafka.secrets.controller_password`, `kafka.listeners.client.sasl.admin.password` and `kafka.listeners.client.sasl.passwords`; their `your-…` defaults work as literal passwords, so never leave them in place. See [Credentials](#credentials). + + + The KRaft cluster ID is a template value, set at install time. Generate one with the Kafka tooling (the same image the template runs): ```bash docker run --rm apache/kafka:3.9.1 /opt/kafka/bin/kafka-storage.sh random-uuid ``` - Pick strong values for `kafka.secrets.inter_broker_password`, `kafka.secrets.controller_password`, `kafka.listeners.client.sasl.admin.password` and `kafka.listeners.client.sasl.passwords`. The `your-…` defaults in `values.yaml` are placeholders that work as literal passwords, so never leave them in place. The cluster ID is written into the log volumes at first start and cannot be changed afterwards without a new cluster. + The cluster ID is written into the log volumes at first start and cannot be changed afterwards without a new cluster. + + + The REST Proxy is on by default, and its JAAS file holds the client admin password. Put the whole file in an opaque secret named by `kafka_rest_proxy.jaas_conf_secret`; see [REST Proxy Credentials](#rest-proxy-credentials). Exposing brokers to clients outside the GVC needs a domain you control and a **dedicated load balancer** enabled on the GVC. Follow the [Configure Domain](/guides/configure-domain) guide first, then see [Public Listener](#public-listener) below. @@ -85,20 +101,19 @@ This template deploys into an **existing** [GVC](/reference/gvc) — the one you ## Installation -Install into an existing GVC, supplying the credentials from the prerequisites: +Install into an existing GVC, naming the credentials secret from the prerequisites: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/kafka \ - --version 4.3.0 --gvc GVC_NAME \ + --version 4.4.0 --gvc GVC_NAME \ --set kafka.secrets.kraft_cluster_id=KRAFT_CLUSTER_ID \ - --set kafka.secrets.inter_broker_password=INTER_BROKER_PASSWORD \ - --set kafka.secrets.controller_password=CONTROLLER_PASSWORD \ - --set kafka.listeners.client.sasl.admin.password=ADMIN_PASSWORD \ - --set kafka.listeners.client.sasl.passwords=USER_PASSWORD + --set kafka.credentialsSecretName=my-kafka-credentials ``` +Without a credentials secret, set the passwords inline instead of `kafka.credentialsSecretName`: `kafka.secrets.inter_broker_password`, `kafka.secrets.controller_password`, `kafka.listeners.client.sasl.admin.password` and `kafka.listeners.client.sasl.passwords`. They are then stored in the release values. + -The REST Proxy ships pointing at a bootstrap address and admin password that only match a release named `kafka-dev` with the default password. Either put `kafka_rest_proxy.properties.bootstrap.servers`, `kafka_rest_proxy.jaas_conf` and `kafka_rest_proxy.password_properties` in a values file (`-f values.yaml`) as shown under [Kafka REST Proxy](#kafka-rest-proxy), or add `--set kafka_rest_proxy.enabled=false`. +The REST Proxy ships pointing at a bootstrap address and admin password that only match a release named `kafka-dev` with the default password. Either put `kafka_rest_proxy.properties.bootstrap.servers`, `kafka_rest_proxy.jaas_conf_secret` (or `jaas_conf`) and `kafka_rest_proxy.password_properties_secret` (or `password_properties`) in a values file (`-f values.yaml`) as shown under [Kafka REST Proxy](#kafka-rest-proxy), or add `--set kafka_rest_proxy.enabled=false`. @@ -246,6 +261,27 @@ public: passwords: "your-public-user-password" ``` +### Credentials + +Every broker password can come from one dictionary secret instead of values; create it as in [Prerequisites](#prerequisites). + +```yaml +kafka: + credentialsSecretName: my-kafka-credentials # "" = the inline passwords are used + listeners: + client: + sasl: + admin: + username: admin # usernames stay in values; their passwords come from the secret + users: "user" +``` + +- When `credentialsSecretName` is set it wins over `sasl.admin.password`, `sasl.passwords`, `secrets.inter_broker_password` and `secrets.controller_password`, which are then ignored. The health sidecar, the Kafka exporter and the deprecated Kafka UI read the admin password from the secret too. +- The secret needs a `--password` key for every SASL admin and user on every listener, including a [public listener](#public-listener)'s `sasl.users`. +- The brokers resolve the passwords through Kafka's `env` config provider. If you set `config.providers` in `kafka.extra_configurations`, keep `env` in the list. +- **Changing a password:** update the secret, then run `cpln workload force-redeployment RELEASE_NAME-cluster --gvc GVC_NAME`. Running brokers keep the old password until they restart, and clients that log in with it need the new one. +- Inline passwords keep working: with `credentialsSecretName: ""` the brokers use the values below. + ### ACL and Cluster Secrets ```yaml @@ -384,9 +420,37 @@ kafka_rest_proxy: # user: your-user-password,user ``` +#### REST Proxy Credentials + +The REST Proxy's JAAS file holds a Kafka password. Put the whole file in an **opaque** secret and name it in `kafka_rest_proxy.jaas_conf_secret`; it then replaces `jaas_conf`. BASIC-auth users work the same way through `password_properties_secret`, which replaces `password_properties`: + +```yaml +kafka_rest_proxy: + jaas_conf_secret: my-kafka-rest-jaas # "" = jaas_conf is used + password_properties_secret: "" # e.g. my-kafka-rest-users: one "user: password,role" line per user +``` + +```bash +cat > kafka-rest.jaas.conf <<'EOF' +KafkaClient { + org.apache.kafka.common.security.plain.PlainLoginModule required + username="admin" + password="THE_CLIENT_ADMIN_PASSWORD"; +}; +KafkaRest { + org.eclipse.jetty.security.jaas.spi.PropertyFileLoginModule required + debug="true" + file="/etc/kafka-rest/password.properties"; +}; +EOF +cpln secret create-opaque --name my-kafka-rest-jaas --encoding plain -f kafka-rest.jaas.conf +``` + +After changing either secret, run `cpln workload force-redeployment RELEASE_NAME-rest-proxy --gvc GVC_NAME`. + ### Kafka Connect -`kafka_connectors` is a list; each entry deploys a Kafka Connect worker cluster named `RELEASE_NAME-connect-` plus the entry's `name`, with a volume set for plugins and a `plugins-downloader` container that fetches plugin artifacts into `plugins_folder` (see [Kafka Connect Plugins](#kafka-connect-plugins)). Connectors listed under `plugins` are created or updated through the Connect REST API when the worker starts; an entry with `enabled: false` is removed. The shipped `values.yaml` carries complete, commented examples for MirrorMaker 2, a Camel S3 sink, a ClickHouse sink and a Snowflake sink. +A connector worker's Kafka login and its plugins' passwords can come from secrets; see [Kafka Connect Credentials](#kafka-connect-credentials). `kafka_connectors` is a list; each entry deploys a Kafka Connect worker cluster named `RELEASE_NAME-connect-` plus the entry's `name`, with a volume set for plugins and a `plugins-downloader` container that fetches plugin artifacts into `plugins_folder` (see [Kafka Connect Plugins](#kafka-connect-plugins)). Connectors listed under `plugins` are created or updated through the Connect REST API when the worker starts; an entry with `enabled: false` is removed. The shipped `values.yaml` carries complete, commented examples for MirrorMaker 2, a Camel S3 sink, a ClickHouse sink and a Snowflake sink. ```yaml kafka_connectors: @@ -444,6 +508,24 @@ kafka_connectors: "topics": "your-topic" ``` +#### Kafka Connect Credentials + +Use Kafka's `env` config provider: add an `env` entry that references the secret (the chart grants the worker `reveal` on it) and use `${env:VAR}` in `connector_properties` or a plugin's `config`: + +```yaml +kafka_connectors: + - name: cluster + env: + - name: CONNECT_ADMIN_PASSWORD + value: cpln://secret/my-kafka-credentials.client-admin-password + connector_properties: + config.providers: env + config.providers.env.class: org.apache.kafka.common.config.provider.EnvVarConfigProvider + sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"admin\" password=\"${env:CONNECT_ADMIN_PASSWORD}\";" +``` + +In a plugin's `config`, write values with single quotes instead (`username='admin' password='${env:VAR}'`): plugin values are placed into the connector's JSON as they are, and a double quote breaks it. + #### Kafka Connect Plugins Every Kafka Connect replica runs two containers: `kafka-connect` and `plugins-downloader`. The downloader fetches every artifact of each plugin that has `enabled: true` onto the plugins volume set. Kafka Connect scans its plugins folder only once per start, so it waits until every artifact is in place before it starts. Each artifact is downloaded to a staging area on the volume, verified, and only then moved into its own directory (`///`), so in a normal start Kafka Connect never sees a half-written file. An artifact that is already installed and unchanged is not downloaded again, so a restart does not repeat the downloads. @@ -494,12 +576,12 @@ kafka_client: # Kafka CLI tools for testing from inside the GVC; set the whole b | What | Address | Notes | |---|---|---| -| Kafka bootstrap server (inside the GVC) | `RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092` | `security.protocol=SASL_PLAINTEXT`, `sasl.mechanism=PLAIN`, credentials from `kafka.listeners.client.sasl` | +| Kafka bootstrap server (inside the GVC) | `RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092` | `security.protocol=SASL_PLAINTEXT`, `sasl.mechanism=PLAIN`, usernames from `kafka.listeners.client.sasl`; passwords from your credentials secret (or the inline values) | | Kafka public listener | `-.:` per broker (direct replica routing) or `:3000`, `:3001`, … (multi-port) | `security.protocol=SASL_SSL`; users from the public listener's `sasl.users` | | Kafbat UI | `status.canonicalEndpoint` of `RELEASE_NAME-kafbat-ui`, or `kafbat_ui.domain` | Login from `spring.security.user` in your configuration secret | -| REST Proxy | `status.canonicalEndpoint` of `RELEASE_NAME-rest-proxy`, `kafka_rest_proxy.domain`, or `RELEASE_NAME-rest-proxy.GVC_NAME.cpln.local:8082` inside the GVC | HTTP BASIC auth with a user from `kafka_rest_proxy.password_properties` | +| REST Proxy | `status.canonicalEndpoint` of `RELEASE_NAME-rest-proxy`, `kafka_rest_proxy.domain`, or `RELEASE_NAME-rest-proxy.GVC_NAME.cpln.local:8082` inside the GVC | HTTP BASIC auth with a user from `kafka_rest_proxy.password_properties_secret` (or `password_properties`) | | Kafka Connect REST API | `RELEASE_NAME-connect-.GVC_NAME.cpln.local:8083` inside the GVC | Standard Connect REST API | -| Stored credentials | `cpln secret reveal RELEASE_NAME-secrets -o yaml` | KRaft cluster ID and the admin, controller and inter-broker passwords | +| Stored credentials | `cpln secret reveal my-kafka-credentials -o yaml` | The passwords, when `kafka.credentialsSecretName` is set. With inline passwords, `cpln secret reveal RELEASE_NAME-secrets -o yaml` holds them and the KRaft cluster ID | Read a canonical endpoint from the workload: @@ -596,6 +678,10 @@ Since 3.0.0 the chart only supports Apache Kafka images. A values file that stil - 4.2.x printed artifact URLs, including any credentials in them, in the downloader log; 4.3.0 redacts them. Consider rotating any credential that was part of an artifact URL. - Rolling back to 4.2.x works and the connectors keep running, but 4.2.x writes its `/.jar` files again beside the 4.3.0 directories (both load in one classloader) and its 60-second startup race returns. Upgrading to 4.3.0 again retires those files. +### Upgrading from 4.3.x + +4.4.0 adds the option to read every credential from a secret; inline passwords keep working unchanged, so nothing in your values has to change. To move a running cluster to the secret, put the passwords the cluster uses today in the credentials secret (see [Prerequisites](#prerequisites)), then upgrade with `kafka.credentialsSecretName` set. The brokers restart once. Move the REST Proxy JAAS file and BASIC-auth users the same way with `kafka_rest_proxy.jaas_conf_secret` and `password_properties_secret`. + ### Scaling and Availability - `kafka.replicas` accepts 1, 3, 4 or 5. Rendering fails with `Invalid value for kafka.replicas. It must be less than or equal to 5 and not equal to 2.` otherwise. Every broker is also a KRaft controller, so never shrink a running cluster below the number of controllers it was formed with — the quorum is lost and the cluster stops. @@ -672,11 +758,12 @@ cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-cluster"} |= "ERROR"' --limit ## Important Notes -- Replace every `your-…` placeholder (`kafka.secrets.*`, `kafka.listeners.client.sasl.*`) before installing; they are accepted as literal passwords. They are plain template values and therefore land in the Helm release. +- Set `kafka.credentialsSecretName` to keep the broker passwords out of the Helm release. Inline passwords still work but are stored in the release values, and their `your-…` defaults are accepted as literal passwords, so replace them. +- A changed password in a credentials secret takes effect only after `cpln workload force-redeployment` of the workloads that read it. - `kafka.secrets.kraft_cluster_id` is fixed once the brokers have started; changing it later means a new cluster. - `kafka.replicas` must be 1, 3, 4 or 5. Never shrink a running cluster below its controller count. - Kafbat UI and the REST Proxy are open to the internet by default (`external_inboundAllowCIDR: 0.0.0.0/0`); set a strong console login and BASIC-auth users, or narrow the CIDR. -- The REST Proxy defaults only work for a release named `kafka-dev`: set `kafka_rest_proxy.properties.bootstrap.servers`, `kafka_rest_proxy.jaas_conf` and `kafka_rest_proxy.password_properties` for your release, or disable it. +- The REST Proxy defaults only work for a release named `kafka-dev`: set `kafka_rest_proxy.properties.bootstrap.servers`, `kafka_rest_proxy.jaas_conf_secret` (or `jaas_conf`) and `kafka_rest_proxy.password_properties_secret` (or `password_properties`) for your release, or disable it. - `ghcr.io/kafbat/kafka-ui`, `confluentinc/cp-kafka-rest:latest` and the JMX exporter image are not pinned, so a redeploy may pick up a newer build; pin a tag in your values if that matters. - Uninstalling deletes the volume sets and every topic's data; a final snapshot is kept for `retentionDuration` because `createFinalSnapshot` is `true`. - Only set `KAFKA_LOG4J_ROOT_LOGLEVEL` in `kafka.env` on `apache/kafka:3.x` images; on a 4.x image it removes every output from the root logger, including `cpln logs`. diff --git a/template-catalog/templates/keycloak.mdx b/template-catalog/templates/keycloak.mdx index 39484f3d..142efba1 100644 --- a/template-catalog/templates/keycloak.mdx +++ b/template-catalog/templates/keycloak.mdx @@ -1,7 +1,7 @@ --- title: Keycloak icon: /template-catalog/templates/icons/keycloak.png -description: Keycloak 26 identity provider (SSO, OIDC, SAML) on Control Plane. Two clustered replicas in production mode with a bundled highly available PostgreSQL store by default, or a single-instance store for dev/test. Deploys into an existing GVC; a bootstrap admin dictionary secret must exist first. +description: Keycloak 26 identity provider (SSO, OIDC, SAML) on Control Plane. Two clustered replicas in production mode with a bundled highly available PostgreSQL store by default, or a single-instance store for dev/test. Deploys into an existing GVC; admin and database dictionary secrets must exist first. keywords: ['auth server', 'oauth2', 'oidc', 'saml', 'sso', 'iam', 'idp', 'auth0 alternative', 'okta alternative', 'single sign-on', 'user federation', 'infinispan', 'realm', 'identity provider', 'login'] --- @@ -30,19 +30,31 @@ All durable state — realms, users, and active sessions — lives in PostgreSQL | Volume set | `RELEASE_NAME-postgres-ha-vs`, `RELEASE_NAME-etcd-vs` | Persistent data for the Patroni and etcd replicas | default store | | Volume set | `RELEASE_NAME-pg-vs` | Persistent data for the single-instance store | `postgres.enabled: true` | | Secret | your `admin.secretName` | Dictionary secret holding the bootstrap admin `username` and `password`. **Created by you, not by the template** — see [Prerequisites](#prerequisites) | — | -| Secret | `my-keycloak-db-credentials` (the value of `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName`) | Dictionary secret holding the database `username`, `password` and `database`. Created by this template from `postgres.credentials.*` for whichever store is enabled | — | +| Secret | your `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` | Dictionary secret holding the database `username`, `password` and `database`. **Created by you, not by the template** — see [Prerequisites](#prerequisites) | — | | Secret | `RELEASE_NAME-keycloak-startup` | Opaque startup script: waits for the database, then configures clustering | — | | Secret | `RELEASE_NAME-patroni-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-etcd-startup`, `RELEASE_NAME-wal-g-backup-script` | Startup and backup scripts for the HA store | default store | | Secret | `RELEASE_NAME-postgres-config` | Non-sensitive backup destination settings for the HA store | `postgresHA.backup.enabled: true` | -| Identity + Policy | `RELEASE_NAME-keycloak-identity`, `RELEASE_NAME-keycloak-policy` | Grants the Keycloak workload `reveal` on exactly three secrets: your admin secret, the startup script and the database credentials | — | +| Identity + Policy | `RELEASE_NAME-keycloak-identity`, `RELEASE_NAME-keycloak-policy` | Grants the Keycloak workload `reveal` on exactly three secrets: your admin secret, your database credentials secret and the startup script | — | | Identity + Policy | `RELEASE_NAME-postgres-ha-identity`, `RELEASE_NAME-postgres-ha-policy`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy` | Least-privilege access for the HA store's own workloads | default store | | Identity + Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | Least-privilege access for the single-instance store | `postgres.enabled: true` | ## Prerequisites -**One dictionary secret must exist before you install.** The bootstrap admin guards a login form on the public endpoint, so its credentials are a prerequisite secret rather than template values. The template creates no admin secret of its own; it references yours by name and grants the workload `reveal` on exactly that one secret. +**Two dictionary secrets must exist before you install.** The database credentials and the bootstrap admin login (which guards a login form on the public endpoint) are prerequisite secrets rather than template values. The template creates neither; it references yours by name and grants the workload `reveal` on exactly those secrets. + + A [dictionary secret](/guides/create-secret/dictionary) holding `username`, `password` and `database`. The bundled PostgreSQL initializes with them and Keycloak connects with them: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=keycloak \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=keycloak + ``` + + Set its name in the key of the store you use: `postgresHA.config.credentialsSecretName` (HA, the default) or `postgres.config.credentialsSecretName` (single-instance). Secret names are org-wide, so give each release its own. `database` must be URL-safe (letters, digits, `_`) because Keycloak's JDBC URL is assembled from it at container start. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password). + A [dictionary secret](/guides/create-secret/dictionary) holding exactly the keys `username` and `password`. Secrets are org-level, so no GVC flag is involved: @@ -82,26 +94,28 @@ The secret ADMIN_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed. ``` -It is **`get-deployments`** — plain `cpln workload get` has no `versions` key. Once you create the secret the deployment recovers on its own; to skip the wait, force a redeployment: +The message names whichever secret is missing — the admin secret or the database credentials secret. It is **`get-deployments`** — plain `cpln workload get` has no `versions` key. Once you create the secret the deployment recovers on its own; to skip the wait, force a redeployment: ```bash cpln workload force-redeployment RELEASE_NAME-keycloak --gvc GVC_NAME ``` -The database password is **not** a prerequisite: this template creates the database credentials secret itself from `postgres.credentials.*`, so there is nothing to create — only a placeholder to change at install time. Optional database backups need a bucket plus a cloud account (AWS or GCP) or a MinIO credentials secret; see [Backing Up](#backing-up). +Optional database backups need a bucket plus a cloud account (AWS or GCP) or a MinIO credentials secret; see [Backing Up](#backing-up). ## Installation -Install from the marketplace registry, naming your admin secret and replacing the placeholder database password: +Install from the marketplace registry, naming your two secrets: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/keycloak \ - --version 1.4.0 --gvc GVC_NAME \ + --version 1.5.0 --gvc GVC_NAME \ --set admin.secretName=ADMIN_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' + --set postgresHA.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME ``` +For the single-instance store set `postgres.config.credentialsSecretName` instead. + To install using another method, follow the instructions for your preferred tool: @@ -159,14 +173,12 @@ admin: ### Database Credentials ```yaml -postgres: - credentials: # internal plumbing; this template creates the dictionary secret from these - username: keycloak - password: change-me-keycloak-db # change before installing - database: keycloak +postgresHA: + config: + credentialsSecretName: my-keycloak-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install ``` -- `postgres.credentials.*` — Username, password and database name for the bundled database. **They feed whichever store is enabled**, including the default HA store, even though the rest of the `postgres` block is off by default. The template writes them into the dictionary secret named by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName`, and both Keycloak and the database read that secret. Change the password before installing. +- `postgresHA.config.credentialsSecretName` / `postgres.config.credentialsSecretName` — Name of the dictionary secret you created in [Prerequisites](#prerequisites), holding `username`, `password` and `database`. Set it in the block of the store you enable; both Keycloak and the database read that secret. ### Backing Store @@ -178,7 +190,7 @@ Exactly one of the two stores must be enabled; the chart refuses both-on and bot postgresHA: enabled: true # set postgres.enabled: true (and this to false) for a lighter single-instance dev/test store config: - credentialsSecretName: my-keycloak-db-credentials # created by this template; secret names are org-wide, so give each release its own + credentialsSecretName: my-keycloak-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install replicas: 3 volumeset: capacity: 10 # initial capacity in GiB per replica (minimum is 10) @@ -196,7 +208,7 @@ postgresHA: postgres: enabled: true config: - credentialsSecretName: my-keycloak-db-credentials # created by this template; secret names are org-wide, so give each release its own + credentialsSecretName: my-keycloak-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) @@ -204,7 +216,7 @@ postgres: enabled: false ``` -- `postgresHA.config.credentialsSecretName` / `postgres.config.credentialsSecretName` — Name of the database credentials secret this template creates. Secret names are organization-wide: a second release left on the default name is refused at install (see [Troubleshooting](#troubleshooting)). +- `postgresHA.config.credentialsSecretName` / `postgres.config.credentialsSecretName` — Name of your database credentials secret (see [Database Credentials](#database-credentials)). Secret names are organization-wide, so give each release its own. - `postgresHA.proxy.enabled` must stay `true` — Keycloak writes through the HAProxy leader endpoint, and the chart refuses the install at render if it is disabled. - Any other setting of the backing template can be set under the same prefix (`postgresHA.pgbouncer.*`, `postgresHA.etcd.*`, `postgres.pgbouncer.*`, volume-set autoscaling, and so on); see the backing template's page for those keys. @@ -337,7 +349,7 @@ cpln secret create-dictionary --name MINIO_CREDENTIALS_SECRET_NAME \ --entry secretKey=MINIO_SECRET_KEY ``` -Like the admin secret, a MinIO credentials secret that does not exist at install time wedges the backup workload silently; the diagnostic is the same `status.versions[].message` described in [Prerequisites](#prerequisites). +Like the admin and database credentials secrets, a MinIO credentials secret that does not exist at install time wedges the backup workload silently; the diagnostic is the same `status.versions[].message` described in [Prerequisites](#prerequisites). ### Restoring a Backup @@ -351,6 +363,56 @@ cpln workload force-redeployment RELEASE_NAME-keycloak --gvc GVC_NAME This restore path has **not** been verified end to end against a live Keycloak install. The database steps are the backing template's, verified there; the Keycloak-specific part is only the restart above. Test a restore on a non-production release before relying on it. +### Rotating the Database Password + +PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with, so on the single-instance store a rotation takes three steps in order (on the HA store follow [Rotating the Database Password in HA Mode](#rotating-the-database-password-in-ha-mode) instead): + +1. Change it in the database: `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` (`DB_USERNAME` is the secret's `username`; connect as that user from inside the GVC). +2. Update the secret — export it, change only `password`, and apply it: + + ```bash + cpln secret reveal SECRET_NAME -o yaml > secret.yaml + # edit only the password key in secret.yaml + cpln apply -f secret.yaml + ``` + +3. Right away, redeploy Keycloak: + + ```bash + cpln workload force-redeployment RELEASE_NAME-keycloak --gvc GVC_NAME + ``` + +Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. + +### Rotating the Database Password in HA Mode + +The HA store (`postgresHA`) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Change BOTH roles on -0 +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +# 4. Right after, put the same password in the secret (edit only the password key in secret.yaml) +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +cpln apply -f secret.yaml +# 5. Restart the members +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 6. Once the restart completes, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 7. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-keycloak`, which keeps the old password until it restarts. The database drops connections briefly at each switchover, and Keycloak cannot open new connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + ### Upgrading From 1.0.x Version `1.1.0` moved the bootstrap admin login out of Helm values and into the prerequisite dictionary secret described in [Prerequisites](#prerequisites). Earlier versions shipped a username and a working default password as values. @@ -370,29 +432,25 @@ Installs created on `1.0.0` with the default HA store also predate the etcd hist ### Upgrading From 1.1.0 -Version `1.2.0` adopted the `postgres` 3.4.1 template for the single-instance store, which no longer takes credentials as values. This template absorbed that change rather than passing it on — **there is no new prerequisite**, only renames on the single-instance path: +Version `1.2.0` adopted the `postgres` 3.4.1 template for the single-instance store, which no longer takes credentials as values: | Removed key | Replacement | |---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | +| `postgres.config.username` / `postgres.config.password` / `postgres.config.database` | the `username`, `password` and `database` keys of the database credentials secret named by `postgres.config.credentialsSecretName` | | `postgres.backup.minio.accessKey` and `postgres.backup.minio.secretKey` | `postgres.backup.minio.credentialsSecretName` — a dictionary secret you create; MinIO backups only | -An upgrade carrying an old key is refused at render with the **PostgreSQL template's** message, which tells you to create a credentials secret yourself. Ignore that part here: this template creates the database credentials secret from `postgres.credentials.*`. Move the keys and the upgrade proceeds. The HA path was unchanged in `1.2.0`. +An upgrade carrying an old key is refused at render with the **PostgreSQL template's** message. Put the values your database was initialized with into the database credentials secret under [Prerequisites](#prerequisites), set `postgres.config.credentialsSecretName` to its name, and remove the old keys. The HA path was unchanged in `1.2.0`. ### Upgrading From 1.2.x and 1.3.0 -Version `1.3.0` adopted the `postgres-highly-available` 2.5.0 template for the default store, which likewise no longer takes credentials as values. The HA store now reads the same template-created credentials secret as the single-instance store, named by the new `postgresHA.config.credentialsSecretName` key: +Version `1.3.0` adopted the `postgres-highly-available` 2.5.0 template for the default store, which likewise no longer takes credentials as values. The HA store reads the database credentials secret named by `postgresHA.config.credentialsSecretName`: | Removed key | Replacement | |---|---| -| `postgresHA.postgres.username` | `postgres.credentials.username` | -| `postgresHA.postgres.password` | `postgres.credentials.password` | -| `postgresHA.postgres.database` | `postgres.credentials.database` | +| `postgresHA.postgres.username` / `postgresHA.postgres.password` / `postgresHA.postgres.database` | the `username`, `password` and `database` keys of the database credentials secret named by `postgresHA.config.credentialsSecretName` | | `postgresHA.backup.minio.accessKey` and `postgresHA.backup.minio.secretKey` | `postgresHA.backup.minio.credentialsSecretName` — a dictionary secret you create; MinIO backups only (the values file caught up with this in `1.3.1`) | -An upgrade carrying a removed key is refused at render with the **PostgreSQL Highly Available template's** message, which tells you to create a credentials secret yourself; as above, ignore that part — this template creates it. Set `postgres.credentials.username`, `password` and `database` to **the same values your install already uses** under `postgresHA.postgres.*`: the database volume was initialised with those credentials and keeps them. Changing the database password as part of this upgrade has not been verified and is not recommended. +An upgrade carrying a removed key is refused at render with the **PostgreSQL Highly Available template's** message. Put **the same values your install already uses** under `postgresHA.postgres.*` into the database credentials secret under [Prerequisites](#prerequisites) — the database volume was initialized with those credentials and keeps them — set `postgresHA.config.credentialsSecretName` to its name, and delete the `postgresHA.postgres` block. Do not change the database password as part of this upgrade. ### Upgrading From 1.3.1 @@ -405,6 +463,23 @@ Version `1.4.0` renamed the Keycloak resource limits so it is unambiguous which An upgrade that still carries a bare `resources.cpu` or `resources.memory` is refused at render: `keycloak: resources.cpu was RENAMED to resources.maxCpu.` The reservation keys `minCpu` and `minMemory` are unchanged. +### Upgrading From 1.4.0 + +Version `1.5.0` removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the database templates require on their own. Carrying the old key forward fails the render. + +1.4.0 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Keycloak loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-keycloak-db \ + --entry username=keycloak \ + --entry password='YOUR_CURRENT_PASSWORD' \ + --entry database=keycloak +``` + +Your current values are in `postgres.credentials` of the 1.4.0 release, and also in the old secret (`cpln secret reveal my-keycloak-db-credentials -o yaml`, or the name your release set in `credentialsSecretName`) until the upgrade deletes it. Then upgrade with `postgresHA.config.credentialsSecretName` (HA store) or `postgres.config.credentialsSecretName` (single-instance store) set to the new name, and `postgres.credentials` removed. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Keycloak can briefly lose its database; it reconnects by itself. The new version of a workload can also wait several minutes with `is not allowed to reveal the secret` in `get-deployments` while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. + ### Scaling and Availability - **Keycloak replicas.** Change `replicas` and upgrade the release; there is deliberately no autoscaling, so cluster membership only changes when you change it. Clustered replicas (`2` or more) share realm, user and session state through the database, so one replica restarting does not interrupt sign-ins. With `replicas: 1` a restart is a short outage. @@ -417,7 +492,7 @@ An upgrade that still carries a bare `resources.cpu` or `resources.memory` is re - **Cause:** the admin secret named by `admin.secretName` (or, with MinIO backups, the secret named by `backup.minio.credentialsSecretName`) does not exist. The container never ran, so there is nothing to log. + **Cause:** the admin secret named by `admin.secretName`, the database credentials secret named by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` (or, with MinIO backups, the secret named by `backup.minio.credentialsSecretName`) does not exist. The container never ran, so there is nothing to log. **Fix:** read the message that names the missing secret, create it, then force a redeployment to skip the wait: @@ -440,8 +515,9 @@ An upgrade that still carries a bare `resources.cpu` or `resources.memory` is re | `keycloak: replicas > 1 requires internalAccess.type other than 'none'` | Set `internalAccess.type` to `same-gvc`, `same-org` or `workload-list`, or run `replicas: 1` | | `keycloak: enable exactly one backing store` | Set exactly one of `postgresHA.enabled` and `postgres.enabled` to `true` | | `keycloak: postgresHA.proxy.enabled must remain true` | Remove the override; Keycloak connects through the HAProxy leader endpoint | - | `postgres-highly-available: the postgres block was REMOVED in 2.5.0` | Move `postgresHA.postgres.*` to `postgres.credentials.*`; see [Upgrading From 1.2.x and 1.3.0](#upgrading-from-1-2-x-and-1-3-0) | - | `config.username was REMOVED in postgres 3.4.0` | Move `postgres.config.{username,password,database}` to `postgres.credentials.*`; see [Upgrading From 1.1.0](#upgrading-from-1-1-0) | + | `postgres-highly-available: the postgres block was REMOVED in 2.5.0` | Move `postgresHA.postgres.*` into the database credentials secret; see [Upgrading From 1.2.x and 1.3.0](#upgrading-from-1-2-x-and-1-3-0) | + | `config.username was REMOVED in postgres 3.4.0` | Move `postgres.config.{username,password,database}` into the database credentials secret; see [Upgrading From 1.1.0](#upgrading-from-1-1-0) | + | `keycloak: postgres.credentials was REMOVED in 1.5.0` | Move the values into a database credentials secret; see [Upgrading From 1.4.0](#upgrading-from-1-4-0) | | `backup.minio.accessKey and backup.minio.secretKey were REMOVED` | Create the MinIO credentials secret and set `…backup.minio.credentialsSecretName`; see [Backing Up](#backing-up) | @@ -462,11 +538,6 @@ An upgrade that still carries a bare `resources.cpu` or `resources.memory` is re cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres-ha"}' --limit 50 --since 10m ``` - - **Cause:** the database credentials secret is named by a plain value (`my-keycloak-db-credentials` by default) and secret names are organization-wide, so a second Keycloak release on the same name tries to take over a secret the first release owns. Nothing is shared, overwritten or deleted; the second release simply does not install. - - **Fix:** give each release its own `postgresHA.config.credentialsSecretName` (and `postgres.config.credentialsSecretName`) and install again. - **Cause:** the bootstrap credentials are read on first boot only, when no admin account exists yet. Afterwards the account lives in the database, and neither editing the secret nor redeploying the workload changes it. @@ -487,9 +558,9 @@ An upgrade that still carries a bare `resources.cpu` or `resources.memory` is re ## Important Notes -- **Create the admin secret before installing** — a missing prerequisite secret wedges the deployment with zero log lines; `status.versions[].message` is the only diagnostic (see [Prerequisites](#prerequisites)). -- **Change the database password before installing** (`postgres.credentials.password`) — it is used exactly as given and feeds whichever store is enabled. -- **Give each release its own `credentialsSecretName`** (`postgresHA.config.` and `postgres.config.`) — secret names are organization-wide, and a second release on the default name is refused at install. +- **Create the admin and database credentials secrets before installing** — a missing prerequisite secret wedges the deployment with zero log lines; `status.versions[].message` is the only diagnostic (see [Prerequisites](#prerequisites)). +- **Rotating the database password needs `ALTER ROLE` plus a forced redeployment of Keycloak** (and a leader switchover first on the HA store) — see [Rotating the Database Password](#rotating-the-database-password). +- **Give each release its own database credentials secret** — secret names are organization-wide. - **The bootstrap admin is temporary by design** — sign in, create a permanent admin, then delete the bootstrap account. Its credentials are read on first boot only, so change the password in the console, not by editing the secret. - **Keep `publicAccess` enabled for browser SSO** — end-user browsers must reach Keycloak's login endpoints; disable it only for pure service-to-service deployments. - **Do not disable the HA proxy** (`postgresHA.proxy.enabled`) — Keycloak writes through the HAProxy leader endpoint; the chart refuses the install otherwise. diff --git a/template-catalog/templates/langfuse.mdx b/template-catalog/templates/langfuse.mdx index 7f4818fc..ffeb6b82 100644 --- a/template-catalog/templates/langfuse.mdx +++ b/template-catalog/templates/langfuse.mdx @@ -1,7 +1,7 @@ --- title: Langfuse icon: /template-catalog/templates/icons/langfuse.png -description: Langfuse LLM observability and evaluation, self-hosted. Web and worker tiers with bundled PostgreSQL, Redis and ClickHouse, trace data in your own S3 or GCS bucket. Deploys into an existing GVC. An auth secret and the bucket must exist before install. +description: Langfuse LLM observability and evaluation, self-hosted. Web and worker tiers with bundled PostgreSQL, Redis and ClickHouse, trace data in your own S3 or GCS bucket. Deploys into an existing GVC. Auth and datastore credentials secrets and the bucket must exist before install. keywords: ['llm observability', 'llm tracing', 'prompt management', 'llm evaluation', 'langsmith alternative', 'helicone alternative', 'bullmq', 'nextauth', 'openai api key storage', 'ai agent monitoring', 'trace ingestion', 'headless initialization', 'encryptionKey'] --- @@ -28,8 +28,6 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Volume set | `RELEASE_NAME-pg-vs` | PostgreSQL data | — | | Volume set | `RELEASE_NAME-langfuse-redis-vs` | Redis append-only file | — | | Volume set | `RELEASE_NAME-langfuse-clickhouse-vs` | ClickHouse local metadata only — data parts live in your bucket | — | -| Secret | `RELEASE_NAME-langfuse-config` | Bundled datastore credentials read by the web and worker tiers (dictionary) | — | -| Secret | `my-langfuse-db-credentials` | Bundled PostgreSQL credentials, named by `postgres.config.credentialsSecretName` and created by the chart (dictionary) | — | | Secret | `RELEASE_NAME-langfuse-clickhouse-startup` | ClickHouse startup script (opaque) | — | | Secret | `RELEASE_NAME-langfuse-clickhouse-storage` | ClickHouse S3 / GCS disk configuration (opaque) | — | | Identity | `RELEASE_NAME-langfuse-identity` | Identity of the web, worker, Redis and ClickHouse workloads; linked to your AWS cloud account on the `aws` path | — | @@ -37,11 +35,11 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Identity | `RELEASE_NAME-pg-identity` | Identity of the bundled PostgreSQL | — | | Policy | `RELEASE_NAME-pg-policy` | Grants PostgreSQL `reveal` on its own credentials secret | — | -One bucket serves both ClickHouse (prefix `clickhouse/`) and Langfuse (prefixes `events/` and `media/`). The auth secret and, on GCS, the credentials secret are prerequisite secrets you create yourself: the chart references them by name and never creates, modifies, or deletes them. +One bucket serves both ClickHouse (prefix `clickhouse/`) and Langfuse (prefixes `events/` and `media/`). The auth secret, the database, Redis and ClickHouse credentials secrets and, on GCS, the storage credentials secret are prerequisite secrets you create yourself: the chart references them by name and never creates, modifies, or deletes them. ## Prerequisites -Two things must exist before you install: the auth secret, and the object storage that traces are written to. +Three things must exist before you install: the auth secret, the datastore credentials secrets, and the object storage that traces are written to. @@ -70,6 +68,32 @@ Two things must exist before you install: the auth secret, and the object storag Set `langfuse.auth.secretName` to this name. + + A [dictionary secret](/guides/create-secret/dictionary) with the keys `username`, `password` and `database`. The bundled PostgreSQL initializes with it and Langfuse connects with it: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=langfuse \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=langfuse + ``` + + Set `postgres.config.credentialsSecretName` to this name. Keep all three values URL-safe (letters, digits, `-`, `_`): Langfuse puts them in its connection URL. PostgreSQL reads the secret only when its data directory is first initialized; to change the password later, see [Rotating Credentials](#rotating-credentials). + + + One dictionary secret per datastore. Both datastores and both Langfuse workloads read them at every start: + + ```bash + cpln secret create-dictionary --name REDIS_SECRET_NAME \ + --entry password="$(openssl rand -hex 24)" + + cpln secret create-dictionary --name CLICKHOUSE_SECRET_NAME \ + --entry username=default \ + --entry password="$(openssl rand -hex 24)" + ``` + + Set `redis.credentialsSecretName` and `clickhouse.credentialsSecretName` to those names. Keep the ClickHouse values URL-safe (letters, digits, `-`, `_`; the username must also start with a letter): Langfuse puts them in its migration URL. + Pick one provider. ClickHouse and Langfuse share the bucket under separate prefixes. @@ -139,7 +163,7 @@ Two things must exist before you install: the auth secret, and the object storag -**A missing prerequisite secret wedges the deployment instead of failing it.** `cpln helm install` reports success, but the web and worker workloads never start and `cpln logs` returns nothing. The only place the missing secret is named is `status.versions[].message`: +**A missing prerequisite secret wedges the deployment instead of failing it.** `cpln helm install` reports success, but the workloads that read the missing secret never start and `cpln logs` returns nothing. The only place the missing secret is named is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-langfuse-web --gvc GVC_NAME -o yaml @@ -152,19 +176,19 @@ Backing up the bundled PostgreSQL to a bucket is optional and needs its own buck ## Installation -Install from the marketplace registry, naming your auth secret and bucket and replacing the three bundled datastore passwords, which the chart uses exactly as written: +Install from the marketplace registry, naming your auth secret, your three datastore credentials secrets and your bucket: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/langfuse \ - --version 1.2.1 --gvc GVC_NAME \ + --version 1.3.0 --gvc GVC_NAME \ --set langfuse.auth.secretName=AUTH_SECRET_NAME \ + --set postgres.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME \ + --set redis.credentialsSecretName=REDIS_SECRET_NAME \ + --set clickhouse.credentialsSecretName=CLICKHOUSE_SECRET_NAME \ --set objectStore.aws.bucket=YOUR_BUCKET_NAME \ --set objectStore.aws.region=us-east-1 \ --set objectStore.aws.cloudAccountName=YOUR_CLOUD_ACCOUNT_NAME \ - --set objectStore.aws.policyName=YOUR_IAM_POLICY_NAME \ - --set postgres.credentials.password='YOUR-POSTGRES-PASSWORD' \ - --set redis.auth.password='YOUR-REDIS-PASSWORD' \ - --set clickhouse.config.password='YOUR-CLICKHOUSE-PASSWORD' + --set objectStore.aws.policyName=YOUR_IAM_POLICY_NAME ``` On the GCS path replace the four `objectStore.aws.*` values with `objectStore.provider=gcp`, `objectStore.gcp.bucket` and `objectStore.gcp.credentialsSecretName`. @@ -267,12 +291,8 @@ postgres: maxCpu: 1 minMemory: 512Mi maxMemory: 1Gi - credentials: # the chart writes these into the secret named below — nothing for you to create - username: langfuse - password: change-me-langfuse-db - database: langfuse config: - credentialsSecretName: my-langfuse-db-credentials # secret names are org-wide — give each release its own + credentialsSecretName: my-langfuse-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) ``` @@ -289,8 +309,7 @@ redis: maxCpu: 400m minMemory: 256Mi maxMemory: 512Mi - auth: - password: change-me-langfuse-redis + credentialsSecretName: my-langfuse-redis # PREREQUISITE dictionary secret (password) — must exist BEFORE install volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) ``` @@ -305,14 +324,12 @@ clickhouse: maxCpu: 2 minMemory: 2Gi maxMemory: 4Gi - config: - password: change-me-langfuse-clickhouse - database: langfuse + credentialsSecretName: my-langfuse-clickhouse # PREREQUISITE dictionary secret (username, password) — must exist BEFORE install volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) ``` -The three datastore passwords are internal plumbing — PostgreSQL, Redis and ClickHouse serve Langfuse only — but they are used exactly as written, so change all three before installing. +The three datastore credentials come from the prerequisite secrets you created; none of them is a Helm value. ## Connecting @@ -323,9 +340,9 @@ The three datastore passwords are internal plumbing — PostgreSQL, Redis and Cl | Health check | `/api/public/health` on either address | Returns `{"status":"OK","version":"..."}`. | | First login | `adminEmail` / `adminPassword` | From your auth secret: `cpln secret reveal AUTH_SECRET_NAME -o yaml`. | | Langfuse API keys | **Settings → API Keys** in the UI | Per project, a public / secret key pair. | -| PostgreSQL | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Internal only. Credentials are `postgres.credentials.*`. | -| Redis | `RELEASE_NAME-langfuse-redis.GVC_NAME.cpln.local:6379` | Internal only. Password is `redis.auth.password`. | -| ClickHouse | `RELEASE_NAME-langfuse-clickhouse.GVC_NAME.cpln.local:8123` (HTTP) and `:9000` (native) | Internal only. User `default`, password `clickhouse.config.password`. | +| PostgreSQL | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Internal only. `username` / `password` / `database` from your `postgres.config.credentialsSecretName` secret. | +| Redis | `RELEASE_NAME-langfuse-redis.GVC_NAME.cpln.local:6379` | Internal only. `password` from your `redis.credentialsSecretName` secret. | +| ClickHouse | `RELEASE_NAME-langfuse-clickhouse.GVC_NAME.cpln.local:8123` (HTTP) and `:9000` (native) | Internal only. `username` / `password` from your `clickhouse.credentialsSecretName` secret. | Verify the web tier is serving once `cpln workload get-deployments RELEASE_NAME-langfuse-web --gvc GVC_NAME -o yaml` reports it ready: @@ -402,7 +419,7 @@ A zero-length backup object is a failed run, not a backup. If the dump cannot re These paths have not been exercised against this template; verify them on a throwaway release before relying on them. -- **From a scheduled dump:** the dump is a plain `pg_dumpall` output restored with `psql` against `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432`, using the `postgres.credentials.username` and `postgres.credentials.password` you installed with. The command sequence per provider is on the [postgres template's restore section](/template-catalog/templates/postgres#restoring-a-backup); it must run from a client that can reach the bucket and the database, which means a workload inside the GVC. +- **From a scheduled dump:** the dump is a plain `pg_dumpall` output restored with `psql` against `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432`, using the `username` and `password` from your database credentials secret. The command sequence per provider is on the [postgres template's restore section](/template-catalog/templates/postgres#restoring-a-backup); it must run from a client that can reach the bucket and the database, which means a workload inside the GVC. - **From a volume set snapshot:** `cpln volumeset snapshot restore` exists (check `cpln volumeset snapshot restore --help`) but has not been run against this template. - **ClickHouse:** trace data is in your bucket under `clickhouse/`. A fresh ClickHouse workload pointed at the same bucket is the recovery path, but it has not been verified; deleting the bucket loses all trace history regardless of snapshots. @@ -447,15 +464,7 @@ named by langfuse.auth.secretName (key: nextAuthSecret) — see the README. ### Upgrading from 1.1.0 -Version 1.2.0 moved the bundled PostgreSQL to the `postgres` 3.4.1 template, which reads its credentials from a dictionary secret. Langfuse creates that secret for you from values, so there is no new prerequisite — only a rename: - -| Removed key | Replacement | -|---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | - -Carrying an old key fails the render with the **postgres template's** message, which tells you to create a dictionary secret yourself. Ignore that advice here: move the three keys to `postgres.credentials.*` and you are done. `postgres.config.credentialsSecretName` is new and names the secret the chart creates; the default is fine for a single release. The auth, Redis and ClickHouse paths are unchanged. +`postgres.config.username`, `.password` and `.database` were removed. Put the values your database was initialized with into a database credentials secret as under [Prerequisites](#prerequisites) (keys `username`, `password`, `database`), under any name except `RELEASE_NAME-pg-config` (the secret 1.1.0 created, which this upgrade deletes). Set `postgres.config.credentialsSecretName` to it and remove the three keys. The Redis and ClickHouse passwords move to prerequisite secrets too: follow [Upgrading from 1.2.x](#upgrading-from-1-2-x) for them (1.1.0 keeps them in the same `RELEASE_NAME-langfuse-config` secret). The auth path is unchanged. ### Upgrading from 1.2.0 @@ -463,6 +472,55 @@ Version 1.2.1 removes the AWS managed policy `aws::ReadOnlyAccess` from the iden **Before upgrading, update your IAM policy to the ten-action list in [Prerequisites](#prerequisites)** — 1.2.0 documented six actions. If it already matches, no action is needed. Nothing else changes. +### Upgrading from 1.2.x + +Version 1.3.0 removed `postgres.credentials`, `redis.auth.password` and `clickhouse.config.password`: all three datastores now read prerequisite secrets, and carrying any old key forward fails the render. + +1.2.x created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Langfuse loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-langfuse-db \ + --entry username='YOUR_CURRENT_USERNAME' --entry password='YOUR_CURRENT_PASSWORD' --entry database='YOUR_CURRENT_DATABASE' +``` + +Your current values are in `postgres.credentials` of the 1.2.x release, and also in the old secret named by `postgres.config.credentialsSecretName` in 1.2.x (`cpln secret reveal SECRET_NAME -o yaml`) until the upgrade deletes it. + +1.2.x kept the Redis and ClickHouse passwords in `RELEASE_NAME-langfuse-config` (keys `redisPassword`, `clickhousePassword`), which **this upgrade also deletes**, so use other names. Create them holding your current passwords, so the old and new versions keep talking to each other during the rollout: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-langfuse-redis-credentials \ + --entry password='YOUR_CURRENT_REDIS_PASSWORD' + +cpln secret create-dictionary --name RELEASE_NAME-langfuse-clickhouse-credentials \ + --entry username=default --entry password='YOUR_CURRENT_CLICKHOUSE_PASSWORD' +``` + +Your current values are `redis.auth.password` and `clickhouse.config.password` of the 1.2.x release, and also in `cpln secret reveal RELEASE_NAME-langfuse-config -o yaml` until the upgrade deletes it. + +Then upgrade with `postgres.config.credentialsSecretName`, `redis.credentialsSecretName` and `clickhouse.credentialsSecretName` set to those names, and `postgres.credentials`, `redis.auth` and `clickhouse.config.password` removed. The upgrade restarts the bundled PostgreSQL (it now reads the new secret), so Langfuse briefly loses its database and reconnects by itself. While access to the new secrets propagates, the new version of a workload can wait in `get-deployments` reporting that it is not allowed to reveal the secret; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating Credentials + +A running workload keeps the credential values it started with, so every rotation ends with a forced redeployment. Change one key and keep the others with export, edit, apply: + +```bash +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only the key you are rotating in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +``` + +- **Database password:** PostgreSQL reads the secret only when it is first initialized, so run `ALTER ROLE` in the database first, then update the secret, then right away force a redeployment of `RELEASE_NAME-langfuse-web` and `RELEASE_NAME-langfuse-worker`. Do not redeploy the database; it would only add downtime. Connections already open keep working after `ALTER ROLE`, new ones fail until the redeploy finishes, so rotate in a quiet window. +- **Redis and ClickHouse passwords:** neither datastore keeps its password in its data (Redis takes it on the command line, ClickHouse defines its user from the secret at every start), so there is no `ALTER` step. Update the secret, then force a redeployment of the datastore and of everything that connects to it, together: + +```bash +cpln workload force-redeployment RELEASE_NAME-langfuse-redis --gvc GVC_NAME # or RELEASE_NAME-langfuse-clickhouse +cpln workload force-redeployment RELEASE_NAME-langfuse-web --gvc GVC_NAME +cpln workload force-redeployment RELEASE_NAME-langfuse-worker --gvc GVC_NAME +``` + +Between the datastore restarting and the Langfuse workloads coming back, Langfuse cannot reach that datastore, so rotate in a quiet window. Changing the ClickHouse `username` works the same way. + ### Scaling and Availability - The web tier autoscales on CPU between `langfuse.web.minReplicas` and `langfuse.web.maxReplicas`. Keep the minimum at 2 or more for rolling deploys without downtime. @@ -477,7 +535,7 @@ Version 1.2.1 removes the AWS managed policy `aws::ReadOnlyAccess` from the iden **Symptom:** `cpln helm install` succeeded, but `RELEASE_NAME-langfuse-web` and `RELEASE_NAME-langfuse-worker` stay not-ready with no container output. `status.versions[].message` reads `The secret no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` - **Cause:** the secret named by `langfuse.auth.secretName` (or, on GCS, `objectStore.gcp.credentialsSecretName`) did not exist at install time. + **Cause:** a secret named by `langfuse.auth.secretName`, `postgres.config.credentialsSecretName`, `redis.credentialsSecretName`, `clickhouse.credentialsSecretName` (or, on GCS, `objectStore.gcp.credentialsSecretName`) did not exist at install time. Check the datastore workloads the same way. **Fix:** create it as shown in [Prerequisites](#prerequisites). The deployment recovers on its own after a few minutes, or force it: @@ -486,12 +544,12 @@ Version 1.2.1 removes the AWS managed policy `aws::ReadOnlyAccess` from the iden cpln workload force-redeployment RELEASE_NAME-langfuse-web --gvc GVC_NAME ``` - - **Symptom:** a second Langfuse release in the same org fails at install with `The resource 'my-langfuse-db-credentials' cannot be updated because it is being managed by a different release` and creates nothing. + + **Symptom:** `helm upgrade` aborts with `postgres.credentials was removed in 1.3.0 ...` (or the same for `redis.auth.password` or `clickhouse.config.password`). - **Cause:** `postgres.config.credentialsSecretName` names an org-wide secret, and the first release owns the default name. Nothing is shared, overwritten or deleted; the first release is unaffected. + **Cause:** your values file still carries a 1.2.x datastore credential key. - **Fix:** give the second release its own `--set postgres.config.credentialsSecretName=...` and install again. + **Fix:** follow [Upgrading from 1.2.x](#upgrading-from-1-2-x). **Symptom:** `helm upgrade` aborts with `langfuse.auth.nextAuthSecret was removed in 1.1.0 ...` (or the same for `encryptionKey`, `salt`, `objectStore.gcp.accessKeyId`, `objectStore.gcp.secretAccessKey`, `langfuse.firewall.inboundAllowCIDR`). @@ -548,15 +606,15 @@ Version 1.2.1 removes the AWS managed policy `aws::ReadOnlyAccess` from the iden ## Important Notes -- Create the auth secret **before** installing; a missing one leaves the web and worker workloads waiting on a secret that does not exist. +- Create the auth, database, Redis and ClickHouse credentials secrets **before** installing; a missing one leaves the workloads waiting on a secret that does not exist, named only in `status.versions[].message` of `cpln workload get-deployments RELEASE_NAME-langfuse-web --gvc GVC_NAME -o yaml`. +- Rotating any of these secrets needs a forced redeployment; the database password also needs `ALTER ROLE` first (see [Rotating Credentials](#rotating-credentials)). - `encryptionKey` can never be rotated: it encrypts every provider key stored under **Settings → LLM Connections**, and changing it makes all of them unreadable. Keep a copy outside Control Plane and never delete the secret. - Sign-up is closed by default and the owner account comes from your secret. Record `adminPassword` — it is displayed nowhere, and with sign-up closed there is no other way to create the first account. -- Change `postgres.credentials.password`, `redis.auth.password` and `clickhouse.config.password` before installing; they ship as `change-me-` placeholders and are used exactly as written. - Give each install its own bucket. Two installs on one bucket share ClickHouse data parts under `clickhouse/`. -- Give each release its own `postgres.config.credentialsSecretName`; secret names are org-wide and a second release on the default name is refused at install. +- Give each release its own database, Redis and ClickHouse secrets. Secret names are org-wide, so two releases pointed at one secret share a password, and rotating it for one breaks the other. - Serving Langfuse on your own domain requires `langfuse.publicUrl`, including the scheme. - With `publicAccess.enabled: false` there is no browser path to the UI; only other workloads in the GVC can reach the API. -- Prerequisite secrets are not owned by the release: the auth and GCS credentials secrets survive `helm uninstall` and must be deleted by hand if no longer wanted. +- Prerequisite secrets are not owned by the release: the auth, datastore credentials and GCS credentials secrets survive `helm uninstall` and must be deleted by hand if no longer wanted. - Uninstall deletes the volume sets, including PostgreSQL's. Trace data already in your bucket is untouched, but users, projects and API keys are not — back up PostgreSQL first. ## External References diff --git a/template-catalog/templates/listmonk.mdx b/template-catalog/templates/listmonk.mdx index c23b354f..4934dd6c 100644 --- a/template-catalog/templates/listmonk.mdx +++ b/template-catalog/templates/listmonk.mdx @@ -1,7 +1,7 @@ --- title: Listmonk icon: /template-catalog/templates/icons/listmonk.png -description: Deploy Listmonk, the self-hosted newsletter and mailing list manager, on Control Plane. A single-instance app with bundled PostgreSQL (single-instance by default, highly available with one flag), public subscription pages, and a Super Admin dictionary secret you create before installing. +description: Deploy Listmonk, the self-hosted newsletter and mailing list manager, on Control Plane. A single-instance app with bundled PostgreSQL (single-instance by default, highly available with one flag), public subscription pages, and admin and database credentials secrets you create before installing. keywords: ['listmonk', 'newsletter', 'mailing list manager', 'email marketing', 'mailchimp alternative', 'sendy alternative', 'mautic alternative', 'campaign management', 'subscriber lists', 'opt-in signup form', 'transactional email', 'smtp relay', 'self-hosted email'] --- @@ -22,7 +22,6 @@ This template does not create a GVC. You must deploy it into an existing GVC. | Workload | `RELEASE_NAME-listmonk` | Listmonk server — admin UI, public pages, HTTP API on port `9000` (stateful, 1 replica) | — | | Volume set | `RELEASE_NAME-listmonk-uploads` | Media uploaded through the admin UI, mounted at `/listmonk/uploads` (`ext4`, 10 GiB by default) | — | | Identity & Policy | `RELEASE_NAME-listmonk-identity`, `RELEASE_NAME-listmonk-policy` | Grants the listmonk workload `reveal` on exactly two secrets: your admin secret and the database credentials secret | — | -| Secret | `my-listmonk-db-credentials` (named by `postgres.config.credentialsSecretName`) | Bundled database credentials (`username`, `password`, `database`), built by this template from `postgres.credentials.*` — nothing for you to create | — | | Workload | `RELEASE_NAME-postgres` | Single-instance PostgreSQL 18 (stateful, 1 replica, port `5432`) | default — `postgres.enabled: true` | | Volume set | `RELEASE_NAME-pg-vs` | PostgreSQL data (`ext4`, 10 GiB by default) | `postgres.enabled: true` | | Identity & Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | Grants the database workload `reveal` on the database credentials secret | `postgres.enabled: true` | @@ -34,13 +33,25 @@ This template does not create a GVC. You must deploy it into an existing GVC. | Workload | `RELEASE_NAME-postgres-ha-backup` | Scheduled backup of the HA store (cron) | `postgresHA.backup.enabled: true` | | Secret | `RELEASE_NAME-postgres-config` | Backup bucket and region for the HA store | `postgresHA.backup.enabled: true` | -The Super Admin credentials are **not** created by this template. You create that dictionary secret before installing and reference it by name — see [Prerequisites](#prerequisites). +The Super Admin and database credentials are **not** created by this template. You create those dictionary secrets before installing and reference them by name — see [Prerequisites](#prerequisites). ## Prerequisites -**One dictionary secret must exist before you install.** The Super Admin guards a login form on the public endpoint, so its credentials are a prerequisite secret rather than template values — a value would sit in plaintext in the Helm release for the life of the install. Secrets are org-level, so no GVC flag is involved. +**Two dictionary secrets must exist before you install.** They hold credentials, so they are prerequisite secrets rather than template values — a value would sit in plaintext in the Helm release for the life of the install. Secrets are org-level, so no GVC flag is involved. + + A [dictionary secret](/guides/create-secret/dictionary) with the keys `username`, `password` and `database`. The bundled PostgreSQL initializes with it and listmonk connects with it: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=listmonk \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=listmonk + ``` + + Set its name in the key of the store you use: `postgres.config.credentialsSecretName` (single-instance, the default) or `postgresHA.config.credentialsSecretName` (HA). Secret names are org-wide, so give each listmonk release its own secret. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password) for changing it later. + A [dictionary secret](/guides/create-secret/dictionary) holding exactly two keys, `username` (at least 3 characters) and `password` (at least 8 characters) — both are upstream requirements: @@ -62,7 +73,7 @@ The Super Admin credentials are **not** created by this template. You create tha -**Create the secret before installing, or the deployment wedges silently.** A name that points at a secret which does not exist installs "successfully" and then never starts. The container never runs, so `cpln logs` returns **zero lines** — there is nothing to log. The one place the reason appears is `status.versions[].message`: +**Create both secrets before installing, or the deployment wedges silently.** A name that points at a secret which does not exist installs "successfully" and then never starts. The container never runs, so `cpln logs` returns **zero lines** — there is nothing to log. The one place the reason appears is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-listmonk --gvc GVC_NAME -o yaml @@ -73,20 +84,20 @@ The secret ADMIN_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed. ``` -Use `get-deployments` — plain `cpln workload get` has no `versions` key. Creating the missing secret repairs the deployment on its own, or run `cpln workload force-redeployment RELEASE_NAME-listmonk --gvc GVC_NAME` to pick it up immediately. +Use `get-deployments` — plain `cpln workload get` has no `versions` key; a missing database secret shows the same way on `RELEASE_NAME-postgres` (or `RELEASE_NAME-postgres-ha` in HA mode). Creating the missing secret repairs the deployment on its own, or run `cpln workload force-redeployment RELEASE_NAME-listmonk --gvc GVC_NAME` to pick it up immediately. -Nothing else is required for a default install. The bundled database password is **not** a prerequisite — it is internal plumbing no human types elsewhere, so this template creates that secret for you from `postgres.credentials.*` (change the password before installing). Database backups are optional and need a bucket and provider access first — see [Backing Up](#backing-up). SMTP is configured in listmonk's own admin UI after installing — see [First Run](#first-run). +Nothing else is required for a default install. Database backups are optional and need a bucket and provider access first — see [Backing Up](#backing-up). SMTP is configured in listmonk's own admin UI after installing — see [First Run](#first-run). ## Installation -Install into an existing GVC, naming the admin secret you created and a database password of your own: +Install into an existing GVC, naming the admin and database credentials secrets you created: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/listmonk \ - --version 1.5.0 --gvc GVC_NAME \ + --version 1.6.0 --gvc GVC_NAME \ --set admin.secretName=ADMIN_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-PASSWORD' + --set postgres.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME ``` Or follow the instructions for your preferred method: @@ -180,18 +191,13 @@ The default backing store. Exactly one of `postgres` or `postgresHA` must be ena postgres: enabled: true image: postgres:18 # listmonk requires Postgres 12+ - # Credentials for the bundled database — internal plumbing no human types elsewhere. - # Listmonk CREATES the dictionary secret named by config.credentialsSecretName below - # out of these three values, so there is nothing for you to create before installing. - credentials: - username: listmonk - password: change-me-listmonk-db # change before installing - database: listmonk config: - # Name of the dictionary secret this chart creates and the bundled postgres reads. - # Secret names are org-wide: give each listmonk release its own name. A second - # release left on this name is REFUSED at install (it is owned by the first - # release) — nothing is shared, overwritten or deleted. + # REQUIRED PREREQUISITE SECRET (single-instance mode) — CREATE IT BEFORE YOU + # INSTALL. If it does not exist the deployment WEDGES silently and looks like + # a platform fault. A `dictionary` secret holding exactly three keys: + # username, password, database. PostgreSQL reads it once, when its data is + # first initialized, and listmonk connects with the same keys. See + # Prerequisites for the command. credentialsSecretName: my-listmonk-db-credentials resources: minCpu: 200m @@ -229,11 +235,11 @@ postgres: prefix: listmonk/backups ``` -The database password seeds PostgreSQL on its first boot and is used exactly as given — change it before installing, not after. Provider setup for backups is under [Backing Up](#backing-up). +The credentials secret seeds PostgreSQL on its first boot; editing it afterwards does not change the database password — see [Rotating the Database Password](#rotating-the-database-password). Provider setup for backups is under [Backing Up](#backing-up). ### Highly Available PostgreSQL -Set `postgresHA.enabled: true` for near-zero-downtime database upgrades and automatic failover: a three-replica Patroni cluster coordinated by etcd, fronted by an HAProxy leader endpoint that listmonk connects through for writes and schema migrations (the chart refuses to render if that proxy is disabled). The credentials are the same `postgres.credentials.*` values above, written by this template into the secret named here. +Set `postgresHA.enabled: true` for near-zero-downtime database upgrades and automatic failover: a three-replica Patroni cluster coordinated by etcd, fronted by an HAProxy leader endpoint that listmonk connects through for writes and schema migrations (the chart refuses to render if that proxy is disabled). The credentials come from the prerequisite secret named by `postgresHA.config.credentialsSecretName`. ```yaml postgres: @@ -242,10 +248,9 @@ postgres: postgresHA: enabled: true # set postgresHA.enabled: true for near-zero-downtime upgrades and failover config: - # Name of the dictionary secret this chart creates and the bundled - # postgres-highly-available reads. Since pgHA 2.5.0 the subchart creates no - # credentials secret of its own, so both stores are fed the same one, built - # from postgres.credentials.* above. + # REQUIRED PREREQUISITE SECRET (HA mode) — the same kind of `dictionary` + # secret as postgres.config.credentialsSecretName above (keys: username, + # password, database). Create it BEFORE you install. credentialsSecretName: my-listmonk-db-credentials replicas: 3 volumeset: @@ -297,8 +302,8 @@ The HA store's own tuning (Patroni resources, etcd compaction, PgBouncer) is doc | HTTP API | `https://CANONICAL_ENDPOINT/api/...` | Authenticated; lists, subscribers, campaigns, transactional mail | | Internal (same GVC) | `http://RELEASE_NAME-listmonk.GVC_NAME.cpln.local:9000` | Subject to `internalAccess.type` | | Admin credentials | `cpln secret reveal ADMIN_SECRET_NAME -o yaml` | The secret you created in [Prerequisites](#prerequisites) | -| Database (single mode) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Credentials: `cpln secret reveal my-listmonk-db-credentials -o yaml` (the name set by `postgres.config.credentialsSecretName`) | -| Database (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | HAProxy leader endpoint; same credentials secret | +| Database (single mode) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Credentials from your `postgres.config.credentialsSecretName` secret: `cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml` | +| Database (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | HAProxy leader endpoint; credentials from your `postgresHA.config.credentialsSecretName` secret | Read the public address once the workload is ready, then open `/admin` in your browser: @@ -396,7 +401,7 @@ cpln volumeset snapshot create RELEASE_NAME-listmonk-uploads --gvc GVC_NAME --sn ### Restoring a Backup -A restore replays the SQL dump into the backing database from a client inside the GVC that can reach both the bucket and the database — the per-provider commands are on the [PostgreSQL](/template-catalog/templates/postgres#restoring-a-backup) page for single mode and the [PostgreSQL Highly Available](/template-catalog/templates/postgres-highly-available#restoring-a-backup) page for HA mode. For this template, substitute the host `RELEASE_NAME-postgres.GVC_NAME.cpln.local` (single) or `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local` (HA), and the `username`, `password` and `database` values from `postgres.credentials.*`. +A restore replays the SQL dump into the backing database from a client inside the GVC that can reach both the bucket and the database — the per-provider commands are on the [PostgreSQL](/template-catalog/templates/postgres#restoring-a-backup) page for single mode and the [PostgreSQL Highly Available](/template-catalog/templates/postgres-highly-available#restoring-a-backup) page for HA mode. For this template, substitute the host `RELEASE_NAME-postgres.GVC_NAME.cpln.local` (single) or `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local` (HA), and the `username`, `password` and `database` values from your database credentials secret. This restore path has **not been verified against a live install of this template**; it is the backing store's own documented procedure. Restore while listmonk is not serving campaigns, and expect to restart the listmonk workload afterwards so it re-reads its settings. Restoring the uploads volume set from a snapshot is likewise unverified here — see [Volume Set snapshots](/reference/volumeset#snapshots). @@ -433,29 +438,15 @@ installing; see Prerequisites in the README. ### Upgrading from 1.1.0 -Version `1.2.0` moved the bundled database to the `postgres` 3.4.1 template, which no longer takes credentials as values. Listmonk absorbed that change rather than passing it on, so **there is no new prerequisite** — only a rename on the default (single-instance) path: +`postgres.config.username`, `.password` and `.database` were removed. Put the values your single-instance database was initialized with into the database credentials secret under [Prerequisites](#prerequisites) (keys `username`, `password`, `database`), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. -| Removed key | Replacement | -|---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | -| `postgres.backup.minio.accessKey` / `.secretKey` | `postgres.backup.minio.credentialsSecretName` (a dictionary secret you create; MinIO backups only) | - -Carrying an old key fails the render with the **PostgreSQL template's** message, which tells you to create a dictionary secret yourself. Ignore that advice here — this template creates it. Move the three keys and you are done. +With `postgres.backup.provider: minio`, `postgres.backup.minio.accessKey` / `.secretKey` were likewise removed — put them in a dictionary secret holding `accessKey` and `secretKey` as shown in [Backing Up](#backing-up) and set `postgres.backup.minio.credentialsSecretName` to its name. ### Upgrading from 1.2.0 Version `1.3.0` adopted the `postgres-highly-available` 2.5.0 template for the HA path. It affects only installs with `postgresHA.enabled: true`; single-instance installs need no action. -| Removed key | Replacement | -|---|---| -| `postgresHA.postgres.username` | `postgres.credentials.username` | -| `postgresHA.postgres.password` | `postgres.credentials.password` | -| `postgresHA.postgres.database` | `postgres.credentials.database` | -| `postgresHA.backup.minio.accessKey` / `.secretKey` | `postgresHA.backup.minio.credentialsSecretName` (a dictionary secret you create; MinIO backups only) | - -Carrying the old block forward fails at render with `the postgres block was REMOVED in 2.5.0`. Delete the `postgres` block under `postgresHA` and set `postgres.credentials.*` to the **same** values it held — the HA database was initialised with them. This template writes them into the secret named by `postgresHA.config.credentialsSecretName`, so do not create that secret yourself. If you use MinIO backups on the HA path, create a dictionary secret holding `accessKey` and `secretKey` as shown in [Backing Up](#backing-up) and set `postgresHA.backup.minio.credentialsSecretName` to its name. +Put the values of `postgresHA.postgres.username` / `.password` / `.database` **unchanged** into the database credentials secret under [Prerequisites](#prerequisites), set `postgresHA.config.credentialsSecretName` to its name, and delete the `postgres` block under `postgresHA`. The HA database was initialized with those values; a different password only changes the secret, and listmonk then fails to log in to its own database. If you use MinIO backups on the HA path, create a dictionary secret holding `accessKey` and `secretKey` as shown in [Backing Up](#backing-up) and set `postgresHA.backup.minio.credentialsSecretName` to its name. ### Upgrading from 1.3.1 @@ -466,7 +457,62 @@ Version `1.4.0` renamed the listmonk resource limits so it is unambiguous which | `resources.cpu` | `resources.maxCpu` | | `resources.memory` | `resources.maxMemory` | -Version `1.5.0` then added `rootUrl` and the boot-time Root URL handling described under [Root URL](#root-url); no values change is required for it. +Version `1.5.0` then added `rootUrl` and the boot-time Root URL handling described under [Root URL](#root-url); no values change is required for it. Then follow [Upgrading from 1.5.0](#upgrading-from-1-5-0). + +### Upgrading from 1.5.0 + +Version `1.6.0` removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the database templates require on their own. Carrying the old key forward fails the render. + +1.5.0 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or listmonk loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-listmonk-db \ + --entry username=listmonk --entry password='YOUR_CURRENT_PASSWORD' --entry database=listmonk +``` + +Then upgrade with `postgres.config.credentialsSecretName` (single-instance mode) or `postgresHA.config.credentialsSecretName` (HA mode) set to that name, and `postgres.credentials` removed. Your current values are in `postgres.credentials` of the 1.5.0 release, and also in the old secret (`cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is the `credentialsSecretName` your release used, `my-listmonk-db-credentials` by default) until the upgrade deletes it. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so listmonk can briefly lose its database; it reconnects by itself. While access to the new secret propagates, the new version of a workload can wait in `get-deployments` reporting that it is not allowed to reveal the secret; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating the Database Password + +A running workload keeps the credential values it started with, and PostgreSQL reads the secret only when its data is first initialized, so editing the secret alone does not change the database password. Change one key and keep the others with export, edit, apply: + +```bash +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only the key you are rotating in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +``` + +**Single-instance mode:** run `ALTER ROLE` for the secret's `username` in the database first, then update the secret, then right away force a redeployment of `RELEASE_NAME-listmonk`. Do not redeploy the database; restarting it only adds downtime. + +```bash +cpln workload force-redeployment RELEASE_NAME-listmonk --gvc GVC_NAME +``` + +**HA mode** needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Back to back: change BOTH roles on -0, put the same password in the secret (export, edit, apply above), restart the members +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 4. Once the members are back, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 5. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-listmonk`, which keeps the old password until it restarts. The app cannot open new database connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` ### Scaling and Availability @@ -478,7 +524,7 @@ Version `1.5.0` then added `rootUrl` and the boot-time Root URL handling describ - **Cause:** the secret named by `admin.secretName` did not exist at install time. The container never ran, so there is no output to log. + **Cause:** the secret named by `admin.secretName`, or the database credentials secret, did not exist at install time. The container never ran, so there is no output to log. A missing database secret shows the same way on `RELEASE_NAME-postgres` (or `RELEASE_NAME-postgres-ha`). **Fix:** read the one place the reason appears: @@ -512,9 +558,9 @@ Version `1.5.0` then added `rootUrl` and the boot-time Root URL handling describ listmonk: waiting for database... ``` - **Cause:** PostgreSQL is not accepting connections yet. A few of these lines on a fresh install are normal while the database initializes (the HA store also needs its leader endpoint to settle). If they continue indefinitely, listmonk cannot authenticate — typically because `postgres.credentials.*` was changed after the database volume was first initialized, so listmonk presents a password the database never received. + **Cause:** PostgreSQL is not accepting connections yet. A few of these lines on a fresh install are normal while the database initializes (the HA store also needs its leader endpoint to settle). If they continue indefinitely, listmonk cannot authenticate — typically because the database credentials secret was changed after the database volume was first initialized, so listmonk presents a password the database never received. - **Fix:** check the database workload with `cpln workload get-deployments RELEASE_NAME-postgres --gvc GVC_NAME -o yaml` (or `RELEASE_NAME-postgres-ha` in HA mode). If the credentials were changed after the first boot, restore the original values; the password is applied to the database on its first boot only. + **Fix:** check the database workload with `cpln workload get-deployments RELEASE_NAME-postgres --gvc GVC_NAME -o yaml` (or `RELEASE_NAME-postgres-ha` in HA mode). If the secret was changed after the first boot, restore the original values or follow [Rotating the Database Password](#rotating-the-database-password); the password is applied to the database on its first boot only. **Cause:** listmonk's Root URL is still the fresh-install default. On a public install the container sets it at boot; if that step could not run, the boot log carries a `WARN could not set app.root_url` line. On a private install (`publicAccess.enabled: false` with no `rootUrl`) nothing sets it. @@ -530,28 +576,19 @@ Version `1.5.0` then added `rootUrl` and the boot-time Root URL handling describ **Fix:** add your provider in **Settings → SMTP** (see [First Run](#first-run)) and start the campaign again. - - ```text - cannot be updated because it is being managed by a different release - ``` - - **Cause:** both releases use the same `postgres.config.credentialsSecretName` (or `postgresHA.config.credentialsSecretName`). Secret names are org-wide, and the first release owns the default `my-listmonk-db-credentials`. Nothing is shared, overwritten or deleted — the second install simply creates nothing. - - **Fix:** give the second release its own secret name and install again. - - **Cause:** your values file still carries a key removed in a later version — `admin.username` / `admin.password` (1.1.0), `postgres.config.username` / `.password` / `.database` (1.2.0), or `resources.cpu` / `resources.memory` (1.4.0). Each guard names its replacement, and nothing is applied, so the running release is untouched. + **Cause:** your values file still carries a key removed in a later version — `admin.username` / `admin.password` (1.1.0), `postgres.config.username` / `.password` / `.database` (1.2.0), `resources.cpu` / `resources.memory` (1.4.0), or `postgres.credentials` (1.6.0). Each guard names its replacement, and nothing is applied, so the running release is untouched. - **Fix:** follow the matching subsection above — [1.0.x](#upgrading-from-1-0-x), [1.1.0](#upgrading-from-1-1-0), or [1.3.1](#upgrading-from-1-3-1) — then upgrade again. + **Fix:** follow the matching subsection above — [1.0.x](#upgrading-from-1-0-x), [1.1.0](#upgrading-from-1-1-0), [1.3.1](#upgrading-from-1-3-1), or [1.5.0](#upgrading-from-1-5-0) — then upgrade again. ## Important Notes -- **Create the admin secret before installing.** A reference to a secret that does not exist wedges the workload with zero log lines; see [Prerequisites](#prerequisites) for the one command that shows the reason. +- **Create the admin and database credentials secrets before installing.** A reference to a secret that does not exist wedges the workload with zero log lines; see [Prerequisites](#prerequisites) for the one command that shows the reason. - **The Super Admin is created on first boot only.** Editing the secret later does not change the account, and upgrades never change your existing password — manage users in **Admin → Users**. -- **Change `postgres.credentials.password` before installing, not after.** It seeds the database on its first boot and is used exactly as given; changing it afterwards leaves listmonk with a password the database does not know. -- **Give each listmonk release in an org its own `postgres.config.credentialsSecretName`.** Secret names are org-wide; a second release left on the default name is refused at install. +- **Editing the database credentials secret does not change the database password.** Run `ALTER ROLE` first and then force-redeploy listmonk; in HA mode move the leader first — see [Rotating the Database Password](#rotating-the-database-password). +- **Give each listmonk release in an org its own database credentials secret.** Secret names are org-wide, so two releases pointed at one secret share a password. - **Do not scale listmonk past one replica or point a second instance at the same database** — duplicate campaign sends result. See [Scaling and Availability](#scaling-and-availability). - **No mail is delivered until SMTP is configured** in **Settings → SMTP**. Starting a campaign before that is not an error — it finishes with nothing sent. - **Keep `publicAccess` enabled for subscriber-facing pages to work,** and keep the Root URL equal to the address subscribers use — see [Root URL](#root-url). diff --git a/template-catalog/templates/litellm.mdx b/template-catalog/templates/litellm.mdx index 837d57ca..fbdc89a8 100644 --- a/template-catalog/templates/litellm.mdx +++ b/template-catalog/templates/litellm.mdx @@ -1,7 +1,7 @@ --- title: LiteLLM icon: /template-catalog/templates/icons/litellm.png -description: Deploy LiteLLM, the OpenAI-compatible LLM gateway with virtual keys, spend tracking, budgets and rate limiting, on Control Plane. A stateless proxy tier (one replica by default, scale with a knob) over bundled PostgreSQL and Sentinel Redis. Needs an existing GVC and a pre-created dictionary secret. +description: Deploy LiteLLM, the OpenAI-compatible LLM gateway with virtual keys, spend tracking, budgets and rate limiting, on Control Plane. A stateless proxy tier (one replica by default, scale with a knob) over bundled PostgreSQL and Sentinel Redis. Needs an existing GVC plus keys and database secrets. keywords: ['litellm', 'llm gateway', 'llm proxy', 'openai-compatible api', 'ai gateway', 'llm router', 'virtual api keys', 'llm spend tracking', 'openai proxy', 'self-hosted llm gateway', 'openrouter alternative', 'portkey alternative', 'model fallbacks', 'litellm template'] --- @@ -24,22 +24,19 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-sentinel` | Redis Sentinel that monitors the Redis master as `mymaster` and promotes a replica on failure (stateful, `redis.sentinel.replicas` replicas) | `redis.enabled: true` (default) | | Volume set | `RELEASE_NAME-pg-vs` | PostgreSQL data directory | — | | Secret | `RELEASE_NAME-litellm-config` | Opaque secret the chart creates with the proxy's `config.yaml` (model list and Redis cache block), mounted at `/app/config.yaml` | — | -| Secret | `RELEASE_NAME-litellm-db` | Dictionary secret the chart creates (`postgresUsername`, `postgresPassword`) from which the proxy assembles `DATABASE_URL` | — | -| Secret | the name in `postgres.config.credentialsSecretName` | Dictionary secret the chart creates (`username`, `password`, `database`) and hands to the bundled PostgreSQL | — | | Secret | `RELEASE_NAME-redis-config`, `RELEASE_NAME-sentinel-config` | Redis and Sentinel configuration files created by the bundled redis template | `redis.enabled: true` (default) | -| Secret | `RELEASE_NAME-redis-auth-password` | Redis master password, created by the bundled redis template and wired into the proxy's cache config | `redis.redis.auth.password.enabled: true` | | Identity | `RELEASE_NAME-litellm-identity` | Bound to the proxy workload | — | -| Policy | `RELEASE_NAME-litellm-policy` | Grants that identity `reveal` on exactly the secrets it reads: the config and database secrets, your prerequisite secret, and the Redis password secret when auth is on | — | +| Policy | `RELEASE_NAME-litellm-policy` | Grants that identity `reveal` on exactly the secrets it reads: the config secret, your keys and database credentials secrets, and your Redis password secret when auth is on | — | | Identity | `RELEASE_NAME-pg-identity` | Bound to the bundled PostgreSQL (and its backup job) | — | | Policy | `RELEASE_NAME-pg-policy` | Grants the PostgreSQL identity `reveal` on its credentials secret only | — | | Identity | `RELEASE_NAME-redis-identity`, `RELEASE_NAME-sentinel-identity` | Bound to the Redis and Sentinel workloads | `redis.enabled: true` (default) | | Policy | `RELEASE_NAME-redis-policy`, `RELEASE_NAME-sentinel-policy` | Grant those identities `reveal` on their configuration secrets | `redis.enabled: true` (default) | -Your master key, salt key, and provider API keys are **not** created by the template — see [Prerequisites](#prerequisites). +Your master key, salt key, and provider API keys, your database credentials secret and, when Redis auth is on, your Redis password secret are **not** created by the template — see [Prerequisites](#prerequisites). ## Prerequisites -LiteLLM needs three kinds of credential that never pass through Helm values: the **master key** that gates the Admin API and UI and mints virtual keys, the **salt key** that encrypts provider keys stored in PostgreSQL, and one **provider API key** per entry in `litellm.providerEnv`. All of them live in a single [dictionary secret](/guides/create-secret/dictionary) you create **before** installing; the chart references it by name and grants the proxy identity `reveal` on exactly that secret. +LiteLLM needs three kinds of credential that never pass through Helm values: the **master key** that gates the Admin API and UI and mints virtual keys, the **salt key** that encrypts provider keys stored in PostgreSQL, and one **provider API key** per entry in `litellm.providerEnv`. All of them live in a single [dictionary secret](/guides/create-secret/dictionary) you create **before** installing; the chart references it by name and grants the proxy identity `reveal` on exactly that secret. The bundled PostgreSQL's credentials live in a second dictionary secret you also create before installing (a third holds the Redis password when Redis auth is on). @@ -57,10 +54,22 @@ LiteLLM needs three kinds of credential that never pass through Helm values: the `LITELLM_SALT_KEY` is write-once: it encrypts every provider key stored in PostgreSQL, and changing it corrupts all of them. The master key can be rotated later; the salt key cannot once models or keys are stored. Keep a copy somewhere safe. + + The bundled PostgreSQL initializes with it and the proxy connects with it. Set `postgres.config.credentialsSecretName` to its name (default `my-litellm-db-credentials`); secret names are org-wide, so give each LiteLLM release its own: + + ```bash + cpln secret create-dictionary --name my-litellm-db-credentials \ + --entry username=litellm \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=litellm + ``` + + All three values must be URL-safe (letters, digits, `_`): the proxy's `DATABASE_URL` is assembled from them at container start. The snippet's hex password is URL-safe. PostgreSQL reads the secret only when its data directory is first initialized — see [Important Notes](#important-notes) for how to change the password later. + -Create the secret **before** installing. Installing without it reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message`: +Create the secrets **before** installing. Installing without one reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-litellm --gvc GVC_NAME -o yaml @@ -69,20 +78,20 @@ cpln workload get-deployments RELEASE_NAME-litellm --gvc GVC_NAME -o yaml The message reads `The secret LITELLM_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` Creating the secret lets the deployment recover on its own; `cpln workload force-redeployment RELEASE_NAME-litellm --gvc GVC_NAME` skips the wait. -Everything else works with the defaults. The bundled database password is template-managed plumbing — the chart creates the PostgreSQL credentials secret for you — and scheduled database backups, which need a bucket and cloud account, are described under [Backing Up](#backing-up). +Everything else works with the defaults. Redis authentication needs a third dictionary secret holding the key `password` (see [Bundled Redis](#bundled-redis)), and scheduled database backups, which need a bucket and cloud account, are described under [Backing Up](#backing-up). ## Installation -Once your dictionary secret exists, install from the marketplace registry, pointing `secrets.name` at it and replacing the bundled database password: +Once both dictionary secrets exist, install from the marketplace registry, pointing `secrets.name` and `postgres.config.credentialsSecretName` at them: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/litellm \ - --version 1.1.0 --gvc GVC_NAME \ + --version 1.2.1 --gvc GVC_NAME \ --set secrets.name=LITELLM_SECRET_NAME \ - --set postgres.credentials.password='YOUR-DB-PASSWORD' + --set postgres.config.credentialsSecretName=my-litellm-db-credentials ``` -Installing a second LiteLLM release in the same org? Also set `postgres.config.credentialsSecretName` to a name no other release uses — secret names are org-wide. +Installing a second LiteLLM release in the same org? Create a separate database credentials secret for it — secret names are org-wide. Other ways to install and manage the template: @@ -161,17 +170,13 @@ The public endpoint is key-gated, not open: every request still needs a valid ma ### Bundled PostgreSQL -The database is the [postgres](/template-catalog/templates/postgres) template as a subchart, so every knob of that template is available under `postgres.*`. The chart builds the credentials secret from `postgres.credentials.*` for you — nothing to create before installing. +The database is the [postgres](/template-catalog/templates/postgres) template as a subchart, so every knob of that template is available under `postgres.*`. Its credentials come from the prerequisite database credentials secret you created (see [Prerequisites](#prerequisites)). ```yaml postgres: image: postgres:18 - credentials: # the chart writes these into the secret named below - username: litellm - password: change-me-litellm-pg # change before installing — seeds the database on first boot - database: litellm config: - credentialsSecretName: my-litellm-db-credentials # chart-created; secret names are org-wide, so give each release its own + credentialsSecretName: my-litellm-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 250m maxCpu: 1000m @@ -196,9 +201,9 @@ redis: image: redis:7.4 replicas: 3 auth: - password: - enabled: false # true = require AUTH; the master password is wired into LiteLLM's cache config - value: change-me-litellm-redis # change before enabling + fromSecret: + enabled: false # true = require AUTH (Redis, Sentinel and the proxy read the secret below) + name: "" # prerequisite dictionary secret (key: password); used only when enabled resources: cpu: 200m memory: 256Mi @@ -206,7 +211,7 @@ redis: minMemory: 128Mi sentinel: image: redis:7.4 - replicas: 3 + replicas: 3 # no `auth:` block — Sentinel auth (incl. a plaintext `auth.password`) is refused at render resources: cpu: 200m memory: 256Mi @@ -214,10 +219,17 @@ redis: minMemory: 128Mi ``` -Redis ships authless by default, with the same-GVC firewall as the boundary. Set `redis.redis.auth.password.enabled: true` to require `AUTH`: the redis template then creates `RELEASE_NAME-redis-auth-password`, and the chart passes that master password into the proxy's cache config. Sentinel authentication must stay off — the proxy sends only the master password, and the chart refuses to render if `redis.sentinel.auth` is enabled. +Redis ships authless by default, with the same-GVC firewall as the boundary. To require `AUTH`, create a dictionary secret holding the key `password` **before** you install, then set `redis.redis.auth.fromSecret.enabled: true` and `redis.redis.auth.fromSecret.name` to its name. Redis, Sentinel and the proxy all read it: + +```bash +cpln secret create-dictionary --name my-litellm-redis \ + --entry password="$(openssl rand -hex 24)" +``` + +Sentinel authentication must stay off — the proxy sends only the master password, and the chart refuses to render with any `redis.sentinel.auth` setting, including a plaintext `redis.sentinel.auth.password`. Redis keeps no password on disk, so changing it later follows [Rotating the Redis Password](#rotating-the-redis-password). -The database password seeds PostgreSQL on first boot and is not updated by later value edits. Uninstalling the release deletes the volume set; reinstall to reset. +The database credentials secret seeds PostgreSQL on first boot; editing it later does not change the database password. Uninstalling the release deletes the volume set; reinstall to reset. ## Connecting @@ -229,7 +241,7 @@ The database password seeds PostgreSQL on first boot and is not updated by later | Health | `/health/readiness`, `/health/liveliness` | The endpoints the readiness and liveness probes use, on the same host. | | Master key | `LITELLM_MASTER_KEY` in your prerequisite secret | `cpln secret reveal LITELLM_SECRET_NAME -o yaml`. Send it as `Authorization: Bearer …`; it is also the Admin UI password. | | PostgreSQL | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | `username` / `password` / `database` from `cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is your `postgres.config.credentialsSecretName`. | -| Redis | `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` (Sentinel, master `mymaster`); `RELEASE_NAME-redis.GVC_NAME.cpln.local:6379` | Internal only. Password is the `password` key of `cpln secret reveal RELEASE_NAME-redis-auth-password -o yaml` when auth is on. | +| Redis | `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` (Sentinel, master `mymaster`); `RELEASE_NAME-redis.GVC_NAME.cpln.local:6379` | Internal only. When auth is on, the password is the `password` key of the secret named by `redis.redis.auth.fromSecret.name` (`cpln secret reveal SECRET_NAME -o yaml`). | Verify the proxy answers with your master key — the response lists the models from `modelList`: @@ -417,20 +429,73 @@ This restore path has not been exercised against this template end to end. Rehea ### Upgrading from 1.0.0 -Version 1.1.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. LiteLLM absorbed that change rather than passing it on — the chart now creates the credentials secret itself — so there is **no new prerequisite**, only a rename of three keys: +Version 1.1.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. Three keys were removed: -| Removed key (1.0.0) | Replacement (1.1.0) | +| Removed key (1.0.0) | Replacement (1.2.0 and later) | |---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | +| `postgres.config.username` | `username` key in the database credentials secret named by `postgres.config.credentialsSecretName` | +| `postgres.config.password` | `password` key in the same secret | +| `postgres.config.database` | `database` key in the same secret | -Move the three keys, keep the same values so the running database still matches, and upgrade in place. A values file that still carries an old key fails the render with the **postgres template's** message (`config.username was REMOVED in postgres 3.4.0`), which tells you to create a dictionary secret yourself — ignore that advice here; this template creates it, and the database password stays a value exactly as before. +Put the values your database was initialized with into the database credentials secret under [Prerequisites](#prerequisites), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. A values file that still carries an old key fails the render with the **postgres template's** message (`config.username was REMOVED in postgres 3.4.0`). The first `helm upgrade` after an install can restart the bundled PostgreSQL and Redis even when their values are unchanged, so expect a brief interruption. +### Upgrading from 1.1.0 + +Version 1.2.0 removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the `postgres` template requires on its own. Carrying the old key forward fails the render. + +1.1.0 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or the proxy loses its database connection. Read your current values from the old secret (named by `postgres.config.credentialsSecretName` in the installed release, `my-litellm-db-credentials` by default) before the upgrade deletes it: + +```bash +cpln secret reveal my-litellm-db-credentials -o yaml +cpln secret create-dictionary --name RELEASE_NAME-postgres-credentials \ + --entry username=litellm --entry password='YOUR_CURRENT_PASSWORD' --entry database=litellm +``` + +(`litellm` / `litellm` were the 1.1.0 defaults; use your own values if you overrode them.) **If you never overrode the password, it is literally `change-me-litellm-pg`** — put that in the new secret, and rotate it afterwards as described under [Important Notes](#important-notes). + +Then upgrade with `postgres.config.credentialsSecretName` set to the new name and `postgres.credentials` removed. The old `RELEASE_NAME-litellm-db` secret is deleted too; nothing replaces it. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so LiteLLM briefly loses its database; it reconnects by itself. The new version of a workload can also wait for several minutes, with `get-deployments` reporting that it is not allowed to reveal the secret, while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. + +**If you ran 1.1.0 with Redis auth on**, `redis.redis.auth.password` was removed too (carrying it forward fails the render), and the chart-created `RELEASE_NAME-redis-auth-password` secret is deleted by the upgrade. Before upgrading, copy your current Redis password into a secret under a **new** name, keeping the same value so the replicas keep replicating while they restart: + +```bash +cpln secret reveal RELEASE_NAME-redis-auth-password -o yaml +cpln secret create-dictionary --name RELEASE_NAME-litellm-redis \ + --entry password='YOUR_CURRENT_REDIS_PASSWORD' +``` + +Then upgrade with `redis.redis.auth.fromSecret.enabled: true`, `redis.redis.auth.fromSecret.name` set to the new name, and `redis.redis.auth.password` removed. With Redis auth off in 1.1.0 (the default), just remove `redis.redis.auth.password` from your values. Version 1.2.1 also refuses any `redis.sentinel.auth` setting, so remove that block too if your values carry one. + +### Rotating the Redis Password + +Only with Redis auth on. Redis keeps no password on disk, but a running Sentinel holds the one it started with, so a new password is applied live in three places before the proxy restarts. Each Redis node first accepts both passwords, which keeps replication and every connected client working throughout. `rx` and `sx` run a command on one Redis node or Sentinel (`LOCATION` is your GVC's location; nodes and Sentinels are numbered from 0 up to `redis.redis.replicas` and `redis.sentinel.replicas` minus one): + +```bash +rx() { cpln workload exec RELEASE_NAME-redis --gvc GVC_NAME --location LOCATION \ + --container redis --replica "RELEASE_NAME-redis-$1" -- "${@:2}"; } +sx() { cpln workload exec RELEASE_NAME-sentinel --gvc GVC_NAME --location LOCATION \ + --container sentinel --replica "RELEASE_NAME-sentinel-$1" -- "${@:2}"; } +# 1. On EVERY Redis node: accept the new password alongside the old one, and replicate with it +rx 0 redis-cli --no-auth-warning -a OLD_PASSWORD ACL SETUSER default ">NEW_PASSWORD" +rx 0 redis-cli --no-auth-warning -a OLD_PASSWORD CONFIG SET masterauth NEW_PASSWORD +# 2. On EVERY Sentinel +sx 0 redis-cli -p 26379 SENTINEL SET mymaster auth-pass NEW_PASSWORD +# 3. Put the new password in the secret and read it back +cpln secret edit SECRET_NAME +cpln secret reveal SECRET_NAME -o yaml +# 4. Restart the proxy, which keeps the old password until it restarts +cpln workload force-redeployment RELEASE_NAME-litellm --gvc GVC_NAME +# 5. Once get-deployments shows a higher version, alone and ready: retire the old password on EVERY Redis node +rx 0 redis-cli --no-auth-warning -a NEW_PASSWORD ACL SETUSER default " **Symptom:** `cpln helm install` reported success, `cpln logs` returns nothing, and `cpln workload get-deployments RELEASE_NAME-litellm --gvc GVC_NAME -o yaml` shows `The secret LITELLM_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` under `status.versions[].message`. - **Cause:** The prerequisite dictionary secret named by `secrets.name` does not exist, or it is missing one of the keys the workload references (`LITELLM_MASTER_KEY`, `LITELLM_SALT_KEY`, or a name listed in `litellm.providerEnv`). + **Cause:** The prerequisite dictionary secret named by `secrets.name` does not exist, or it is missing one of the keys the workload references (`LITELLM_MASTER_KEY`, `LITELLM_SALT_KEY`, or a name listed in `litellm.providerEnv`). The database credentials secret named by `postgres.config.credentialsSecretName`, or with Redis auth on the secret named by `redis.redis.auth.fromSecret.name`, may also be missing. **Fix:** Create the secret with every required key (see [Prerequisites](#prerequisites)). The deployment recovers on its own once the secret exists; `cpln workload force-redeployment RELEASE_NAME-litellm --gvc GVC_NAME` skips the wait. @@ -471,33 +536,33 @@ The bundled Redis is already highly available: three Redis replicas with three S A single `P1001` (cannot reach database server) followed by one restart on a cold install is normal and recovers on its own. - - **Symptom:** Installing a second LiteLLM release fails with `cannot be updated because it is being managed by a different release` and creates nothing. + + **Symptom:** `litellm: redis.redis.auth.fromSecret.name is required when redis.redis.auth.fromSecret.enabled is true`. - **Cause:** Both releases use the same `postgres.config.credentialsSecretName`. Secret names are org-wide, and the first release owns that one. Nothing is shared, overwritten, or deleted. + **Cause:** `redis.redis.auth.fromSecret.enabled` is `true` with an empty `redis.redis.auth.fromSecret.name`. - **Fix:** Set `postgres.config.credentialsSecretName` to a distinct name for the second release and install again. - - - **Symptom:** `litellm: redis.redis.auth.password.value is required when redis.redis.auth.password.enabled is true — LiteLLM's cache config reads this master password`. - - **Cause:** `redis.redis.auth.password.enabled` is `true` with an empty `redis.redis.auth.password.value`. - - **Fix:** Set `redis.redis.auth.password.value` to the password the bundled Redis should require, or set `enabled` back to `false`. + **Fix:** Create the Redis password secret and set `redis.redis.auth.fromSecret.name` to its name (see [Bundled Redis](#bundled-redis)), or set `enabled` back to `false`. - **Symptom:** `litellm: redis.sentinel.auth must stay disabled — LiteLLM sends only the master password, not a sentinel password`. + **Symptom:** `litellm: redis.sentinel.auth must stay disabled — LiteLLM sends only the Redis master password, not a Sentinel password`. - **Cause:** `redis.sentinel.auth.password.enabled` or `redis.sentinel.auth.fromSecret.enabled` is `true`. The proxy authenticates to the Redis master only, so a Sentinel password would lock it out. + **Cause:** Your values carry a `redis.sentinel.auth` setting — `redis.sentinel.auth.password.enabled`, a plaintext `redis.sentinel.auth.password.value`, or `redis.sentinel.auth.fromSecret.enabled`. The proxy authenticates to the Redis master only, so a Sentinel password would lock it out. - **Fix:** Leave both Sentinel auth toggles at `false`. Use `redis.redis.auth.password.enabled: true` to protect the master; the same-GVC firewall is the boundary for Sentinel. + **Fix:** Remove `redis.sentinel.auth` from your values. Use `redis.redis.auth.fromSecret` to protect the master; the same-GVC firewall is the boundary for Sentinel. **Symptom:** The render fails with `config.username was REMOVED in postgres 3.4.0` and the postgres template's advice to create a prerequisite dictionary secret. **Cause:** Your values file still carries `postgres.config.username`, `postgres.config.password`, or `postgres.config.database`. - **Fix:** Rename them to `postgres.credentials.*` as described in [Upgrading from 1.0.0](#upgrading-from-1-0-0). Do not create a secret yourself — this template creates it. + **Fix:** Put their values into the database credentials secret and remove the keys, as described in [Upgrading from 1.0.0](#upgrading-from-1-0-0). + + + **Symptom:** The render fails with `postgres.credentials was removed in litellm 1.2.0` or `redis.redis.auth.password was removed in litellm 1.2.0`. + + **Cause:** Your values file still carries the 1.1.0 plaintext database credentials or Redis password. + + **Fix:** Create the prerequisite secrets holding your current values and remove the old keys, as described in [Upgrading from 1.1.0](#upgrading-from-1-1-0). **Symptom:** API calls return `401`, including on the public endpoint. @@ -510,13 +575,13 @@ The bundled Redis is already highly available: three Redis replicas with three S ## Important Notes -- **Create the prerequisite secret before installing.** `secrets.name` must be a dictionary secret with `LITELLM_MASTER_KEY`, `LITELLM_SALT_KEY`, and every `litellm.providerEnv` key; without it the deployment wedges silently. +- **Create every prerequisite secret before installing.** `secrets.name` must be a dictionary secret with `LITELLM_MASTER_KEY`, `LITELLM_SALT_KEY`, and every `litellm.providerEnv` key; `postgres.config.credentialsSecretName` one with `username`, `password` and `database`; and, with Redis auth on, `redis.redis.auth.fromSecret.name` one with `password`. A missing one wedges the deployment silently. - **Never rotate `LITELLM_SALT_KEY`.** It encrypts the provider keys stored in PostgreSQL; changing it corrupts every stored key. The master key may be rotated — force a redeployment of `RELEASE_NAME-litellm` afterwards so the proxy picks up the new value. - **Hand out virtual keys, not the master key.** The public endpoint is key-gated, and virtual keys carry their own budgets, rate limits, and model lists. - **Keep Redis enabled for `litellm.replicas` of 2 or more.** Otherwise rate limits are counted per replica. -- **Redis is authless by default**, bounded by the same-GVC firewall. Set `redis.redis.auth.password.enabled: true` (with a changed `value`) to require a master password; Sentinel auth must stay off. -- **Give each release its own `postgres.config.credentialsSecretName`.** Secret names are org-wide; a second release on the default name is refused at install. -- **Change `postgres.credentials.password` before installing.** It seeds the database on first boot and later value edits do not change it. +- **Redis is authless by default**, bounded by the same-GVC firewall. To require a master password, create the Redis password secret and set `redis.redis.auth.fromSecret.enabled: true` with its name; changing it later follows [Rotating the Redis Password](#rotating-the-redis-password). Sentinel auth must stay off. +- **Give each release its own database credentials secret.** Secret names are org-wide. +- **Rotating the database password needs `ALTER ROLE` plus a forced redeployment of the proxy** (not the database). PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with. Run `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` first, then `cpln secret edit SECRET_NAME`, then right away `cpln workload force-redeployment RELEASE_NAME-litellm --gvc GVC_NAME`, and wait until `get-deployments` shows a higher version number, alone and ready. Connections the proxy already has open keep working after `ALTER ROLE`; new ones fail until its redeploy finishes, so rotate in a quiet window. - **PostgreSQL data survives proxy restarts and redeploys.** To reset virtual keys and spend, uninstall the whole release — that deletes the volume set. ## External References diff --git a/template-catalog/templates/metabase.mdx b/template-catalog/templates/metabase.mdx index 6f7ea6e1..cf406f6c 100644 --- a/template-catalog/templates/metabase.mdx +++ b/template-catalog/templates/metabase.mdx @@ -1,7 +1,7 @@ --- title: Metabase icon: /template-catalog/templates/icons/metabase.png -description: Metabase open-source BI — dashboards, SQL editor and scheduled reports — as a single stateless workload over a bundled PostgreSQL app database (highly available by default, single-instance by one flag), deployed into an existing GVC. Two prerequisite secrets must exist first. +description: Metabase open-source BI — dashboards, SQL editor and scheduled reports — as a single stateless workload over a bundled PostgreSQL app database (highly available by default, single-instance by one flag), deployed into an existing GVC. Three prerequisite secrets must exist first. keywords: ['business intelligence', 'data visualization', 'looker alternative', 'tableau alternative', 'power bi alternative', 'superset alternative', 'redash alternative', 'self-hosted', 'reporting', 'charts', 'sql questions', 'patroni', 'encryption key'] --- @@ -26,7 +26,6 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-postgres-backup` | Scheduled logical dump of the single-instance database (cron) | `postgres.backup.enabled: true` | | Volume set | `RELEASE_NAME-postgres-ha-vs`, `RELEASE_NAME-etcd-vs` | Highly available database and consensus-store data (10 GiB per replica by default; takes no scheduled snapshots; a final snapshot is taken when the volume set is deleted and kept 7 days) | Default; `postgresHA.enabled: true` | | Volume set | `RELEASE_NAME-pg-vs` | Single-instance database data (10 GiB by default; takes no scheduled snapshots; a final snapshot is taken when the volume set is deleted and kept 7 days) | `postgres.enabled: true` | -| Secret | The name in `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` (default `my-metabase-db-credentials`) | App-database credentials, **created by this chart** from `postgres.credentials.*` and read by whichever database is enabled | — | | Secret | `RELEASE_NAME-metabase-start` | Start script that runs the first-boot admin setup and hands off to Metabase | — | | Secret | `RELEASE_NAME-etcd-startup`, `RELEASE_NAME-patroni-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-wal-g-backup-script` | Startup scripts of the highly available database tier | Default; `postgresHA.enabled: true` | | Secret | `RELEASE_NAME-postgres-config` | Non-sensitive backup settings of the highly available database (bucket, region, endpoint) | `postgresHA.backup.enabled: true` | @@ -34,11 +33,11 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Identity / Policy | `RELEASE_NAME-postgres-ha-identity`, `RELEASE_NAME-postgres-ha-policy`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy` | The highly available database tier's own identities and policies | Default; `postgresHA.enabled: true` | | Identity / Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | The single-instance database's own identity and policy | `postgres.enabled: true` | -Metabase itself has no volume set — every question, dashboard, user and saved connection lives in the app database. The admin login and the encryption key are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. +Metabase itself has no volume set — every question, dashboard, user and saved connection lives in the app database. The admin login, the encryption key and the app-database credentials are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. ## Prerequisites -Two secrets must exist before you install. The admin login guards a form on a public endpoint and the encryption key is long-lived key material, so neither is a Helm value and neither ever lands in the release. Secrets are org-level, so no `--gvc` flag is involved. +Three secrets must exist before you install. The admin login guards a form on a public endpoint, the encryption key is long-lived key material and the app-database credentials are credentials, so none is a Helm value and none ever lands in the release. Secrets are org-level, so no `--gvc` flag is involved. @@ -61,8 +60,17 @@ Two secrets must exist before you install. The admin login guards a form on a pu Set `encryptionKey.secretName` to this name, and store a copy of the key somewhere safe outside Control Plane. - - The bundled database's credentials are internal plumbing: this chart writes `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database` into a dictionary secret it creates, and the database reads that secret. Nothing to create — but the password ships as the placeholder `change-me-metabase-db-password`, so set `postgres.credentials.password` to a value of your own before installing. This applies in **both** database modes, even with `postgres.enabled: false`. + + A [dictionary secret](/guides/create-secret/dictionary) with the keys `username`, `password` and `database`. The bundled PostgreSQL initializes with it and Metabase connects with it: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=metabase \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=metabase + ``` + + Set its name in the key of the database mode you use: `postgresHA.config.credentialsSecretName` (HA, the default) or `postgres.config.credentialsSecretName` (single-instance). Secret names are org-wide, so give each Metabase release its own secret. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password) for changing it later. @@ -77,26 +85,26 @@ Two secrets must exist before you install. The admin login guards a form on a pu cpln workload get-deployments RELEASE_NAME-metabase --gvc GVC_NAME -o yaml ``` -The message reads `The secret no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` It is `get-deployments` — plain `cpln workload get` has no `versions` key. Create the missing secret and the deployment recovers on its own after several minutes, or run `cpln workload force-redeployment RELEASE_NAME-metabase --gvc GVC_NAME` to skip the wait. +The message reads `The secret no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` It is `get-deployments` — plain `cpln workload get` has no `versions` key. A missing app-database secret also holds the database workload (`RELEASE_NAME-postgres-ha`, or `RELEASE_NAME-postgres` in single-instance mode), and Metabase recovers after it. Create the missing secret and the deployment recovers on its own after several minutes; to skip the wait, force a redeployment of the database workload first, then run `cpln workload force-redeployment RELEASE_NAME-metabase --gvc GVC_NAME`. -The two secrets have **different lifecycles** — read [Admin Account and Encryption Key](#admin-account-and-encryption-key) before you change or delete either one. +The admin and encryption-key secrets have **different lifecycles** — read [Admin Account and Encryption Key](#admin-account-and-encryption-key) before you change or delete either one. Optional database backups need a bucket and access setup on AWS S3, Google Cloud Storage or an S3-compatible server, and S3-compatible servers additionally need a dictionary secret with the bucket keys — see [Backing Up](#backing-up). ## Installation -Once both secrets exist, install the template into your GVC, naming the secrets and replacing the database password placeholder: +Once all three secrets exist, install the template into your GVC, naming the secrets: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/metabase \ - --version 1.4.0 --gvc GVC_NAME \ + --version 1.5.0 --gvc GVC_NAME \ --set admin.secretName=ADMIN_SECRET_NAME \ --set encryptionKey.secretName=ENCRYPTION_KEY_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' + --set postgresHA.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME ``` -Prefer a values file (`--values`) over `--set` for the database password if you do not want it in your shell history. If you run more than one release of this template in the same org, also give each its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName` in single-instance mode) — see [App Database](#app-database). +In single-instance mode set `postgres.config.credentialsSecretName` instead — see [App Database](#app-database). @@ -206,7 +214,7 @@ Exactly one of the two database modes must be enabled — the chart enforces thi postgresHA: # default: highly available PostgreSQL app database enabled: true config: - credentialsSecretName: my-metabase-db-credentials # dictionary secret this chart CREATES from postgres.credentials.*; org-wide name — give each release its own + credentialsSecretName: my-metabase-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install replicas: 3 volumeset: capacity: 10 # initial capacity in GiB per replica (minimum is 10) @@ -217,19 +225,15 @@ postgresHA: enabled: false # set postgresHA.enabled: true (the default) for automatic database failover postgres: # dev/lightweight: single-instance PostgreSQL enabled: true - credentials: # the chart writes these into the secret named below — nothing for you to create - username: metabase - password: change-me-metabase-db-password # change before installing - database: metabase config: - credentialsSecretName: my-metabase-db-credentials # secret names are org-wide — give each release its own + credentialsSecretName: my-metabase-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) ``` -In **both** modes the database credentials come from `postgres.credentials.*` — the chart writes them into the dictionary secret named by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` (whichever database is enabled), and the database reads that secret. Change `postgres.credentials.password` before installing even when `postgres.enabled` is `false`. In HA mode the proxy must stay enabled — that endpoint is Metabase's stable database address, and the chart refuses to render with `postgresHA.proxy.enabled: false`. +In **both** modes the database credentials come from the prerequisite secret named by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` (whichever database is enabled); the database initializes from it and Metabase connects with it. In HA mode the proxy must stay enabled — that endpoint is Metabase's stable database address, and the chart refuses to render with `postgresHA.proxy.enabled: false`. -Secret names are org-wide, so a second release of this template left on the default `credentialsSecretName` is **refused at install** with `cannot be updated because it is being managed by a different release` and creates nothing — the first release is unaffected. Give each release its own name. +Secret names are org-wide, so give each release its own credentials secret; two releases pointed at one secret share a password. Everything else the bundled databases can do — volume autoscaling, PgBouncer pooling, etcd tuning — is documented on the [postgres-highly-available](/template-catalog/templates/postgres-highly-available) and [postgres](/template-catalog/templates/postgres) pages and is set under the same `postgresHA.*` / `postgres.*` prefixes. Their backup settings are covered under [Backing Up](#backing-up). @@ -479,29 +483,25 @@ Two changes landed in the 1.0.x to 1.1.0 range; apply both when upgrading from 1 ### Upgrading from 1.1.0 -Version 1.2.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. Metabase absorbed that change — the chart now creates the credentials secret itself — so there is **no new prerequisite**, only a rename of three keys in single-instance mode: +Version 1.2.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values: -| Removed key (1.1.0) | Replacement (1.2.0 and later) | +| Removed key (1.1.0) | Replacement | |---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | +| `postgres.config.username` / `.password` / `.database` | the app-database credentials secret under [Prerequisites](#prerequisites) — put your existing values in it **unchanged** and set `postgres.config.credentialsSecretName` to its name | | `postgres.backup.minio.accessKey` / `postgres.backup.minio.secretKey` | a dictionary secret named by `postgres.backup.minio.credentialsSecretName` | -Move the keys **keeping the same values** — PostgreSQL enforces the credentials written when the volume was first created, and different values just leave Metabase unable to authenticate. A values file that still carries an old key fails the render with the **postgres template's** message, which tells you to create a dictionary secret yourself — ignore that advice here; this template creates it from `postgres.credentials.*`. If you back up to MinIO, create the dictionary secret described under [Backing Up](#backing-up) and set `postgres.backup.minio.credentialsSecretName`. The HA path is unchanged by this step. +Keep the **same values** — PostgreSQL enforces the credentials written when the volume was first created, and different values just leave Metabase unable to authenticate. A values file that still carries an old key fails the render with the **postgres template's** message, which tells you to create a dictionary secret — that is correct here too. If you back up to MinIO, create the dictionary secret described under [Backing Up](#backing-up) and set `postgres.backup.minio.credentialsSecretName`. The HA path is unchanged by this step. ### Upgrading from 1.2.0 -Version 1.3.0 adopted the `postgres-highly-available` 2.5.0 template, which made the same change on the HA path, and 1.3.1 did the same for its MinIO backup keys. The HA database credentials now come from the same `postgres.credentials.*` block as the single-instance ones, and the chart creates the secret named by `postgresHA.config.credentialsSecretName` for both stores. +Version 1.3.0 adopted the `postgres-highly-available` 2.5.0 template, which made the same change on the HA path, and 1.3.1 did the same for its MinIO backup keys. -| Removed key (1.2.0) | Replacement (1.3.x and later) | +| Removed key (1.2.0) | Replacement | |---|---| -| `postgresHA.postgres.username` | `postgres.credentials.username` | -| `postgresHA.postgres.password` | `postgres.credentials.password` | -| `postgresHA.postgres.database` | `postgres.credentials.database` | +| `postgresHA.postgres.username` / `.password` / `.database` | the app-database credentials secret under [Prerequisites](#prerequisites) — put your existing values in it **unchanged** and set `postgresHA.config.credentialsSecretName` to its name | | `postgresHA.backup.minio.accessKey` / `postgresHA.backup.minio.secretKey` | a dictionary secret named by `postgresHA.backup.minio.credentialsSecretName` (1.3.1) | -Move the three credential values **unchanged** into `postgres.credentials.*` — the running cluster was initialized with them — and delete the `postgresHA.postgres` block entirely. Leaving `postgres.enabled: false` is correct; the `postgres.credentials.*` values feed both database modes. A values file that still carries the old block stops the upgrade at render: +Keep the values **unchanged** — the running cluster was initialized with them — and delete the `postgresHA.postgres` block entirely. A values file that still carries the old block stops the upgrade at render: ```text Error: ... postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres.username`, @@ -510,7 +510,7 @@ Error: ... postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres:` block from your values. See Prerequisites in the README. ``` -The message's instruction to create the secret yourself does not apply — this template creates it. If you back up the HA database to an S3-compatible server, create the credentials secret described under [Backing Up](#backing-up) with the same access key and secret key, set `postgresHA.backup.minio.credentialsSecretName`, and remove the two old keys; if you are already on 1.3.0, that MinIO change is the only part of this step that applies. The details of the subchart change are under [Upgrading From 2.4.x](/template-catalog/templates/postgres-highly-available#upgrading-from-2-4-x) on the postgres-highly-available page. +The message's instruction to create the secret yourself is correct here too. If you back up the HA database to an S3-compatible server, create the credentials secret described under [Backing Up](#backing-up) with the same access key and secret key, set `postgresHA.backup.minio.credentialsSecretName`, and remove the two old keys; if you are already on 1.3.0, that MinIO change is the only part of this step that applies. The details of the subchart change are under [Upgrading From 2.4.x](/template-catalog/templates/postgres-highly-available#upgrading-from-2-4-x) on the postgres-highly-available page. ### Upgrading from 1.3.x @@ -523,12 +523,67 @@ Version 1.4.0 renamed the Metabase container's resource limits: a block that exp An upgrade that still carries either old key is refused at render with `metabase: resources.cpu was RENAMED to resources.maxCpu` (or the `memory` equivalent); rename the two keys and upgrade. `resources.minCpu` and `resources.minMemory` keep their names. +### Upgrading from 1.4.0 + +Version 1.5.0 removed `postgres.credentials`: the app-database credentials are now a prerequisite secret, the same one the database templates require on their own. Carrying the old key forward fails the render. + +1.4.0 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Metabase loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-metabase-db \ + --entry username=metabase --entry password='YOUR_CURRENT_PASSWORD' --entry database=metabase +``` + +Then upgrade with `postgresHA.config.credentialsSecretName` (HA mode) or `postgres.config.credentialsSecretName` (single-instance mode) set to that name, and `postgres.credentials` removed. Your current values are in `postgres.credentials` of the 1.4.0 release, and also in the old secret (`cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is the `credentialsSecretName` your release used, `my-metabase-db-credentials` by default) until the upgrade deletes it. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Metabase can briefly lose its database; it reconnects by itself. While access to the new secret propagates, the new version of a workload can wait in `get-deployments` reporting that it is not allowed to reveal the secret; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating the Database Password + +A running workload keeps the credential values it started with, and PostgreSQL reads the secret only when its data is first initialized, so editing the secret alone does not change the database password. Change one key and keep the others with export, edit, apply: + +```bash +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only the key you are rotating in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +``` + +**Single-instance mode:** run `ALTER ROLE` for the secret's `username` in the database first, then update the secret, then right away force a redeployment of `RELEASE_NAME-metabase`. Do not redeploy the database; restarting it only adds downtime. + +```bash +cpln workload force-redeployment RELEASE_NAME-metabase --gvc GVC_NAME +``` + +**HA mode** (the default) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Back to back: change BOTH roles on -0, put the same password in the secret (export, edit, apply above), restart the members +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 4. Once the members are back, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 5. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-metabase`, which keeps the old password until it restarts. The app cannot open new database connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + ### Scaling and Availability - **Metabase runs as a single replica in this template**, and no `replicas` knob is exposed. It holds no state of its own, so a replaced replica comes back with everything intact; the highly available default database removes the database as a failure point. - **Database replicas** are set with `postgresHA.replicas`; the single-instance database must stay at one replica. - **Every Helm upgrade restarts the Metabase replica**, so expect a short UI outage per upgrade. The first `helm upgrade` after an install can additionally restart the bundled database even when no database value changed. -- **Rotating a prerequisite secret does not restart anything.** The admin secret is only read on first boot, so there is nothing to rotate there — change the password in the UI. Never rotate the encryption key. After fixing a rejected admin secret on a fresh install, run `cpln workload force-redeployment RELEASE_NAME-metabase --gvc GVC_NAME` so the container reads the new contents. +- **Rotating a prerequisite secret does not restart anything.** The admin secret is only read on first boot, so there is nothing to rotate there — change the password in the UI. Never rotate the encryption key. The app-database password is rotated as described in [Rotating the Database Password](#rotating-the-database-password). After fixing a rejected admin secret on a fresh install, run `cpln workload force-redeployment RELEASE_NAME-metabase --gvc GVC_NAME` so the container reads the new contents. - **Confirm an upgrade is live from the deployment, not from the Helm output** — `cpln workload get-deployments RELEASE_NAME-metabase --gvc GVC_NAME -o yaml`. - **Uninstall deletes every volume set the chart created** — the database and, in HA mode, the consensus store — so the data does not survive a reinstall. The GVC is yours and is left alone, and the prerequisite secrets you created survive. Enable backups if the data matters. @@ -538,7 +593,7 @@ An upgrade that still carries either old key is refused at render with `metabase **Symptom:** `cpln helm install` succeeded, but `RELEASE_NAME-metabase` shows no containers and `cpln logs` returns nothing. - **Cause:** A prerequisite secret named in your values does not exist — the admin secret or the encryption-key secret. The deployment pauses on the unresolved reference. + **Cause:** A prerequisite secret named in your values does not exist — the admin secret, the encryption-key secret or the app-database credentials secret. The deployment pauses on the unresolved reference; a missing app-database secret holds the database workload the same way. **Fix:** Read the message naming the secret, then create it: @@ -573,13 +628,7 @@ An upgrade that still carries either old key is refused at render with `metabase **Fix:** Enable exactly one — see [App Database](#app-database). To use the single-instance database, set `postgresHA.enabled: false` together with `postgres.enabled: true`. - - **Symptom:** Installing a second release of this template in the same org fails with `cannot be updated because it is being managed by a different release`, naming the database credentials secret, and creates nothing. - - **Cause:** Secret names are org-wide, and both releases are trying to create the same `credentialsSecretName` (default `my-metabase-db-credentials`). The first release owns it. - **Fix:** Give the new release its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName` in single-instance mode) and install again. The first release is unaffected. - **Symptom:** Data sources that worked before now fail, and Metabase cannot read the connection details it stored for them. @@ -588,21 +637,22 @@ An upgrade that still carries either old key is refused at render with `metabase **Fix:** Restore the original key from your backup of it and run `cpln workload force-redeployment RELEASE_NAME-metabase --gvc GVC_NAME`. If the original key is gone, re-enter the credentials of every saved data source. - **Symptom:** `helm upgrade` stops at render with `admin.password was REMOVED in 1.1.0`, `config.username was REMOVED in postgres 3.4.0`, `the postgres block was REMOVED in 2.5.0`, `backup.minio.accessKey and backup.minio.secretKey were REMOVED`, or `resources.cpu was RENAMED to resources.maxCpu`. + **Symptom:** `helm upgrade` stops at render with `admin.password was REMOVED in 1.1.0`, `config.username was REMOVED in postgres 3.4.0`, `the postgres block was REMOVED in 2.5.0`, `backup.minio.accessKey and backup.minio.secretKey were REMOVED`, `resources.cpu was RENAMED to resources.maxCpu`, or `postgres.credentials was REMOVED in 1.5.0`. - **Cause:** Your values still carry a key that moved in template versions 1.1.0, 1.2.0, 1.3.0, 1.3.1 or 1.4.0. The existing release is left untouched and running. + **Cause:** Your values still carry a key that moved in template versions 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0 or 1.5.0. The existing release is left untouched and running. - **Fix:** Follow the matching migration note above — [Upgrading from 1.0.x](#upgrading-from-1-0-x), [Upgrading from 1.1.0](#upgrading-from-1-1-0), [Upgrading from 1.2.0](#upgrading-from-1-2-0) or [Upgrading from 1.3.x](#upgrading-from-1-3-x) — keeping the credentials the database already uses. Do not create a database credentials secret yourself; this template creates it. + **Fix:** Follow the matching migration note above — [Upgrading from 1.0.x](#upgrading-from-1-0-x), [Upgrading from 1.1.0](#upgrading-from-1-1-0), [Upgrading from 1.2.0](#upgrading-from-1-2-0), [Upgrading from 1.3.x](#upgrading-from-1-3-x) or [Upgrading from 1.4.0](#upgrading-from-1-4-0) — keeping the credentials the database already uses in the credentials secret you create. ## Important Notes -- **Create the admin and encryption-key secrets before installing**, and change `postgres.credentials.password` from its `change-me` placeholder. A missing prerequisite secret does not fail the install — the workload sits at zero replicas waiting on it. +- **Create the admin, encryption-key and app-database credentials secrets before installing.** A missing prerequisite secret does not fail the install — the workload sits at zero replicas waiting on it. - **The encryption-key secret is permanent: never delete it, never rotate it.** Back the key up outside Control Plane; a database backup is useless without it. - **The admin secret applies on first boot only.** Change the password in the Metabase UI; editing the secret later changes nothing, and an upgrade never resets your password. - **Neither the admin email nor the password may contain a double quote or a backslash**, and the password needs letters and digits, 8 or more characters — a rejected setup keeps the workload unready by design rather than exposing a setup page. -- **Give each release its own `credentialsSecretName`** — secret names are org-wide, and a second release on the default name is refused at install. +- **Give each release its own app-database credentials secret** — secret names are org-wide, so two releases pointed at one secret share a password. +- **Editing the app-database credentials secret does not change the database password.** Run `ALTER ROLE` first and then force-redeploy Metabase; in HA mode move the leader first — see [Rotating the Database Password](#rotating-the-database-password). - **Questions, dashboards and users live in the app database** and survive Metabase restarts. **Uninstall deletes the database volume sets** — enable backups if the data matters. - **Metabase is single-replica in this template** — every upgrade restarts it; the highly available default database keeps running and no data is lost. - **Prerequisite secrets you create are not owned by the release** — they survive `helm uninstall` and must be deleted manually if you no longer need them. diff --git a/template-catalog/templates/n8n.mdx b/template-catalog/templates/n8n.mdx index c473fd27..a1ca5d91 100644 --- a/template-catalog/templates/n8n.mdx +++ b/template-catalog/templates/n8n.mdx @@ -1,7 +1,7 @@ --- title: n8n icon: /template-catalog/templates/icons/n8n.png -description: Deploy n8n workflow automation on Control Plane with a bundled PostgreSQL database — highly available by default, single-instance optional — into an existing GVC. Two secrets must exist before install, the credential-encryption key and the owner login as a bcrypt hash. +description: Deploy n8n workflow automation on Control Plane with a bundled PostgreSQL database — highly available by default, single-instance optional — into an existing GVC. Three secrets must exist before install: the credential-encryption key, the owner login as a bcrypt hash, and the database credentials. keywords: ['zapier alternative', 'make alternative', 'ipaas', 'no-code', 'low-code', 'self-hosted', 'fair-code', 'integration platform', 'api orchestration', 'scheduled triggers', 'webhook automation', 'n8n template', 'prerequisite secret', 'bcrypt hash', 'htpasswd'] --- @@ -9,7 +9,7 @@ keywords: ['zapier alternative', 'make alternative', 'ipaas', 'no-code', 'low-co n8n is a workflow automation platform (fair-code, distributed under n8n's [Sustainable Use License](https://docs.n8n.io/choose-n8n/faircode-license/)). This template deploys a single n8n instance serving the editor, REST API, and webhooks on one HTTPS endpoint, backed by a bundled PostgreSQL database — the [postgres-highly-available](/template-catalog/templates/postgres-highly-available) template by default, or the single-instance [postgres](/template-catalog/templates/postgres) template for lighter installs. -The instance owner account is pre-provisioned from a secret at install, so there is never an unauthenticated setup page. Both the owner login and the credential-encryption key come from secrets you create **before** installing; neither passes through Helm values. +The instance owner account is pre-provisioned from a secret at install, so there is never an unauthenticated setup page. The owner login, the credential-encryption key and the database credentials come from secrets you create **before** installing; none passes through Helm values. This template does not create a GVC. It deploys into an existing GVC that you already have. @@ -22,9 +22,8 @@ This template does not create a GVC. It deploys into an existing GVC that you al | Workload | `RELEASE_NAME-n8n` | n8n editor, API, and webhooks (stateful, 1 replica, port `5678`) | — | | Volume set | `RELEASE_NAME-n8n-vs` | `/home/node/.n8n` — instance config and binary execution data | — | | Secret | `RELEASE_NAME-n8n-start` | Start script that derives the public URLs from the canonical endpoint | — | -| Secret | `my-n8n-db-credentials` | Database credentials for the bundled PostgreSQL, built from `postgres.credentials.*`; name set by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` | — | | Identity | `RELEASE_NAME-n8n-identity` | Workload identity | — | -| Policy | `RELEASE_NAME-n8n-policy` | `reveal` on exactly the four secrets n8n reads, including your two prerequisite secrets | — | +| Policy | `RELEASE_NAME-n8n-policy` | `reveal` on exactly the four secrets n8n reads, including your three prerequisite secrets | — | | Workload | `RELEASE_NAME-postgres-ha` | Patroni PostgreSQL cluster (stateful, 3 replicas) | `postgresHA.enabled: true` (default) | | Workload | `RELEASE_NAME-etcd` | etcd for Patroni leader election (stateful, 3 replicas) | `postgresHA.enabled: true` (default) | | Workload | `RELEASE_NAME-postgres-ha-proxy` | HAProxy leader endpoint n8n connects through (standard) | `postgresHA.enabled: true` (default) | @@ -38,7 +37,7 @@ The highly available database also creates its own identities, policies, and sta ## Prerequisites -**Two secrets must exist before you install.** Secrets are org-level, so no GVC flag is involved. +**Three secrets must exist before you install.** Secrets are org-level, so no GVC flag is involved. @@ -77,6 +76,18 @@ The highly available database also creates its own identities, policies, and sta | `email` | The instance owner's login email. | | `passwordHash` | A bcrypt hash of the owner password. | + + A [dictionary secret](/guides/create-secret/dictionary) with the keys `username`, `password` and `database`. The bundled PostgreSQL initializes with it and n8n connects with it: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=n8n \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=n8n + ``` + + Set the `credentialsSecretName` of the database mode you use to this name — `postgresHA.config.credentialsSecretName` (default, HA) or `postgres.config.credentialsSecretName` (single-instance). PostgreSQL applies it only when its data directory is first initialized, so editing the secret later does not change the database password — see [Rotating the Database Password](#rotating-the-database-password). Secret names are org-wide, so give each n8n release its own. + @@ -97,22 +108,22 @@ cpln workload force-redeployment RELEASE_NAME-n8n --gvc GVC_NAME ``` -The bundled database password is **not** a prerequisite: this template creates the database credentials secret itself from `postgres.credentials.*`. Change `postgres.credentials.password` before installing — it is used exactly as written. - Optional database backups need a bucket and access setup for one provider, and backups to MinIO need one more dictionary secret — see [Backing Up](#backing-up). ## Installation -Once both secrets exist, install from the marketplace registry with the two secret names and a database password of your own: +Once all three secrets exist, install from the marketplace registry with the three secret names: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/n8n \ - --version 1.3.1 --gvc GVC_NAME \ + --version 1.4.0 --gvc GVC_NAME \ --set encryptionKey.secretName=ENCRYPTION_KEY_SECRET_NAME \ --set owner.secretName=OWNER_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-PASSWORD' + --set postgresHA.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME ``` +In single-instance mode set `postgres.config.credentialsSecretName` instead. + Or follow the instructions for your preferred method: @@ -190,7 +201,7 @@ With `publicAccess.enabled: false`, the editor and webhooks are reachable only f ### Database -Exactly one of the two database modes must be enabled; the chart refuses to render otherwise. `postgres.credentials.*` feeds **both** modes — this template builds the database credentials secret from those three values and hands its name to whichever database is enabled, so there is nothing to create before installing. Change the password before you install; it is used exactly as written. +Exactly one of the two database modes must be enabled; the chart refuses to render otherwise. The database credentials come from the prerequisite secret named by the enabled mode's `credentialsSecretName` (see [Prerequisites](#prerequisites)). | | `postgresHA` (default) | `postgres` | |---|---|---| @@ -202,24 +213,20 @@ Exactly one of the two database modes must be enabled; the chart refuses to rend postgresHA: # set postgresHA.enabled: false and postgres.enabled: true for a lighter single-instance database enabled: true config: - credentialsSecretName: my-n8n-db-credentials # secret this chart CREATES from postgres.credentials.*; org-wide name — one per release + credentialsSecretName: my-n8n-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install replicas: 3 volumeset: capacity: 10 # initial capacity in GiB per replica (minimum is 10) postgres: enabled: false - credentials: # used in BOTH modes — this chart writes them into the credentials secret - username: n8n - password: change-me-n8n-db-password # change before installing - database: n8n config: - credentialsSecretName: my-n8n-db-credentials # secret this chart CREATES; org-wide name — one per release + credentialsSecretName: my-n8n-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) ``` -Secret names are org-wide, so if you run more than one n8n release in the same org, give each its own `credentialsSecretName`. A second release left on the default name is refused at install and creates nothing; the first release is unaffected. +Secret names are org-wide, so if you run more than one n8n release in the same org, give each its own database credentials secret; two releases pointed at one secret share a password. ### Database Backups @@ -298,8 +305,8 @@ cpln workload get RELEASE_NAME-n8n --gvc GVC_NAME -o yaml | Test webhooks | `https://CANONICAL_ENDPOINT/webhook-test/PATH` | Active only while listening in the editor | | Internal (same GVC) | `http://RELEASE_NAME-n8n.GVC_NAME.cpln.local:5678` | Subject to `internalAccess` | | Login | `email` and the password behind `passwordHash` in your owner secret | `cpln secret reveal OWNER_SECRET_NAME -o yaml` shows the hash, not the password | -| PostgreSQL (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | Credentials are `postgres.credentials.*`, stored in the secret named by `postgresHA.config.credentialsSecretName` | -| PostgreSQL (single mode) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Credentials are `postgres.credentials.*`, stored in the secret named by `postgres.config.credentialsSecretName` | +| PostgreSQL (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | Credentials from your `postgresHA.config.credentialsSecretName` secret | +| PostgreSQL (single mode) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Credentials from your `postgres.config.credentialsSecretName` secret | Verify the instance is serving before you sign in: @@ -355,7 +362,7 @@ cpln volumeset snapshot create RELEASE_NAME-n8n-vs --gvc GVC_NAME --snapshot-nam ### Restoring a Backup -Restoring the database follows the bundled database's own procedure, run from a client with access to the backup bucket: [postgres-highly-available Restoring a Backup](/template-catalog/templates/postgres-highly-available#restoring-a-backup) in HA mode (connect through `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local` so the restore targets the current leader), or [postgres Restoring a Backup](/template-catalog/templates/postgres#restoring-a-backup) in single mode (host `RELEASE_NAME-postgres.GVC_NAME.cpln.local`). The username and password are your `postgres.credentials.*` values. +Restoring the database follows the bundled database's own procedure, run from a client with access to the backup bucket: [postgres-highly-available Restoring a Backup](/template-catalog/templates/postgres-highly-available#restoring-a-backup) in HA mode (connect through `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local` so the restore targets the current leader), or [postgres Restoring a Backup](/template-catalog/templates/postgres#restoring-a-backup) in single mode (host `RELEASE_NAME-postgres.GVC_NAME.cpln.local`). The username and password are in your database credentials secret. Two n8n-specific points: @@ -422,13 +429,13 @@ from those values on every start, so hash the password you are using today to ke Remove `owner.email` and `owner.password` and set `owner.secretName`. Leave `encryptionKey.secretName`, `owner.firstName`, and `owner.lastName` as they are. - Then apply [Upgrading From 1.1.0](#upgrading-from-1-1-0) and [Upgrading From 1.2.0 or 1.3.0](#upgrading-from-1-2-0-or-1-3-0) as they apply to your database mode, and upgrade once. + Then apply [Upgrading From 1.1.0](#upgrading-from-1-1-0), [Upgrading From 1.2.0 or 1.3.0](#upgrading-from-1-2-0-or-1-3-0) and [Upgrading From 1.3.1](#upgrading-from-1-3-1) as they apply to your database mode, and upgrade once. ### Upgrading From 1.1.0 -Version `1.2.0` changed how the **single-instance** database (`postgres.enabled: true`) gets its credentials. `postgres.config.username`, `postgres.config.password`, and `postgres.config.database` were removed; the same three values now live under `postgres.credentials.*`, and this template creates the credentials secret named by the new `postgres.config.credentialsSecretName`. Carrying the old keys fails at render with the bundled database's own message: +Version `1.2.0` changed how the **single-instance** database (`postgres.enabled: true`) gets its credentials. `postgres.config.username`, `postgres.config.password`, and `postgres.config.database` were removed; the same three values now live in the database credentials secret under [Prerequisites](#prerequisites), named by `postgres.config.credentialsSecretName`. Carrying the old keys fails at render with the bundled database's own message: ```text config.username was REMOVED in postgres 3.4.0. Database credentials are no longer values: create a @@ -437,11 +444,11 @@ config.credentialsSecretName to its name. ... If this chart is BUNDLED inside an do NOT create a secret yourself — that template creates it for you ``` -Move the three values from `postgres.config.*` to `postgres.credentials.*`, keeping them **identical** to what the running database was initialised with, and ignore the message's advice to create a secret — this template creates it. `postgres.backup.minio.accessKey` and `secretKey` were removed in the same version: backups to MinIO now need the prerequisite dictionary secret described in [Backing Up](#backing-up), named by `postgres.backup.minio.credentialsSecretName`. Installs in HA mode have nothing to change in this step. +Put the three values from `postgres.config.*` into that secret, keeping them **identical** to what the running database was initialised with — the message's advice to create a secret is correct here. `postgres.backup.minio.accessKey` and `secretKey` were removed in the same version: backups to MinIO now need the prerequisite dictionary secret described in [Backing Up](#backing-up), named by `postgres.backup.minio.credentialsSecretName`. Installs in HA mode have nothing to change in this step. ### Upgrading From 1.2.0 or 1.3.0 -Version `1.3.0` moved the **highly available** database (`postgresHA.enabled: true`, the default) to the same credentials model, and `1.3.1` completed it for MinIO backups. `postgresHA.postgres.username`, `postgresHA.postgres.password`, and `postgresHA.postgres.database` no longer exist; the HA database now reads the credentials secret this template creates from `postgres.credentials.*`, named by `postgresHA.config.credentialsSecretName`. Carrying the old block fails at render: +Version `1.3.0` moved the **highly available** database (`postgresHA.enabled: true`, the default) to the same credentials model, and `1.3.1` completed it for MinIO backups. `postgresHA.postgres.username`, `postgresHA.postgres.password`, and `postgresHA.postgres.database` no longer exist; the HA database now reads the database credentials secret under [Prerequisites](#prerequisites), named by `postgresHA.config.credentialsSecretName`. Carrying the old block fails at render: ```text postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres.username`, @@ -450,15 +457,15 @@ postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres. ``` -**Use the credentials your cluster already has.** They were applied when the database was first initialised and are what PostgreSQL still enforces. Putting *new* values in `postgres.credentials.*` does not change the running database — it gives n8n a password that no longer works. +**Use the credentials your cluster already has.** They were applied when the database was first initialised and are what PostgreSQL still enforces. Putting *new* values in the credentials secret does not change the running database — it gives n8n a password that no longer works. - Delete `postgresHA.postgres.*` from your values and put the same three values under `postgres.credentials.username`, `postgres.credentials.password`, and `postgres.credentials.database`. The block sits under `postgres:` even though `postgres.enabled` stays `false` — it feeds both modes. Ignore the message's advice to create a secret yourself; this template creates it. + Delete `postgresHA.postgres.*` from your values and put the same three values into the database credentials secret as the keys `username`, `password` and `database`. - - `postgresHA.config.credentialsSecretName` defaults to `my-n8n-db-credentials`. Secret names are org-wide, so give each n8n release in the org its own. + + Set `postgresHA.config.credentialsSecretName` to the secret's name. Secret names are org-wide, so give each n8n release in the org its own. `postgresHA.backup.minio.accessKey` and `secretKey` were removed. Create the prerequisite dictionary secret described in [Backing Up](#backing-up) and set `postgresHA.backup.minio.credentialsSecretName` to its name. `1.3.0` still listed the two removed keys in its values while the bundled database already refused them, which is why MinIO backups in HA mode need `1.3.1`. @@ -468,7 +475,62 @@ postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres. -This upgrade path has not been exercised against a live n8n install; if your database password is the shipped default `change-me-n8n-db-password`, treat it as compromised and change it inside PostgreSQL before putting the new value in `postgres.credentials.password`. +If your database password is the shipped default `change-me-n8n-db-password`, treat it as compromised and change it after the upgrade — see [Rotating the Database Password](#rotating-the-database-password). + +### Upgrading From 1.3.1 + +Version `1.4.0` removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the bundled database template requires on its own. Carrying the old key forward fails the render. + +1.3.1 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or n8n loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-n8n-db \ + --entry username=n8n --entry password='YOUR_CURRENT_PASSWORD' --entry database=n8n +``` + +Then upgrade with the `credentialsSecretName` of your database mode (`postgresHA.config.` or `postgres.config.`) set to that name and `postgres.credentials` removed. Your current values are in `postgres.credentials` of the 1.3.1 release, and also in the old secret (`cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is the `credentialsSecretName` your release used, `my-n8n-db-credentials` by default) until the upgrade deletes it. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so n8n can briefly lose its database; it reconnects by itself. While access to the new secret propagates, the new version of a workload can wait in `get-deployments` reporting that it is not allowed to reveal the secret; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating the Database Password + +A running workload keeps the credential values it started with, and PostgreSQL reads the secret only when its data directory is first initialized, so editing the secret alone does not change the database password. Change one key and keep the others with export, edit, apply: + +```bash +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only the key you are rotating in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +``` + +**Single-instance mode:** run `ALTER ROLE` for the secret's `username` in the database first, then update the secret, then right away force a redeployment of `RELEASE_NAME-n8n`. Do not redeploy the database; restarting it only adds downtime. + +```bash +cpln workload force-redeployment RELEASE_NAME-n8n --gvc GVC_NAME +``` + +**HA mode** (the default) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Back to back: change BOTH roles on -0, put the same password in the secret (export, edit, apply above), restart the members +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 4. Once the members are back, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 5. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-n8n`, which keeps the old password until it restarts. The app cannot open new database connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` ### Scaling and Availability @@ -501,15 +563,9 @@ This upgrade path has not been exercised against a live n8n install; if your dat **Cause:** both database modes are enabled, or neither is. **Fix:** set exactly one of `postgresHA.enabled` and `postgres.enabled` to `true`. The HAProxy endpoint must stay on in HA mode (`postgresHA.proxy.enabled` is refused when `false`). - **Cause:** your values still carry a key an earlier version took — `owner.email` / `owner.password` (removed in `1.1.0`), `postgres.config.username` / `password` / `database` (removed in `1.2.0`), or `postgresHA.postgres.*` and `*.backup.minio.accessKey` / `secretKey` (removed in `1.3.0` / `1.3.1`). The render fails before anything is applied, and the running release is untouched. **Fix:** follow the matching section under [Operations](#operations), starting with [Upgrading From 1.0.x](#upgrading-from-1-0-x). + **Cause:** your values still carry a key an earlier version took — `owner.email` / `owner.password` (removed in `1.1.0`), `postgres.config.username` / `password` / `database` (removed in `1.2.0`), `postgresHA.postgres.*` and `*.backup.minio.accessKey` / `secretKey` (removed in `1.3.0` / `1.3.1`), or `postgres.credentials` (removed in `1.4.0`). The render fails before anything is applied, and the running release is untouched. **Fix:** follow the matching section under [Operations](#operations), starting with [Upgrading From 1.0.x](#upgrading-from-1-0-x). - - ```text - cannot be updated because it is being managed by a different release - ``` - **Cause:** two n8n releases in the same org share a `credentialsSecretName` — secret names are org-wide, and the default `my-n8n-db-credentials` is owned by the first release. Nothing is shared or overwritten and the first release is unaffected. **Fix:** give the new release its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName`). - **Cause:** the owner secret holds a different hash than the password you were using. n8n re-applies the owner from the secret on every start, so the secret's hash is the login. **Fix:** sign in with the password that matches the hash in the secret, or put the hash of the password you want there and force a redeployment — see [Rotating the Owner Password](#rotating-the-owner-password). @@ -527,10 +583,10 @@ This upgrade path has not been exercised against a live n8n install; if your dat ## Important Notes - **Back up the encryption key secret before installing** — it cannot be rotated; losing or changing it makes every stored credential undecryptable and n8n fails to start on a mismatch. -- **Create both prerequisite secrets before installing** — a missing one wedges the deployment with no log output; `cpln workload get-deployments RELEASE_NAME-n8n --gvc GVC_NAME -o yaml` is the one command that names it. +- **Create all three prerequisite secrets before installing** — a missing one wedges the deployment with no log output; `cpln workload get-deployments RELEASE_NAME-n8n --gvc GVC_NAME -o yaml` is the one command that names it. - **The owner secret is the login at every restart** — change the password through the secret plus a forced redeployment, never from inside n8n, and when upgrading hash the password you use today. -- **Change `postgres.credentials.password` before installing** — it is used exactly as written and feeds both database modes. -- **Run only one n8n release per `credentialsSecretName`** — secret names are org-wide; a second release on the same name is refused at install. +- **Editing the database credentials secret does not change the database password** — run `ALTER ROLE` first, then force-redeploy n8n; in HA mode move the leader first ([Rotating the Database Password](#rotating-the-database-password)). +- **Give each n8n release its own database credentials secret** — secret names are org-wide, so two releases pointed at one secret share a password. - **Synchronous webhook responses must finish within 30 seconds** — see [Connecting](#connecting). - **Every upgrade restarts the single n8n replica** — expect a brief editor and webhook outage; the first upgrade after an install can also restart the bundled database. - **Uninstall deletes the database and n8n volume sets** — all workflows, credentials, and execution data; enable backups if the data matters. diff --git a/template-catalog/templates/nats.mdx b/template-catalog/templates/nats.mdx index b38a8a52..c044a67a 100644 --- a/template-catalog/templates/nats.mdx +++ b/template-catalog/templates/nats.mdx @@ -1,7 +1,7 @@ --- title: NATS icon: /template-catalog/templates/icons/nats.png -description: Deploy NATS. One NATS cluster per location, joined into a cross-region super cluster over gateways, with optional JetStream persistence and a WebSocket listener on by default (internal-only until you allow CIDRs). Installs into an existing GVC whose locations you list; no secrets to create. +description: Deploy NATS. One NATS cluster per location, joined into a cross-region super cluster over gateways, with optional JetStream persistence and a WebSocket listener on by default (internal-only until you allow CIDRs). Installs into an existing GVC whose locations you list; optional auth from a secret. keywords: ['pubsub', 'event bus', 'lightweight broker', 'rabbitmq alternative', 'kafka alternative', 'streaming', 'request reply', 'cncf', 'message broker', 'key-value store', 'object store', 'nats-server', 'nats cli', 'wss', 'durable consumers'] --- @@ -22,10 +22,10 @@ This template deploys into an existing [GVC](/reference/gvc) you already have; i | Secret | `RELEASE_NAME-nats-secret` | The startup script that builds each server's `nats.conf` at boot from the location and replica it runs as | — | | Secret | `RELEASE_NAME-nats-extra-data` | Your `nats_extra_config`, appended verbatim to the generated configuration | `nats_extra_config` non-empty | | Identity | `RELEASE_NAME-nats-identity` | Bound to the workload so it can read its startup script and its own GVC | — | -| Policy | `RELEASE_NAME-nats-policy` | Grants the identity `reveal` on this release's secrets and nothing else | — | +| Policy | `RELEASE_NAME-nats-policy` | Grants the identity `reveal` on this release's secrets, and on your credentials secret when authentication is on, and nothing else | — | | Policy | `RELEASE_NAME-nats-gvc-policy` | Grants the identity `view` on exactly the one GVC the release is installed into, so each server can confirm at boot that the GVC really has every location in `locations` | — | -The template creates no credential secret: NATS here runs without authentication, and access is governed by the internal firewall (see [Internal Access](#internal-access)) and the public CIDR list (see [Public Access](#public-access)). +The template creates no credential secret. Authentication is off by default, and access is then governed by the internal firewall (see [Internal Access](#internal-access)) and the public CIDR list (see [Public Access](#public-access)). To require a username and password, name your own secret in `auth.credentialsSecretName` (see [Authentication](#authentication)). ## Prerequisites @@ -41,7 +41,26 @@ The template creates no credential secret: NATS here runs without authentication -Nothing else is needed: no secret to create, no cloud account, no bucket, no custom domain. +Nothing else is needed for a default install: no cloud account, no bucket, no custom domain. + +To require authentication, create a [dictionary secret](/guides/create-secret/dictionary) with the keys `username` and `password` **before** you install or upgrade, and set `auth.credentialsSecretName` to its name: + +```bash +cpln secret create-dictionary --name my-nats-credentials \ + --entry username=nats --entry password="$(openssl rand -hex 24)" +``` + +Store the plain password (any characters except a line break); a bcrypt hash is refused, because the servers also present the password to each other. + + +If the secret is missing, the deployment waits silently and `cpln logs` shows nothing. The reason is in `status.versions[].message`: + +```bash +cpln workload get-deployments RELEASE_NAME-nats --gvc GVC_NAME -o yaml +``` + +A secret without a `username` or `password` key waits the same way, and the same message names the missing key; an empty value stops each server at boot with a `[nats] FATAL` line. + ## Installation @@ -56,7 +75,7 @@ locations: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/nats \ - --version 3.0.0 --gvc GVC_NAME -f nats-locations.yaml + --version 3.1.0 --gvc GVC_NAME -f nats-locations.yaml ``` To install with another method, follow the instructions for your preferred tool: @@ -151,6 +170,15 @@ nats_defaults: The template refuses at render time to bind a port Control Plane reserves for system use, and to put two listeners on the same port. Both failures are otherwise invisible to `helm template`: a reserved port is rejected by the API at install, and a port collision is accepted and then stops the container at boot with `address already in use`. +### Authentication + +```yaml +auth: + credentialsSecretName: "" # dictionary secret with `username` and `password`; empty = no authentication +``` + +When set, every connection must present these credentials: clients on 4222, WebSocket clients, and the route and gateway connections between the servers themselves. The credentials reach the servers at container start and never appear in the Helm release. To turn authentication on for a running release or change the password, see [Rotating the Credentials](#rotating-the-credentials). + ### Public Access ```yaml @@ -162,7 +190,7 @@ allowCIDR: [] - `allowCIDR` — The inbound CIDR ranges allowed to reach the WebSocket endpoint from the internet. It applies to the WebSocket port only — the client (`4222`), route (`6222`) and gateway (`7222`) ports are never exposed publicly. -**This template configures no NATS authentication, so opening `allowCIDR` publishes an unauthenticated message bus.** An anonymous outside client can publish, subscribe, and read the cluster name and every server name. Version 3.0.0 changed the default from `0.0.0.0/0` to `[]` for this reason. Open it only alongside an `authorization { ... }` block in [`nats_extra_config`](#extra-nats-configuration), and prefer the narrowest ranges that cover your clients. +**Without `auth.credentialsSecretName`, opening `allowCIDR` publishes an unauthenticated message bus.** An anonymous outside client can publish, subscribe, and read the cluster name and every server name. Version 3.0.0 changed the default from `0.0.0.0/0` to `[]` for this reason. Open it only with [authentication](#authentication) on, and prefer the narrowest ranges that cover your clients. @@ -193,23 +221,22 @@ Streams default to a single replica, so stream data lives on one server. A strea ```yaml # Any additional valid NATS configuration — appended verbatim to the generated # server config at startup. A syntax error here is a container that will not -# start, not a render failure. +# start, not a render failure. It lands in the Helm release as plain text, so +# never put credentials here — use `auth.credentialsSecretName` instead. nats_extra_config: "" ``` -- `nats_extra_config` — Appended verbatim to the generated server configuration. Use it for anything the template does not expose — payload limits, accounts, authorization: +- `nats_extra_config` — Appended verbatim to the generated server configuration. Use it for anything the template does not expose, such as payload limits: ```yaml nats_extra_config: | max_payload: 8MB - authorization { - user: USERNAME - password: PASSWORD - } ``` +With `auth.credentialsSecretName` set, the chart refuses an `authorization` or `accounts` block here, since it would replace the generated credentials. + -**This block is injected verbatim and is not validated at render time.** A syntax error here is a container that will not start, which is easy to mistake for an infrastructure problem — read the server's own parse error in the workload log (see [Troubleshooting](#troubleshooting)). Note also that the template has no prerequisite-secret path for NATS credentials: anything you put here is stored in the Helm release values and in the `RELEASE_NAME-nats-extra-data` secret. +**This block is injected verbatim and is not validated at render time.** A syntax error here is a container that will not start, which is easy to mistake for an infrastructure problem — read the server's own parse error in the workload log (see [Troubleshooting](#troubleshooting)). Anything you put here is stored in the Helm release values and in the `RELEASE_NAME-nats-extra-data` secret, so never put credentials here; use [Authentication](#authentication). ### Internal Access @@ -244,7 +271,9 @@ internalAccess: | A specific server | `nats://replica-N.RELEASE_NAME-nats.LOCATION.GVC_NAME.cpln.local:4222` | `N` is `0` to `replicas - 1`; `LOCATION` is one of the names in `locations` | | Browser and WebSocket clients from the internet | `wss://` + the workload's canonical endpoint | Only once `allowCIDR` covers the client; Control Plane terminates TLS and serves it on port 443 | | Monitoring | `http://127.0.0.1:8222` from inside a server's container | `/healthz`, `/varz`, `/routez`, `/gatewayz`, `/jsz`; not published as a container port | -| Credentials | none | No authentication unless you add an `authorization { ... }` block in [`nats_extra_config`](#extra-nats-configuration) | +| Credentials | your credentials secret | With `auth.credentialsSecretName` set, the `username` and `password` from that secret on every address above; otherwise none | + +Clients pass the credentials as options (`nats --user USERNAME --password PASSWORD`, or the user/password options of your client library), or in the URL as `nats://USERNAME:PASSWORD@host:4222` with the password URL-encoded. A workload can read them from the same secret with `cpln://secret/SECRET_NAME.username` and `.password` env vars, given `reveal` on the secret. Always use the fully qualified hostname; the bare workload name is not reliably resolvable. Read the public hostname rather than assembling it — its shape varies between GVCs: @@ -274,6 +303,8 @@ nats --server nats://127.0.0.1:4222 sub demo.hello nats --server nats://127.0.0.1:4222 pub demo.hello 'hello from Control Plane' ``` +With authentication on, add `--user USERNAME --password PASSWORD` to each `nats` command. + ## Operations ### Backing Up @@ -356,6 +387,13 @@ Values that moved or were removed in 3.0.0. The chart names each one at render t | `volumeset.autoscaling` | Gone. `volumeset.capacity` is the only storage knob | | `locations[].replicas: 0` | Not allowed. Remove the location instead | +### Rotating the Credentials + +Both turning authentication on and changing the password take effect only when the servers restart. + +- **Turning authentication on for a running release** restarts every server and disconnects clients that send no credentials, so update your clients in the same window. If the servers then wait on the new secret, give it about 5 minutes, then run `cpln workload force-redeployment RELEASE_NAME-nats --gvc GVC_NAME`. +- **Rotating the password does not apply by itself.** After updating the secret, run `cpln workload force-redeployment RELEASE_NAME-nats --gvc GVC_NAME`; until then the old password keeps working. Clients need the new password once the servers restart. While the redeploy rolls, restarted and not-yet-restarted servers refuse each other's routes, so messages published on one side do not reach subscribers on the other until every server has restarted; rotate in a quiet window. + ### Scaling and Availability - **Fault tolerance follows the total server count.** Core NATS keeps routing with any number of servers up. JetStream keeps serving while a majority of servers is up: 3 survive losing one, 5 survive losing two, and 2 survive none, which is why that shape is refused (see [Locations and Cluster Sizing](#locations-and-cluster-sizing)). @@ -437,6 +475,11 @@ cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-nats"} |= "[nats]"' --limit 5 cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-nats"}' --limit 50 --since 10m ``` + + **Cause:** `auth.credentialsSecretName` is set, and the chart generates the authorization itself from that secret; a second block in `nats_extra_config` would replace it. + + **Fix:** Remove the `authorization` or `accounts` block from `nats_extra_config` and keep the credentials in the secret. + **Cause:** `ready: true` says nothing about clustering — this workload has no readiness probe. An empty `routes` list on a multi-server deployment means the servers cannot reach each other on the route port: `internalAccess` is `workload-list` with a list that was edited outside the chart, or a declared location is missing from the GVC so the peers never started. @@ -445,7 +488,7 @@ cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-nats"} |= "[nats]"' --limit 5 **Cause:** `allowCIDR` is empty (the default), or does not cover the client's current egress address — a client behind a proxy pool or a rotating address moves to an IP outside a narrow range. - **Fix:** Set `allowCIDR` to a range that covers your clients' egress, together with an `authorization { ... }` block in `nats_extra_config`, and upgrade the release. Allow a few minutes for the change to take effect. + **Fix:** Set `allowCIDR` to a range that covers your clients' egress, with [authentication](#authentication) on, and upgrade the release. Allow a few minutes for the change to take effect. @@ -454,11 +497,13 @@ cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-nats"} |= "[nats]"' --limit 5 - **Do not `cpln helm upgrade` a 2.x release onto this version.** It deletes the GVC the old release created and everything inside it. Follow [Migrating from 2.x](#migrating-from-2-x). - **`locations` must match the GVC's locations.** A declared location the GVC lacks is not caught at install; a fresh JetStream server refuses to start on it, and every other server warns. Read the GVC's list with `cpln gvc get GVC_NAME -o yaml` before installing. - **A GVC location you did not list runs nothing.** Its deployment reads `This workload location is deactivated because maxScale is set to 0`; that is expected. -- **No public access by default, and no authentication.** Opening `allowCIDR` without an `authorization { ... }` block in `nats_extra_config` publishes an unauthenticated message bus. Never pin a single `/32`. +- **No public access by default, and no authentication.** Opening `allowCIDR` without `auth.credentialsSecretName` publishes an unauthenticated message bus. Turn authentication on before adding ranges. Never pin a single `/32`. +- **Create the credentials secret before turning authentication on.** A missing secret waits silently; `status.versions[].message` from `cpln workload get-deployments` names it. +- **A rotated password applies only after `cpln workload force-redeployment`.** Cluster routes between restarted and not-yet-restarted servers are refused until every server has restarted; see [Rotating the Credentials](#rotating-the-credentials). - **`internalAccess` governs the servers' traffic to each other.** With `workload-list`, list only your clients — the template adds its own workload. `none` is refused for more than one server. - **JetStream data lives on the volume set and is deleted when you uninstall.** Streams default to one replica; create them with `--replicas=3` if a stream must survive a server loss. - **Do not run JetStream on exactly two servers.** The chart refuses it in values and warns if a missing GVC location produces it at runtime. -- **`nats_extra_config` is injected verbatim**, so a syntax error there stops the container rather than failing the render, and anything you put in it is stored in the release values. +- **`nats_extra_config` is injected verbatim**, so a syntax error there stops the container rather than failing the render, and anything you put in it is stored in the release values; never put credentials in it. - **A firewall change takes a few minutes to take effect.** After changing `allowCIDR` or `internalAccess`, keep re-testing rather than concluding the knob is broken. ## External References diff --git a/template-catalog/templates/nocodb.mdx b/template-catalog/templates/nocodb.mdx index 18c305e7..71013f6e 100644 --- a/template-catalog/templates/nocodb.mdx +++ b/template-catalog/templates/nocodb.mdx @@ -1,7 +1,7 @@ --- title: NocoDB icon: /template-catalog/templates/icons/nocodb.png -description: Deploy NocoDB, the open-source no-code database and Airtable alternative, on Control Plane. Single server with bundled PostgreSQL (single-instance by default, HA with one flag) and Redis, optional multi-replica on S3 attachment storage. Needs an existing GVC and a pre-created key secret. +description: Deploy NocoDB, the open-source no-code database and Airtable alternative, on Control Plane. Single server with bundled PostgreSQL (single-instance by default, HA with one flag) and Redis, optional multi-replica on S3 attachment storage. Needs an existing GVC and three pre-created secrets. keywords: ['nocodb', 'airtable alternative', 'baserow alternative', 'teable alternative', 'no-code database', 'smart spreadsheet', 'grid kanban gallery', 'form builder', 'rest api database', 'low code app builder', 'webhooks', 'fair-code'] --- @@ -29,20 +29,40 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Volume set | `RELEASE_NAME-nocodb-redis-vs` | Redis append-only file at `/data` | — | | Volume set | `RELEASE_NAME-pg-vs` | Single-instance PostgreSQL data directory; takes no scheduled snapshots; a final snapshot is taken when the volume set is deleted and kept 7 days | `postgres.enabled: true` (default) | | Volume set | `RELEASE_NAME-postgres-ha-vs`, `RELEASE_NAME-etcd-vs` | Per-replica Patroni and etcd data directories in HA mode | `postgresHA.enabled: true` | -| Secret | `RELEASE_NAME-nocodb-creds` | Dictionary secret the chart creates with the Redis password NocoDB uses to assemble `NC_REDIS_URL` | — | -| Secret | the name in `postgres.config.credentialsSecretName` | Dictionary secret the chart creates (`username`, `password`, `database`) from `postgres.credentials.*` and hands to whichever database is enabled | — | | Identity | `RELEASE_NAME-nocodb-identity` | Bound to the NocoDB and Redis workloads; carries your AWS cloud account link in keyless S3 mode | — | -| Policy | `RELEASE_NAME-nocodb-policy` | Grants that identity `reveal` on exactly the secrets it reads: the two chart-created secrets, your key secret, and your admin, S3-key, and SMTP secrets when configured | — | +| Policy | `RELEASE_NAME-nocodb-policy` | Grants that identity `reveal` on exactly the secrets it reads: your key, database credentials and Redis secrets, and your admin, S3-key, and SMTP secrets when configured | — | | Identity / Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | Bound to the single-instance PostgreSQL and its backup job; `reveal` on the credentials secret only | `postgres.enabled: true` (default) | | Identity / Policy | `RELEASE_NAME-postgres-ha-identity`, `RELEASE_NAME-postgres-ha-policy`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy` | The HA subchart's own identities and policies, plus its startup secrets (`RELEASE_NAME-patroni-startup`, `RELEASE_NAME-etcd-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-postgres-config`, `RELEASE_NAME-wal-g-backup-script`) | `postgresHA.enabled: true` | -Your key secret is **not** created by the template — see [Prerequisites](#prerequisites). +Your prerequisite secrets are **not** created by the template — see [Prerequisites](#prerequisites). ## Prerequisites -NocoDB signs its auth tokens and encrypts the stored credentials of external data sources with two keys that you supply through a [dictionary secret](/guides/create-secret/dictionary) created **before** installing. The values never pass through Helm values. +Three [dictionary secrets](/guides/create-secret/dictionary) must exist **before** installing: NocoDB's two signing and encryption keys, the bundled database's credentials, and the bundled Redis password. The values never pass through Helm values. + + The bundled PostgreSQL initializes with these and NocoDB connects with them: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=nocodb \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=nocodb + ``` + + Set its name in the key of the database mode you use: `postgres.config.credentialsSecretName` (single-instance, the default) or `postgresHA.config.credentialsSecretName` (HA). Use only letters, digits, `-` and `_` in all three values: NocoDB receives them inside its `NC_DB` connection URL, unencoded, so other characters break the connection. Secret names are org-wide, so give each NocoDB release its own secret. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password). + + + The bundled Redis requires this password and NocoDB connects with it: + + ```bash + cpln secret create-dictionary --name REDIS_SECRET_NAME \ + --entry password="$(openssl rand -hex 24)" + ``` + + Set `redis.credentialsSecretName` to this name. Use only letters, digits, `-` and `_`: NocoDB receives the password inside its `NC_REDIS_URL`, unencoded. Give each NocoDB release its own secret. To change it later, see [Rotating the Redis Password](#rotating-the-redis-password). + The secret holds two random values under exactly these keys: @@ -60,7 +80,7 @@ NocoDB signs its auth tokens and encrypts the stored credentials of external dat -Create the secret **before** installing. Installing without it reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message`: +Create the secrets **before** installing. Installing without one reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message` (for the Redis secret, also check `RELEASE_NAME-nocodb-redis`; for the database secret, the database workload): ```bash cpln workload get-deployments RELEASE_NAME-nocodb --gvc GVC_NAME -o yaml @@ -78,17 +98,17 @@ Everything else works with the defaults. Four optional features need their own s ## Installation -Once your key secret exists, install from the marketplace registry, pointing `secrets.name` at it and replacing both bundled-component passwords (letters, digits, `-`, and `_` only — both are embedded in connection URLs): +Once your three secrets exist, install from the marketplace registry, pointing the chart at them: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/nocodb \ - --version 1.3.0 --gvc GVC_NAME \ + --version 1.4.0 --gvc GVC_NAME \ --set secrets.name=KEYS_SECRET_NAME \ - --set postgres.credentials.password='YOUR-DB-PASSWORD' \ - --set redis.auth.password='YOUR-REDIS-PASSWORD' + --set postgres.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME \ + --set redis.credentialsSecretName=REDIS_SECRET_NAME ``` -Installing a second NocoDB release in the same org? Also set `postgres.config.credentialsSecretName` to a name no other release uses — secret names are org-wide. +In HA mode set `postgresHA.config.credentialsSecretName` instead of `postgres.config.credentialsSecretName`. Other ways to install and manage the template: @@ -234,8 +254,7 @@ Redis is required: it carries the job and event pub/sub, the metadata cache, and ```yaml redis: image: redis:8.10.0 - auth: - password: change-me-nocodb-redis # change before installing; letters/digits/-/_ only — embedded in NC_REDIS_URL + credentialsSecretName: my-nocodb-redis # PREREQUISITE dictionary secret (key: password; letters/digits/-/_ only) — must exist BEFORE install resources: minCpu: 100m maxCpu: 400m @@ -258,18 +277,14 @@ Enable exactly one of `postgres` (single instance, the default) and `postgresHA` The single instance is the default because on Control Plane the difference between the two modes is minutes of downtime, not data loss. The two paths run **different PostgreSQL majors**, so pick the mode before you have data — moving an existing database between them is a dump and restore, not a values change. -In both modes the chart builds the database credentials secret from `postgres.credentials.*` for you — nothing to create before installing. The database **name** stays a plain value because it is written into NocoDB's `NC_DB` connection URL at install time; the username and password are read from the secret. +In both modes the database credentials come from the prerequisite secret named by the enabled mode's `credentialsSecretName` (see [Prerequisites](#prerequisites)). ```yaml postgres: enabled: true image: postgres:18 # NocoDB supports PostgreSQL 12+ (the HA path below runs 17) - credentials: # the chart writes these into the secret named below — used by BOTH database modes - username: nocodb - password: change-me-nocodb-db # change before install; letters/digits/-/_ only (embedded in NC_DB) - database: nocodb config: - credentialsSecretName: my-nocodb-db-credentials # chart-created; secret names are org-wide, so give each release its own + credentialsSecretName: my-nocodb-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 250m maxCpu: 1000m @@ -287,7 +302,7 @@ postgres: postgresHA: enabled: false config: - credentialsSecretName: my-nocodb-db-credentials # the same chart-created secret, built from postgres.credentials.* above + credentialsSecretName: my-nocodb-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install replicas: 3 resources: # per Patroni replica; the default install runs 3 of these minCpu: 500m @@ -303,7 +318,7 @@ postgresHA: ``` -Both bundled passwords seed their component on first boot and are not updated by later value edits. Uninstalling the release deletes the volume sets; reinstall to reset. +PostgreSQL reads its password from the secret only when its data is first initialized; Redis reads its password each time it starts. To change either, follow [Rotating the Database Password](#rotating-the-database-password) or [Rotating the Redis Password](#rotating-the-redis-password). Uninstalling the release deletes the volume sets; reinstall to reset. ## Connecting @@ -315,8 +330,8 @@ Both bundled passwords seed their component on first boot and are not updated by | Internal (same GVC) | `http://RELEASE_NAME-nocodb.GVC_NAME.cpln.local:8080` | Reachable per `internalAccess.type`. | | Login | The first account created after install, or the `admin.secretName` credentials | NocoDB ships no default account. | | PostgreSQL (single instance) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | `username` / `password` / `database` from `cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is your `postgres.config.credentialsSecretName`. | -| PostgreSQL (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | Same credentials secret, named by `postgresHA.config.credentialsSecretName`. | -| Redis | `RELEASE_NAME-nocodb-redis.GVC_NAME.cpln.local:6379` | Password is the `redisPassword` key of `cpln secret reveal RELEASE_NAME-nocodb-creds -o yaml`. | +| PostgreSQL (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | Same keys, from the secret named by `postgresHA.config.credentialsSecretName`. | +| Redis | `RELEASE_NAME-nocodb-redis.GVC_NAME.cpln.local:6379` | Password is the `password` key of your `redis.credentialsSecretName` secret: `cpln secret reveal SECRET_NAME -o yaml`. | With `publicAccess.enabled: false` there is no public endpoint; reach the UI from your machine through a tunnel instead and open `http://localhost:8080`: @@ -522,15 +537,9 @@ This restore path has not been exercised against this template end to end. Rehea ### Upgrading from 1.0.x -Version 1.1.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. NocoDB absorbed that change rather than passing it on — the chart now creates the credentials secret itself — so there is **no new prerequisite**, only a rename of three keys: +Version 1.1.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. `postgres.config.username`, `.password` and `.database` were removed: put the values your single-instance database was initialized with into the database credentials secret under [Prerequisites](#prerequisites) (keys `username`, `password`, `database`), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. -| Removed key (1.0.x) | Replacement (1.1.0 and later) | -|---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | - -Move the three keys, keep the same values so the running database still matches, and upgrade in place. A values file that still carries an old key fails the render with the **postgres template's** message — `config.username was REMOVED in postgres 3.4.0` — which tells you to create a dictionary secret yourself; ignore that advice here, this template creates it. `postgres.backup.minio.accessKey` and `postgres.backup.minio.secretKey` were removed the same way: that path now reads a dictionary secret named by `postgres.backup.minio.credentialsSecretName` (see [Backing Up](#backing-up)). +Keep the same values so the running database still matches. A values file that still carries an old key fails the render with the **postgres template's** message — `config.username was REMOVED in postgres 3.4.0` — which tells you to create a dictionary secret; that is correct here too. `postgres.backup.minio.accessKey` and `postgres.backup.minio.secretKey` were removed the same way: that path now reads a dictionary secret named by `postgres.backup.minio.credentialsSecretName` (see [Backing Up](#backing-up)). Version 1.0.1 also moved the bundled `postgres-highly-available` template to 2.4.2, which turns on [etcd history compaction](/template-catalog/templates/postgres-highly-available#etcd-history-compaction). HA installs still on 1.0.0 should upgrade; compaction stops further growth but cannot shrink a backend that has already grown. @@ -538,21 +547,96 @@ If you run the HA database, continue with the section below as well. ### Upgrading from 1.1.x -Version 1.2.0 adopted `postgres-highly-available` 2.5.0, which no longer creates its own credentials secret. The HA database now reads the same chart-created secret as the single-instance path, built from `postgres.credentials.*`, and the `postgresHA.postgres.*` keys were removed: - -| Removed key (1.1.x) | Replacement (1.2.0 and later) | -|---|---| -| `postgresHA.postgres.username` | `postgres.credentials.username` | -| `postgresHA.postgres.password` | `postgres.credentials.password` | -| `postgresHA.postgres.database` | `postgres.credentials.database` | -| — | `postgresHA.config.credentialsSecretName` (name of the chart-created secret) | - -Set `postgres.credentials.*` to exactly the values your running Patroni cluster was initialised with — the database does not pick up a new password from a values change — and drop the old keys. Version 1.2.1 removed `postgresHA.backup.minio.accessKey` and `postgresHA.backup.minio.secretKey` the same way; that path now reads `postgresHA.backup.minio.credentialsSecretName`. An in-place upgrade across this boundary has not been exercised on a live HA install; rehearse it on a throwaway release first. +Version 1.2.0 adopted `postgres-highly-available` 2.5.0, which no longer creates its own credentials secret, and the `postgresHA.postgres.*` keys were removed. Put `postgresHA.postgres.username`, `.password` and `.database` **unchanged** into the database credentials secret under [Prerequisites](#prerequisites), set `postgresHA.config.credentialsSecretName` to its name, and delete the `postgresHA.postgres` block — the running Patroni cluster was initialized with those values, and the database does not pick up a new password from a secret change. Version 1.2.1 removed `postgresHA.backup.minio.accessKey` and `postgresHA.backup.minio.secretKey` the same way; that path now reads `postgresHA.backup.minio.credentialsSecretName`. The first `helm upgrade` after an install can restart the bundled Redis even when nothing about Redis changed, and NocoDB exits and restarts rather than reconnecting — at a single replica that is a brief outage on a routine config change; a second replica absorbs it. Later upgrades of the same release do not restart Redis. +### Upgrading from 1.3.x + +Version 1.4.0 removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the database templates require on their own. `redis.auth.password` was removed the same way: the Redis password is now a prerequisite secret named by `redis.credentialsSecretName`. Carrying either old key forward fails the render. + +1.3.x created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or NocoDB loses its database connection: + +```bash +cpln secret reveal SECRET_NAME -o yaml # SECRET_NAME = your 1.3.x credentialsSecretName, my-nocodb-db-credentials by default +cpln secret create-dictionary --name RELEASE_NAME-nocodb-db \ + --entry username='YOUR_CURRENT_USERNAME' --entry password='YOUR_CURRENT_PASSWORD' --entry database='YOUR_CURRENT_DATABASE' +``` + +1.3.x also kept the Redis password in a secret it created, `RELEASE_NAME-nocodb-creds` (key `redisPassword`), which this upgrade deletes too. Create the Redis secret under a different name, holding the **current** Redis password: + +```bash +cpln secret reveal RELEASE_NAME-nocodb-creds -o yaml # redisPassword is the CURRENT Redis password +cpln secret create-dictionary --name RELEASE_NAME-nocodb-redis \ + --entry password='YOUR_CURRENT_REDIS_PASSWORD' +``` + +Then upgrade with `postgres.config.credentialsSecretName` (single-instance mode) or `postgresHA.config.credentialsSecretName` (HA mode) set to the database secret's name, `redis.credentialsSecretName` set to the Redis secret's name, and `postgres.credentials` and `redis.auth` removed. Your current values are also in `postgres.credentials` and `redis.auth.password` of the 1.3.x release. + +The upgrade restarts the bundled PostgreSQL and Redis too (they now read the new secrets), so NocoDB can briefly lose its database and Redis; it recovers by itself. While access to the new secrets propagates, the new version of a workload can wait in `get-deployments` reporting that it is not allowed to reveal the secret; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating the Database Password + +PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with, so editing the secret alone does not change the database password. Change one key and keep the others with export, edit, apply (letters, digits, `-` and `_` only): + +```bash +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only the key you are rotating in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +``` + +**Single-instance mode** takes three steps in order: + +1. Change it in the database: `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` (connected as that user, from inside the GVC). +2. Update the secret as above. +3. Right away, redeploy NocoDB, and wait until `get-deployments` shows a higher version number, alone and ready (right after the command the status may not show the restart yet, and `ready: true` alone can still describe the old replica). Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. + +```bash +cpln workload force-redeployment RELEASE_NAME-nocodb --gvc GVC_NAME +``` + +Connections the app already has open keep working after `ALTER ROLE`; new ones fail until its redeploy finishes, so redeploy right away and rotate in a quiet window. + +**HA mode** needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Back to back: change BOTH roles on -0, put the same password in the secret (export, edit, apply above), restart the members +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 4. Once the members are back, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 5. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-nocodb`, which keeps the old password until it restarts. The app cannot open new database connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + +### Rotating the Redis Password + +Redis keeps no password on disk: it takes the password from the secret each time it starts, and NocoDB does the same. So rotation is the secret and then both restarts: + +1. Update the secret with export, edit, apply as above (letters, digits, `-` and `_` only), and check the new value with `cpln secret reveal SECRET_NAME -o yaml`. +2. Redeploy Redis and NocoDB together. Neither picks up the change without its redeploy: + +```bash +cpln workload force-redeployment RELEASE_NAME-nocodb-redis --gvc GVC_NAME +cpln workload force-redeployment RELEASE_NAME-nocodb --gvc GVC_NAME +``` + +Expect a short interruption: until both have restarted, one side holds the old password and NocoDB cannot reach Redis. + ### Scaling and Availability `nocodb.replicas` sets how many NocoDB replicas run. Replicas coordinate through the bundled Redis, which carries the job and event pub/sub, the shared metadata cache, and the rate limiter — all three are what make a second replica correct rather than merely present. Rolling upgrades proceed one replica at a time. @@ -585,7 +669,7 @@ Live updates reach the browser over HTTP long-poll (`POST /jobs/listen`) backed **Symptom:** `cpln helm install` reported success, `cpln logs` returns nothing, and `cpln workload get-deployments RELEASE_NAME-nocodb --gvc GVC_NAME -o yaml` shows `The secret KEYS_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` under `status.versions[].message`. - **Cause:** The key secret named by `secrets.name` does not exist. The same applies to an `admin.secretName`, `storage.s3.auth.secretName`, or `smtp.auth.secretName` that was set but never created. + **Cause:** The key secret named by `secrets.name`, the database credentials secret or the Redis secret (`redis.credentialsSecretName`) does not exist. The same applies to an `admin.secretName`, `storage.s3.auth.secretName`, or `smtp.auth.secretName` that was set but never created. Check `RELEASE_NAME-nocodb-redis` and the database workload the same way. **Fix:** Create it (see [Prerequisites](#prerequisites)). The deployment recovers on its own once the secret exists; `cpln workload force-redeployment RELEASE_NAME-nocodb --gvc GVC_NAME` skips the wait. @@ -594,7 +678,7 @@ Live updates reach the browser over HTTP long-poll (`POST /jobs/listen`) backed **Cause:** The replica started before PostgreSQL or Redis was accepting connections. It restarts automatically and runs its migrations on the retry. - **Fix:** None needed. One such restart on a first install is expected; a replica that keeps restarting is a different problem — check that both bundled passwords use only letters, digits, `-`, and `_`. + **Fix:** None needed. One such restart on a first install is expected; a replica that keeps restarting is a different problem — check that the database and Redis secret values use only letters, digits, `-`, and `_`. **Symptom:** A routine `helm upgrade` that changed nothing about Redis makes the endpoint return `503` briefly, and the NocoDB logs show an ioredis error followed by an exit. @@ -603,13 +687,7 @@ Live updates reach the browser over HTTP long-poll (`POST /jobs/listen`) backed **Fix:** Nothing to repair. Run `nocodb.replicas: 2` (with S3 storage) if a single-replica interruption on the first upgrade is unacceptable; later upgrades of the same release do not restart Redis. - - **Symptom:** Installing a second NocoDB release fails with `cannot be updated because it is being managed by a different release` and creates nothing. - - **Cause:** Both releases use the same `postgres.config.credentialsSecretName`. Secret names are org-wide, and the first release owns that one. Nothing is shared, overwritten, or deleted. - **Fix:** Set `postgres.config.credentialsSecretName` (and `postgresHA.config.credentialsSecretName` in HA mode) to a distinct name for the second release and install again. - **Symptom:** `nocodb: nocodb.replicas > 1 requires storage.type: s3 — local attachments live on a per-replica volumeset and would 404 across replicas`. @@ -636,7 +714,7 @@ Live updates reach the browser over HTTP long-poll (`POST /jobs/listen`) backed **Cause:** Your values file still carries `postgres.config.username`, `postgres.config.password`, or `postgres.config.database`. - **Fix:** Rename them to `postgres.credentials.*` as described in [Upgrading from 1.0.x](#upgrading-from-1-0-x). Do not create a secret yourself — this template creates it. + **Fix:** Move their values into the database credentials secret as described in [Upgrading from 1.0.x](#upgrading-from-1-0-x). If your values carry `postgres.credentials` or `redis.auth.password`, follow [Upgrading from 1.3.x](#upgrading-from-1-3-x). **Symptom:** A webhook aimed at an internal address such as `http://WORKLOAD_NAME.GVC_NAME.cpln.local` never fires, and the hook log says to set `NC_ALLOW_LOCAL_HOOKS=true`. @@ -670,7 +748,7 @@ Live updates reach the browser over HTTP long-poll (`POST /jobs/listen`) backed ## Important Notes -- **Create the key secret before installing** — `secrets.name` must name an existing dictionary secret holding `NC_AUTH_JWT_SECRET` and `NC_CONNECTION_ENCRYPT_KEY`. A missing secret wedges the deployment silently. +- **Create the key, database credentials and Redis secrets before installing** — `secrets.name` must name an existing dictionary secret holding `NC_AUTH_JWT_SECRET` and `NC_CONNECTION_ENCRYPT_KEY`. A missing secret wedges the deployment silently; `status.versions[].message` in `cpln workload get-deployments RELEASE_NAME-nocodb --gvc GVC_NAME -o yaml` names it. - **Both keys are write-once** — rotating `NC_AUTH_JWT_SECRET` logs out every user, and changing `NC_CONNECTION_ENCRYPT_KEY` makes the stored credentials of external data sources undecryptable. Back both values up outside Control Plane. - **Sign up immediately after install** — signup is open by default, so the first visitor to sign up becomes the super admin. Then turn on invite-only in **Team & Settings**. - **With SMTP off, invitations and password resets cannot be delivered** — configure `smtp.*` before inviting collaborators. @@ -679,12 +757,12 @@ Live updates reach the browser over HTTP long-poll (`POST /jobs/listen`) backed - **`storage.fileUploadSizeLimit` is a byte count** — the default `20971520` is 20 MiB; over-limit uploads are rejected with `413`, not truncated. - **Webhooks to same-GVC targets are blocked by default** — set `nocodb.allowLocalWebhooks: true` to fire webhooks at other workloads in your GVC. - **`admin.secretName` re-applies on every boot** — a password changed in the UI reverts at the next restart. Leave it empty or treat the secret as the source of truth. -- **Change `postgres.credentials.password` and `redis.auth.password` before installing** — both seed their component on first boot, are not updated by later value edits, and must use only letters, digits, `-`, and `_`. +- **Database and Redis credentials use only letters, digits, `-`, and `_`** — they are embedded in connection URLs. Rotating the database password needs `ALTER ROLE` plus a forced redeployment of NocoDB; rotating the Redis password needs a forced redeployment of both Redis and NocoDB — see [Rotating the Database Password](#rotating-the-database-password) and [Rotating the Redis Password](#rotating-the-redis-password). - **Enable exactly one database** — `postgres` (PostgreSQL 18, default) or `postgresHA` (PostgreSQL 17). Choose before you have data; switching is a dump and restore. -- **Give each release its own `postgres.config.credentialsSecretName`** — secret names are org-wide, and a second release on the same name is refused at install. +- **Give each release its own database and Redis secrets** — secret names are org-wide, so two releases pointed at one secret share a password. - **Set `nocodb.siteUrl` when NocoDB sits behind a custom domain**, with the scheme — it drives invitation and password-reset links and the auth cookie's secure flag. - **Background jobs are in-process** — a long import, export, or base duplication does not survive the replica running it and must be re-run. -- **Uninstalling deletes the volume sets** and every base and local attachment in them; your key secret is yours and survives. +- **Uninstalling deletes the volume sets** and every base and local attachment in them; your prerequisite secrets are yours and survive. - **MCP endpoints are created per base in the UI** (base → **Settings → MCP Server**) and authenticate with an `xc-mcp-token`; account-level MCP and OAuth are Enterprise features and are not available on this edition. - **SSO/SAML/OIDC, audit logs, and row-level security are Enterprise features** — they ship in the same image but need a purchased licence key that this template never sets. - **NocoDB Community Edition is fair-code** — the Sustainable Use License permits self-hosting for your own use; offering NocoDB as a hosted service to others requires a commercial licence. diff --git a/template-catalog/templates/polaris.mdx b/template-catalog/templates/polaris.mdx index 15684cdf..dad862b9 100644 --- a/template-catalog/templates/polaris.mdx +++ b/template-catalog/templates/polaris.mdx @@ -1,7 +1,7 @@ --- title: Polaris icon: /template-catalog/templates/icons/polaris.png -description: Deploy Apache Polaris, an Apache Iceberg REST catalog, on Control Plane. A stateless, horizontally scalable server with a bundled PostgreSQL metastore (single-instance or HA). Needs an existing GVC plus two prerequisite secrets, root client credentials and a token signing key. +description: Deploy Apache Polaris, an Apache Iceberg REST catalog, on Control Plane. A stateless, horizontally scalable server with a bundled PostgreSQL metastore (single-instance or HA). Needs an existing GVC plus three prerequisite secrets: root client credentials, a token signing key, metastore credentials. keywords: ['lakehouse', 'data lakehouse', 'data lake', 'iceberg rest catalog', 'hive metastore alternative', 'aws glue alternative', 'unity catalog alternative', 'nessie alternative', 'table format', 'parquet', 'trino iceberg', 'spark iceberg', 'pyiceberg', 'oauth2 client credentials', 'realm', 'polaris template'] --- @@ -24,7 +24,6 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Secret | `RELEASE_NAME-polaris-bootstrap-script` | Opaque secret holding the bootstrap script, mounted as a file | — | | Identity & Policy | `RELEASE_NAME-polaris-identity`, `RELEASE_NAME-polaris-policy` | Server identity with `reveal` on the metastore credentials, the token signing key and (when set) the object-storage credentials | — | | Identity & Policy | `RELEASE_NAME-polaris-bootstrap-identity`, `RELEASE_NAME-polaris-bootstrap-policy` | Bootstrap identity with `reveal` on the metastore credentials, the root credentials and the bootstrap script. Only this identity can read the root credentials | — | -| Secret | `my-polaris-db-credentials` | Dictionary secret **created by this chart** from `postgres.credentials.*`, holding `username`, `password` and `database`. Its name is the active metastore block's `config.credentialsSecretName` | — | | Workload | `RELEASE_NAME-postgres` | Single-instance PostgreSQL metastore (stateful, 1 replica) | `postgres.enabled: true` (default) | | Volume Set | `RELEASE_NAME-pg-vs` | 10 GiB `ext4` volume for the metastore | `postgres.enabled: true` (default) | | Identity & Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | PostgreSQL subchart identity with `reveal` on the credentials secret | `postgres.enabled: true` (default) | @@ -42,7 +41,7 @@ In HA mode with `postgresHA.backup.mode: wal-g`, no backup cron workload is crea ## Prerequisites -**Two secrets must exist before you install.** They are referenced by name and never pass through Helm values. Secrets are org-level, so no GVC flag is involved. +**Three secrets must exist before you install.** They are referenced by name and never pass through Helm values. Secrets are org-level, so no GVC flag is involved. @@ -65,10 +64,22 @@ In HA mode with `postgresHA.backup.mode: wal-g`, no backup cron workload is crea Set `tokenSigningKey.secretName` to this name. + + A [dictionary secret](/guides/create-secret/dictionary) holding `username`, `password` and `database`. The bundled PostgreSQL initializes with it, and the Polaris server and bootstrap workload connect with it: + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=polaris \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=polaris + ``` + + Set its name in the key of the store you use: `postgres.config.credentialsSecretName` (single-instance, the default) or `postgresHA.config.credentialsSecretName` (HA). **`database` must be URL-safe** (letters, digits, `_`): the JDBC URL is assembled from it at container start. Secret names are org-wide, so give each Polaris release its own secret. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Metastore Password](#rotating-the-metastore-password). + -**Create both secrets before installing, or the deployment wedges silently.** A name that points at a secret which does not exist installs "successfully" and then never starts: the container never runs, so `cpln logs` returns **zero lines**. The one place the reason appears is `status.versions[].message` — check `RELEASE_NAME-polaris` for the signing key and `RELEASE_NAME-polaris-bootstrap` for the root credentials: +**Create all three secrets before installing, or the deployment wedges silently.** A name that points at a secret which does not exist installs "successfully" and then never starts: the container never runs, so `cpln logs` returns **zero lines**. The one place the reason appears is `status.versions[].message` — check `RELEASE_NAME-polaris` for the signing key, `RELEASE_NAME-polaris-bootstrap` for the root credentials, and `RELEASE_NAME-postgres` (or `RELEASE_NAME-postgres-ha`) for the metastore credentials: ```bash cpln workload get-deployments RELEASE_NAME-polaris --gvc GVC_NAME -o yaml @@ -81,8 +92,6 @@ cpln workload force-redeployment RELEASE_NAME-polaris --gvc GVC_NAME ``` -The metastore password is **not** a prerequisite: this chart creates the metastore credentials secret from `postgres.credentials.*`. Change the password before installing. - Optional, and not needed for the install itself: - **Object-storage credentials** for the bucket that holds your Iceberg tables. Polaris runs without them and catalogs can be added later — see [Building a Lakehouse](#building-a-lakehouse). @@ -90,16 +99,18 @@ Optional, and not needed for the install itself: ## Installation -Install with your two secret names and a strong metastore password: +Install with your three secret names: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/polaris \ - --version 1.2.1 --gvc GVC_NAME \ + --version 1.3.0 --gvc GVC_NAME \ --set rootCredentials.secretName=ROOT_CREDENTIALS_SECRET_NAME \ --set tokenSigningKey.secretName=SIGNING_KEY_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-PASSWORD' + --set postgres.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME ``` +In HA mode set `postgresHA.config.credentialsSecretName` instead. + Browse, install, and manage templates visually @@ -239,12 +250,8 @@ The default metastore. Exactly one of `postgres` and `postgresHA` must be enable ```yaml postgres: enabled: true - credentials: - username: polaris - password: change-me-polaris-db # change before installing - database: polaris config: - credentialsSecretName: my-polaris-db-credentials + credentialsSecretName: my-polaris-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 200m maxCpu: 500m @@ -278,12 +285,11 @@ postgres: prefix: postgres/backups ``` -- `postgres.credentials.*` — The metastore's username, password and database name. **Change the password before installing.** The chart writes these three values into the credentials secret in **both** metastore modes, even though they sit under the `postgres` block, and Polaris connects to the database named here. -- `postgres.config.credentialsSecretName` — Name of the dictionary secret this chart creates in single-instance mode. Secret names are organization-wide, so give each Polaris release its own: a second release left on the default name is refused at install and creates nothing, and the first release is unaffected. +- `postgres.config.credentialsSecretName` — Name of the prerequisite metastore credentials secret in single-instance mode (see [Prerequisites](#prerequisites)). Polaris connects to the database named by its `database` key. Secret names are organization-wide, so give each Polaris release its own. - `postgres.backup.*` — Optional scheduled backups. See [Backing Up](#backing-up). -The credentials are applied when the metastore volume is first initialized. Changing `postgres.credentials.password` on an existing release updates the secret but not the password PostgreSQL enforces, so Polaris can no longer authenticate. To rotate, change the password inside PostgreSQL first, then upgrade the release with the matching value and force a redeployment of `RELEASE_NAME-polaris` and `RELEASE_NAME-polaris-bootstrap` so they pick up the new value. +The credentials are applied when the metastore volume is first initialized. Changing the secret on an existing release does not change the password PostgreSQL enforces, so Polaris can no longer authenticate. To rotate, follow [Rotating the Metastore Password](#rotating-the-metastore-password). ### Highly Available Metastore @@ -292,7 +298,7 @@ The credentials are applied when the metastore volume is first initialized. Chan postgresHA: enabled: false config: - credentialsSecretName: my-polaris-db-credentials + credentialsSecretName: my-polaris-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install replicas: 3 volumeset: capacity: 10 # initial capacity in GiB per replica (minimum is 10) @@ -328,7 +334,7 @@ postgresHA: Set `postgresHA.enabled: true` and `postgres.enabled: false` for near-zero-downtime upgrades and automatic database failover: 3 Patroni PostgreSQL replicas, 3 etcd replicas and an HAProxy leader endpoint that Polaris connects through. -- `postgresHA.config.credentialsSecretName` — Name of the dictionary secret this chart creates in HA mode, built from `postgres.credentials.*` above. Give each release its own name, as in single-instance mode. +- `postgresHA.config.credentialsSecretName` — Name of the prerequisite metastore credentials secret in HA mode, the same shape as in single-instance mode. Give each release its own. - `postgresHA.replicas` / `postgresHA.volumeset.capacity` — PostgreSQL replica count and volume size per replica. The remaining cluster settings take the defaults of the [PostgreSQL Highly Available](/template-catalog/templates/postgres-highly-available) template; its HAProxy endpoint must stay enabled, because it is Polaris's database address. - `postgresHA.backup.*` — Optional backups as scheduled logical dumps or continuous WAL-G archiving. See [Backing Up](#backing-up). @@ -351,7 +357,7 @@ Switching an existing release between modes is a new, empty metastore, not a mig | OAuth2 token endpoint | `BASE_URL/api/catalog/v1/oauth/tokens` | Client-credentials grant; every API call needs the resulting bearer token. | | Health and metrics | `http://RELEASE_NAME-polaris.GVC_NAME.cpln.local:8182/q/health` and `/q/metrics` | Inside the GVC only; never on the public endpoint. | | Client credentials | `cpln secret reveal ROOT_CREDENTIALS_SECRET_NAME -o yaml` | `CLIENT_ID` and `CLIENT_SECRET`. | -| Metastore | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432`, or `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` in HA mode | Credentials: `cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml`, where the name is your `config.credentialsSecretName` (default `my-polaris-db-credentials`). | +| Metastore | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432`, or `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` in HA mode | Credentials: `cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml`, your metastore credentials secret named by `config.credentialsSecretName`. | To get a token, request one with the root credentials. From your own machine, open a tunnel to the server first: @@ -607,17 +613,14 @@ The metastore backups are produced by the bundled database templates, so restore Apply every subsection from your version onward, in order, before running `cpln helm upgrade`. A removed key is rejected at render, so an upgrade that still carries one fails before anything is applied and the running release is left untouched. -Version `1.1.0` moved the single-instance metastore to the `postgres` 3.4.1 template, which no longer takes credentials or MinIO keys as values. This template creates the credentials secret itself, so there is **no new prerequisite**, only renames on the single-instance path: +Version `1.1.0` moved the single-instance metastore to the `postgres` 3.4.1 template, which no longer takes credentials or MinIO keys as values: | Removed key | Replacement | |---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | -| — | `postgres.config.credentialsSecretName` names the secret this chart creates (default `my-polaris-db-credentials`) | +| `postgres.config.username` / `.password` / `.database` | the metastore credentials secret under [Prerequisites](#prerequisites), named by `postgres.config.credentialsSecretName` | | `postgres.backup.minio.accessKey`, `postgres.backup.minio.secretKey` | `postgres.backup.minio.credentialsSecretName`, a dictionary secret you create (MinIO backups only, see [Backing Up](#backing-up)) | -Carrying an old key forward fails with the bundled database template's message, which begins `config.username was REMOVED in postgres 3.4.0` and tells you to create a secret. Ignore that advice for the three credentials keys: this template creates that secret from `postgres.credentials.*`. Use the username and password your metastore was initialized with, not new ones. +Carrying an old key forward fails with the bundled database template's message, which begins `config.username was REMOVED in postgres 3.4.0` and tells you to create a secret — that is correct here too. Use the username, password and database your metastore was initialized with, not new ones. **Template version `1.0.0` did not compact the etcd cluster inside the HA metastore**, so its backend grew with time alone and eventually went read-only, taking PostgreSQL failover with it. Only HA-mode installs are affected. Upgrading to `1.0.1` or later turns compaction on, which stops further growth but cannot shrink a backend that has already grown; a cluster that has already raised a `NOSPACE` alarm needs operator recovery rather than an upgrade. See [etcd History Compaction](/template-catalog/templates/postgres-highly-available#etcd-history-compaction). @@ -625,24 +628,72 @@ Carrying an old key forward fails with the bundled database template's message, ### Upgrading from 1.1.0 -Version `1.2.0` moved the HA metastore to `postgres-highly-available` 2.5.0, which removed the `postgresHA.postgres` block. The HA cluster now reads the same chart-created credentials secret as the single-instance path: - -| Removed key | Replacement | -|---|---| -| `postgresHA.postgres.username` | `postgres.credentials.username` | -| `postgresHA.postgres.password` | `postgres.credentials.password` | -| `postgresHA.postgres.database` | `postgres.credentials.database` | -| — | `postgresHA.config.credentialsSecretName` names the secret this chart creates (default `my-polaris-db-credentials`) | +Version `1.2.0` moved the HA metastore to `postgres-highly-available` 2.5.0, which removed the `postgresHA.postgres` block. The HA cluster now reads the metastore credentials secret under [Prerequisites](#prerequisites), named by `postgresHA.config.credentialsSecretName`. -**Use the credentials your cluster already has.** The HA cluster was initialized with the `postgresHA.postgres.*` values you installed with, and PostgreSQL still enforces them. Copy that username, password and database name into `postgres.credentials.*` before upgrading. New values there update the secret but not the database, and Polaris can no longer authenticate. +**Use the credentials your cluster already has.** The HA cluster was initialized with the `postgresHA.postgres.*` values you installed with, and PostgreSQL still enforces them. Put that username, password and database name into the metastore credentials secret before upgrading. New values there do not change the database, and Polaris can no longer authenticate. -An upgrade that still carries the `postgresHA.postgres` block fails with the HA template's message, which begins ``postgres-highly-available: the `postgres` block was REMOVED in 2.5.0`` and tells you to create a dictionary secret. Ignore that part: delete the block and set `postgres.credentials.*` instead. +An upgrade that still carries the `postgresHA.postgres` block fails with the HA template's message, which begins ``postgres-highly-available: the `postgres` block was REMOVED in 2.5.0`` and tells you to create a dictionary secret — that is correct here: delete the block and set `postgresHA.config.credentialsSecretName` to the secret's name. ### Upgrading from 1.2.0 -HA MinIO backups only: version `1.2.1` replaced `postgresHA.backup.minio.accessKey` and `postgresHA.backup.minio.secretKey` with `postgresHA.backup.minio.credentialsSecretName`, a dictionary secret you create holding `accessKey` and `secretKey` — see [Backing Up](#backing-up). An upgrade that still carries the old keys fails with `backup.minio.accessKey and backup.minio.secretKey were REMOVED in 2.5.0`. Create the secret before upgrading, because in `wal-g` mode the database workload itself references it. +HA MinIO backups only: version `1.2.1` replaced `postgresHA.backup.minio.accessKey` and `postgresHA.backup.minio.secretKey` with `postgresHA.backup.minio.credentialsSecretName`, a dictionary secret you create holding `accessKey` and `secretKey` — see [Backing Up](#backing-up). An upgrade that still carries the old keys fails with `backup.minio.accessKey and backup.minio.secretKey were REMOVED in 2.5.0`. Create the secret before upgrading, because in `wal-g` mode the database workload itself references it. Then follow [Upgrading from 1.2.1](#upgrading-from-1-2-1). + +### Upgrading from 1.2.1 + +Version `1.3.0` removed `postgres.credentials`: the metastore credentials are now a prerequisite secret, the same one the database templates require on their own. Carrying the old key forward fails the render. + +1.2.1 created the metastore secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Polaris loses its metastore connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-postgres-credentials \ + --entry username=polaris --entry password='YOUR_CURRENT_PASSWORD' --entry database=polaris +``` + +Your current values are in `postgres.credentials` of the 1.2.1 release, and also in the old secret (`cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is the `credentialsSecretName` your release used, `my-polaris-db-credentials` by default) until the upgrade deletes it. Then upgrade with `postgres.config.credentialsSecretName` (single-instance mode) or `postgresHA.config.credentialsSecretName` (HA mode) set to the new name, and `postgres.credentials` removed. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Polaris briefly loses its database; it reconnects by itself. While access to the new secret propagates, the new version of a workload can wait in `get-deployments` reporting that it is not allowed to reveal the secret; the previous version keeps serving meanwhile, and it clears by itself. When that wait happens, the PostgreSQL restart (and its brief outage) comes only after it clears. + +### Rotating the Metastore Password + +A running workload keeps the credential values it started with, and PostgreSQL reads the secret only when its data is first initialized, so editing the secret alone does not change the database password. Change one key and keep the others with export, edit, apply: + +```bash +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only the key you are rotating in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +``` + +**Single-instance mode:** run `ALTER ROLE` for the secret's `username` in the database first, then update the secret, then right away force a redeployment of the Polaris server, and wait until `get-deployments` shows a higher version number, alone and ready (right after the command the status may not show the restart yet, and `ready: true` alone can still describe the old replica). Do not redeploy the database: restarting it only adds downtime. Connections already open keep working after `ALTER ROLE`; new ones fail until the redeploy finishes, so rotate in a quiet window. + +```bash +cpln workload force-redeployment RELEASE_NAME-polaris --gvc GVC_NAME +``` + +**HA mode** needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Back to back: change BOTH roles on -0, put the same password in the secret (export, edit, apply above), restart the members +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 4. Once the members are back, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 5. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-polaris`, which keeps the old password until it restarts. The app cannot open new database connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` ### Scaling and Availability @@ -651,7 +702,7 @@ HA MinIO backups only: version `1.2.1` replaced `postgresHA.backup.minio.accessK - **The bootstrap workload always runs one replica** and does nothing after its first run. Do not scale it. - **The metastore is the other half of availability.** With the default single-instance `postgres`, a database restart takes Polaris down until PostgreSQL is back. `postgresHA` adds automatic failover behind a stable endpoint, at the cost of a larger footprint. - **The first upgrade after an install can restart the bundled database** even when no database value changed. Expect a short outage on a single-instance metastore. -- **Uninstall deletes the metastore volume sets** and the chart-created credentials secret, taking every catalog definition with them. The Iceberg files in your bucket survive, but nothing indexes them any more. Your two prerequisite secrets are not deleted. +- **Uninstall deletes the metastore volume sets**, taking every catalog definition with them. The Iceberg files in your bucket survive, but nothing indexes them any more. Your three prerequisite secrets are not deleted. ### Troubleshooting @@ -659,7 +710,7 @@ HA MinIO backups only: version `1.2.1` replaced `postgresHA.backup.minio.accessK **Symptom:** `RELEASE_NAME-polaris` or `RELEASE_NAME-polaris-bootstrap` never becomes ready and `cpln logs` returns zero lines. - **Cause:** A prerequisite secret named by `tokenSigningKey.secretName`, `rootCredentials.secretName` or `storage.credentialsSecretName` does not exist. + **Cause:** A prerequisite secret named by `tokenSigningKey.secretName`, `rootCredentials.secretName`, the metastore `config.credentialsSecretName` or `storage.credentialsSecretName` does not exist. A missing metastore secret also holds `RELEASE_NAME-postgres` (or `RELEASE_NAME-postgres-ha`). **Fix:** Read `status.versions[].message` with `cpln workload get-deployments RELEASE_NAME-polaris --gvc GVC_NAME -o yaml`, create the secret it names as shown in [Prerequisites](#prerequisites), then wait several minutes or force a redeployment. @@ -677,13 +728,7 @@ HA MinIO backups only: version `1.2.1` replaced `postgresHA.backup.minio.accessK **Fix:** Set exactly one of them to `true`. - - **Symptom:** Installing a second Polaris release in the same org fails with `The resource 'my-polaris-db-credentials' cannot be updated because it is being managed by a different release`, and nothing is created. - **Cause:** Secret names are organization-wide, and both releases try to create the metastore credentials secret under the default name. - - **Fix:** Give the new release its own `postgres.config.credentialsSecretName` (or `postgresHA.config.credentialsSecretName` in HA mode). The first release is unaffected. - **Symptom:** The token endpoint rejects the `CLIENT_ID` and `CLIENT_SECRET` now in your root credentials secret. @@ -706,18 +751,18 @@ HA MinIO backups only: version `1.2.1` replaced `postgresHA.backup.minio.accessK **Fix:** Set `storage.credentialsSecretName` to a secret with those exact keys, as in [Building a Lakehouse](#building-a-lakehouse), and recreate the catalog with the S3-compatible settings. - **Symptom:** `helm upgrade` stops at render with `config.username was REMOVED in postgres 3.4.0`, ``the `postgres` block was REMOVED in 2.5.0``, or a `backup.minio.accessKey and backup.minio.secretKey were REMOVED` message. + **Symptom:** `helm upgrade` stops at render with `config.username was REMOVED in postgres 3.4.0`, ``the `postgres` block was REMOVED in 2.5.0``, a `backup.minio.accessKey and backup.minio.secretKey were REMOVED` message, or `polaris: postgres.credentials was REMOVED in 1.3.0`. - **Cause:** Your values predate template version `1.1.0`, `1.2.0` or `1.2.1` respectively. + **Cause:** Your values predate template version `1.1.0`, `1.2.0`, `1.2.1` or `1.3.0` respectively. - **Fix:** Follow [Upgrading from 1.0.x](#upgrading-from-1-0-x), [Upgrading from 1.1.0](#upgrading-from-1-1-0) or [Upgrading from 1.2.0](#upgrading-from-1-2-0). Do not create a metastore credentials secret yourself; this chart creates it. + **Fix:** Follow [Upgrading from 1.0.x](#upgrading-from-1-0-x), [Upgrading from 1.1.0](#upgrading-from-1-1-0), [Upgrading from 1.2.0](#upgrading-from-1-2-0) or [Upgrading from 1.2.1](#upgrading-from-1-2-1), keeping the credentials the metastore already uses in the secret you create. ## Important Notes -- **Create both prerequisite secrets before installing**, or the workloads wedge silently waiting on a secret that does not exist. -- **Change `postgres.credentials.password` before installing.** The default is a published placeholder, and it is the metastore password in both modes. +- **Create all three prerequisite secrets before installing**, or the workloads wedge silently waiting on a secret that does not exist; `status.versions[].message` in `cpln workload get-deployments RELEASE_NAME-polaris --gvc GVC_NAME -o yaml` names it. +- **Editing the metastore credentials secret does not change the database password** — run `ALTER ROLE` first, then force-redeploy Polaris; in HA mode move the leader first ([Rotating the Metastore Password](#rotating-the-metastore-password)). - **`realm` is permanent.** Renaming it hides every existing catalog behind a new, empty realm. - **Root credentials are write-once.** Rotate by creating a new principal through the management API. - **Rotating the token signing key invalidates every outstanding token.** Clients must request a new one. diff --git a/template-catalog/templates/redis-cluster.mdx b/template-catalog/templates/redis-cluster.mdx index 124c0c0b..5eb02bb7 100644 --- a/template-catalog/templates/redis-cluster.mdx +++ b/template-catalog/templates/redis-cluster.mdx @@ -21,14 +21,30 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-redis-cluster-backup` | Cron job that uploads one RDB snapshot per primary shard to S3 or GCS | `backup.enabled: true` | | Secret | `RELEASE_NAME-redis-cluster-config` | `redis.conf` (cluster mode, AOF persistence, port and, when set, the password), file-mounted into each node | — | | Secret | `RELEASE_NAME-redis-cluster-startup` | Start script that boots each node and bootstraps the cluster from replica 0 | — | -| Secret | `RELEASE_NAME-redis-cluster-auth` | Dictionary secret holding `REDIS_PASSWORD` | `redis.password` set | +| Secret | `RELEASE_NAME-redis-cluster-auth` | Dictionary secret holding `REDIS_PASSWORD` | only with an inline `redis.password`; a password from your own secret is read from that secret instead | | Identity | `RELEASE_NAME-redis-cluster-identity` | Workload identity; carries the cloud account binding when backup is enabled | — | -| Policy | `RELEASE_NAME-redis-cluster-policy` | `reveal` on the config and start-script secrets, and on the auth secret when it exists | — | +| Policy | `RELEASE_NAME-redis-cluster-policy` | `reveal` on the config and start-script secrets, and on the auth secret or your password secret when authentication is on | — | | Volume set | `RELEASE_NAME-redis-cluster-vs` | Per-node data directory (`/data`), `ext4` on `general-purpose-ssd` | — | ## Prerequisites -None for a default install. +None for a default install, which runs without a password. + +To protect the cluster with a password (recommended), create a [dictionary secret](/guides/create-secret/dictionary) holding it **before** you install, then point `redis.auth.fromSecret` at it (see [Authentication](#authentication)): + +```bash +cpln secret create-dictionary --name my-redis-cluster-auth --entry password="$(openssl rand -hex 24)" +``` + +The key defaults to `password` (`redis.auth.fromSecret.passwordKey`). Use a password without spaces or quotes. + + +If the secret is missing, the nodes never start and `cpln logs` shows nothing. The reason is in `status.versions[].message`: + +```bash +cpln workload get-deployments RELEASE_NAME-redis-cluster --gvc GVC_NAME -o yaml +``` + Scheduled backups need an S3 or GCS bucket, a Control Plane [cloud account](/guides/create-cloud-account) and, on AWS, a bucket-scoped IAM policy before they can be enabled — see [Backing Up](#backing-up). @@ -38,9 +54,11 @@ A default install needs no values: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/redis-cluster \ - --version 1.6.0 --gvc GVC_NAME + --version 1.7.0 --gvc GVC_NAME ``` +To turn on authentication from the secret in [Prerequisites](#prerequisites), add `--set redis.auth.fromSecret.enabled=true --set redis.auth.fromSecret.name=my-redis-cluster-auth`. + Browse, install, and manage templates visually @@ -147,13 +165,22 @@ Each node keeps its data directory (`/data`) on its own volume in the [volume se ### Authentication +Off by default. Turn it on with ONE of the two options; the secret is the recommended one: + ```yaml -# Configure Redis authentication by uncommenting and setting the password field -redis: {} - # password: "your-secure-password-here" +redis: + auth: + fromSecret: + enabled: false + name: "" # e.g. my-redis-cluster-auth; create it before install (see Prerequisites) + passwordKey: password # the secret key that holds the password + # Less secure: an inline password, stored in plain text in the Helm release. + # password: "change-me-redis-cluster-password" ``` -Authentication is off by default. Setting `redis.password` creates the `RELEASE_NAME-redis-cluster-auth` secret and injects the value into both `requirepass` and `masterauth` in `redis.conf`, so clients and the nodes themselves authenticate with it. The password is a Helm value and is stored in the release. Add `--no-auth-warning -a PASSWORD` to every `redis-cli` call on this page when it is set. +Both options set `requirepass` and `masterauth`, so clients and the nodes themselves authenticate with the password. With the secret, the password reaches the nodes only through an environment variable resolved at start; it is never written into a rendered file or the release. An inline `redis.password` creates the `RELEASE_NAME-redis-cluster-auth` secret and is stored in the release. Setting both options at once is refused at install. Add `--no-auth-warning -a PASSWORD` to every `redis-cli` call on this page when authentication is on. + +**Changing the password:** update the secret, then run `cpln workload force-redeployment RELEASE_NAME-redis-cluster --gvc GVC_NAME`. Nodes restart one at a time, and a node on the new password cannot replicate from a node still on the old one until the rollout completes, so plan a maintenance window. The backup job reads the secret on its next run. ### Backup @@ -192,7 +219,7 @@ The template exposes no public endpoint. Reach the cluster from workloads in the |---|---|---| | Cluster seed address, inside the GVC | `RELEASE_NAME-redis-cluster.GVC_NAME.cpln.local:6379` | Resolves to any node; a cluster-aware client discovers the rest from it | | Redirect hostnames | `RELEASE_NAME-redis-cluster-N.RELEASE_NAME-redis-cluster:6379` | What `MOVED` and `ASK` redirects name node *N* as; resolvable only from inside the GVC, which is why clients must run there | -| Password | `cpln secret reveal RELEASE_NAME-redis-cluster-auth -o yaml` | Only when `redis.password` is set | +| Password | `cpln secret reveal my-redis-cluster-auth -o yaml` (your secret), or `cpln secret reveal RELEASE_NAME-redis-cluster-auth -o yaml` with an inline `redis.password` | Only when authentication is on | Use a cluster-aware client (or `redis-cli -c`): a plain client pointed at one node receives `MOVED` redirects it does not follow, which is the most common cause of "it does not work" with this template. @@ -292,6 +319,7 @@ What the backup is: three standard, gzipped RDB files, one per shard, and RDB fi - **From 1.5.0 or earlier — the readiness probe requires a real `PONG`.** Earlier versions ran a bare `redis-cli ping`, which exits 0 even when the server answers `NOAUTH Authentication required`, so a password-protected cluster could report ready without being usable. From 1.6.0 a node that cannot authenticate is not ready, and because the nodes roll in order, a bad credential stalls the rollout at the first replica instead of rolling all six into a broken state. - **From 1.5.0 or earlier — cluster creation retries transient DNS failures.** During first boot the cluster create step re-resolves every peer hostname, and a name that briefly stops resolving used to restart the container. From 1.6.0 creation retries in place and still fails loudly (`FATAL: cluster create did not succeed after 12 attempts`) if it genuinely cannot succeed. - **1.6.0 adds `engine`.** The default `redis` deploys exactly as before; see [Engine](#engine) before setting `valkey`. +- **1.7.0 adds `redis.auth.fromSecret`**, which reads the password from your own secret. An inline `redis.password` keeps working unchanged; see [Authentication](#authentication). Upgrades roll the nodes in order; each node's data and cluster state are on its volume, so the cluster re-forms from `nodes.conf` as the nodes come back. @@ -335,9 +363,9 @@ Upgrades roll the nodes in order; each node's data and cluster state are on its **Symptom:** a client or `redis-cli` call answers `(error) NOAUTH Authentication required.` - **Cause:** `redis.password` is set and the client did not authenticate. + **Cause:** authentication is on (`redis.auth.fromSecret` or `redis.password`) and the client did not authenticate. - **Fix:** pass the password — `redis-cli --no-auth-warning -a PASSWORD`, or your client library's password option. Read it back with `cpln secret reveal RELEASE_NAME-redis-cluster-auth -o yaml`. + **Fix:** pass the password — `redis-cli --no-auth-warning -a PASSWORD`, or your client library's password option. Read it back from your secret with `cpln secret reveal my-redis-cluster-auth -o yaml`, or with `cpln secret reveal RELEASE_NAME-redis-cluster-auth -o yaml` for an inline password. **Symptom:** after a `cpln helm upgrade` that changed `engine`, the first replaced node never reports ready, `cpln logs` returns no lines for it and the deployment message is empty. @@ -365,7 +393,8 @@ Upgrades roll the nodes in order; each node's data and cluster state are on its ## Important Notes - **Your client must speak the Redis Cluster protocol and run inside the same GVC.** A plain client gets `MOVED` redirects it will not follow, and the redirect hostnames resolve only inside the GVC. -- **Authentication is off by default.** Anything `internalAccess` admits has full access until you set `redis.password`. +- **Authentication is off by default.** Anything `internalAccess` admits has full access until you turn it on with `redis.auth.fromSecret` (recommended) or an inline `redis.password`. +- **A changed password secret takes effect only after a forced redeployment** of `RELEASE_NAME-redis-cluster`; nodes on the new and the old password cannot replicate until the rollout completes. - **Keep `replicas` at 6.** Fewer cannot form a cluster; more is rejected by a platform quota. - **Never set `internalAccess.type: none`.** The nodes stop reaching each other and the cluster breaks. - **Choose `engine` at install time.** Switching it on a release that holds data is unsupported, and the failure leaves no log output. diff --git a/template-catalog/templates/redis.mdx b/template-catalog/templates/redis.mdx index 926cf1ea..424f186a 100644 --- a/template-catalog/templates/redis.mdx +++ b/template-catalog/templates/redis.mdx @@ -22,8 +22,8 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-redis-backup` | Cron job that uploads an RDB snapshot to S3 or GCS; runs as the Redis identity | `backup.enabled: true` | | Secret | `RELEASE_NAME-redis-config` | `redis.conf`, file-mounted into each Redis container | — | | Secret | `RELEASE_NAME-sentinel-config` | `sentinel.conf`, file-mounted into each Sentinel container | `mode: production` (default) | -| Secret | `RELEASE_NAME-redis-auth-password` | Redis password | `redis.auth.password.enabled: true` | -| Secret | `RELEASE_NAME-sentinel-auth-password` | Sentinel password | `sentinel.auth.password.enabled: true` (production mode) | +| Secret | `RELEASE_NAME-redis-auth-password` | Redis password, created only for a password set in values | `redis.auth.password.enabled: true` | +| Secret | `RELEASE_NAME-sentinel-auth-password` | Sentinel password, created only for a password set in values | `sentinel.auth.password.enabled: true` (production mode) | | Secret | `RELEASE_NAME-redis-backup-config` | Bucket and region for the backup job | `backup.enabled: true` | | Identity | `RELEASE_NAME-redis-identity` | Identity of the Redis workload and the backup job; carries the cloud account binding when backup is enabled | — | | Identity | `RELEASE_NAME-sentinel-identity` | Sentinel workload identity | `mode: production` (default) | @@ -40,15 +40,14 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha None for a default install. Three optional features need something to exist first: - - By default authentication is off, and `redis.auth.password` creates the password secret for you. To supply the password from a secret you manage instead, create a dictionary secret whose key matches `passwordKey` (default `password`) before installing. Use a separate secret for the Sentinel tier if you enable `sentinel.auth.fromSecret`. + + Authentication is off by default. To turn it on, the recommended way is a password secret you create **before** install, so the password never enters the release values. Create a [dictionary secret](/guides/create-secret/dictionary) with the password under the key `password` (the default `passwordKey`); one secret can serve both Redis and Sentinel: ```bash - cpln secret create-dictionary --name SECRET_NAME \ - --entry password='YOUR-STRONG-PASSWORD' + cpln secret create-dictionary --name my-redis-password --entry password="$(openssl rand -hex 32)" ``` - Set `redis.auth.fromSecret.name` (and `sentinel.auth.fromSecret.name`) to this name and `fromSecret.enabled` to `true`. See [Authentication](#authentication). + Set `redis.auth.fromSecret.name` (and, in production mode, `sentinel.auth.fromSecret.name`) to this name and `fromSecret.enabled` to `true`. See [Authentication](#authentication). If the secret does not exist when the workload starts, the deployment wedges silently: `cpln logs` returns nothing and the workload never becomes ready. The missing secret is named only in `status.versions[].message`: @@ -77,6 +76,8 @@ cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/redis \ --version 3.9.0 --gvc GVC_NAME ``` +To turn on authentication from the password secret in [Prerequisites](#prerequisites), set `redis.auth.fromSecret.enabled=true` and `redis.auth.fromSecret.name=my-redis-password`, and the same two keys under `sentinel.auth` in production mode. + Browse, install, and manage templates visually @@ -189,22 +190,40 @@ redis: #### Authentication -Authentication is off by default. Enable exactly one method per tier; the chart refuses to render with both enabled. In production mode the Sentinel tier takes the same two options under `sentinel.auth`; dev mode uses `redis.auth` only and refuses `sentinel.auth`. +Authentication is off by default. Enable exactly one method per tier; the chart refuses to render with both enabled. In production mode apply the same config under both `redis.auth` and `sentinel.auth`; dev mode uses `redis.auth` only and refuses `sentinel.auth`. + +Recommended: read the password from the secret created in [Prerequisites](#prerequisites): ```yaml redis: auth: fromSecret: - enabled: false # set true to read the password from a secret you created (see Prerequisites) - name: example-redis-auth-password + enabled: true + name: my-redis-password # dictionary secret; must exist before install + passwordKey: password # key holding the password + +sentinel: + auth: + fromSecret: + enabled: true + name: my-redis-password # the same secret can serve both tiers passwordKey: password +``` + +Still supported, but less secure: a password in values is stored in the Helm release, and the chart creates the password secret from it: + +```yaml +redis: + auth: password: - enabled: false # set true to have the chart create the password secret from `value` + enabled: true value: change-me-redis-password ``` +Set the password at install. Nodes read the secret only when they start, so a new value reaches them only after `cpln workload force-redeployment`, and that is a password change on a running release (see the warning below). With Sentinel persistence on, Sentinel also keeps the password it wrote at its first start. + -**Do not change the password on a running release.** Changing `redis.auth.password.value` (or the contents of the referenced secret) stops the rolling update: a restarted replica can no longer authenticate to the master, so it never becomes ready and the rollout stops advancing. Setting the previous password back recovers it. To change the password, uninstall and reinstall. +**Do not change the password on a running release,** in values or in the password secret. Changing `redis.auth.password.value` (or the contents of the referenced secret) stops the rolling update: a restarted replica can no longer authenticate to the master, so it never becomes ready and the rollout stops advancing. Setting the previous password back recovers it. To change the password, uninstall and reinstall. A password may contain spaces, quotes, backslashes and any other printable character. Line breaks, tabs and other control characters are not supported, because each password is written on a single configuration line. @@ -675,7 +694,8 @@ Afterwards, point your clients at Sentinel (`RELEASE_NAME-sentinel.GVC_NAME.cpln - **Choose `engine` at install time.** Switching it on a running release does not migrate anything; Valkey cannot read data written by Redis 8. - **Choose persistence at install time when you can.** To add it to a running release, set `suspend: true` in the same upgrade, then `suspend: false`; data held only in memory is lost. - **`suspend: true` stops every node.** Without persistence, the data in memory is lost. -- **Never change the password in place.** The rolling update stops advancing; set the previous password back to recover, and reinstall to change it. +- **Prefer `auth.fromSecret` over a password in values**, which is stored in the Helm release. Create the secret before install; a missing one is named only in `status.versions[].message` from `cpln workload get-deployments`. +- **Never change the password in place,** in values or in the password secret. The rolling update stops advancing; set the previous password back to recover, and reinstall to change it. - **In production mode, always discover the master through Sentinel** (`mymaster` at `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379`). A client pinned to replica 0 gets `READONLY` errors after any failover or upgrade. In dev mode, connect to `RELEASE_NAME-redis.GVC_NAME.cpln.local:6379` directly. - **Enable `sentinel.persistence` in production** so the elected master survives Sentinel restarts. - **The Redis 7.4 to 8 upgrade is one-way.** Snapshot the volume set before upgrading if you need a rollback path. diff --git a/template-catalog/templates/searxng.mdx b/template-catalog/templates/searxng.mdx index 0689b21c..17b9093b 100644 --- a/template-catalog/templates/searxng.mdx +++ b/template-catalog/templates/searxng.mdx @@ -28,14 +28,13 @@ A default install is private: `publicAccess.enabled` is `false` and `limiter.ena | Workload | `RELEASE_NAME-searxng` | SearXNG web UI and JSON API (standard, `replicas` replicas, HTTP `8080`) | — | | Secret | `RELEASE_NAME-searxng-settings` | Rendered `settings.yml`, mounted read-only at `/etc/searxng/settings.yml` | — | | Identity | `RELEASE_NAME-searxng-identity` | Identity of the SearXNG workload | — | -| Policy | `RELEASE_NAME-searxng-policy` | Grants the identity `reveal` on exactly the secrets the workload reads, including your signing-key secret | — | +| Policy | `RELEASE_NAME-searxng-policy` | Grants the identity `reveal` on exactly the secrets the workload reads, including your signing-key secret and, when configured, your Redis and `/metrics` password secrets | — | | Secret | `RELEASE_NAME-searxng-startup` | Start-up script that waits for the datastore before starting SearXNG | `redis.enabled: true` (default) | | Workload | `RELEASE_NAME-redis` | Redis for limiter counters and shared state (stateful, 1 replica, TCP `6379`); not a result cache | `redis.enabled: true` (default) | | Workload | `RELEASE_NAME-sentinel` | Redis Sentinel the bundled [redis](/template-catalog/templates/redis) template requires (stateful, 1 replica, TCP `26379`) | `redis.enabled: true` (default) | | Identity | `RELEASE_NAME-redis-identity`, `RELEASE_NAME-sentinel-identity` | Identities of the datastore workloads | `redis.enabled: true` (default) | | Policy | `RELEASE_NAME-redis-policy`, `RELEASE_NAME-sentinel-policy` | Secret access for the datastore workloads | `redis.enabled: true` (default) | | Secret | `RELEASE_NAME-redis-config`, `RELEASE_NAME-sentinel-config` | Redis and Sentinel configuration files | `redis.enabled: true` (default) | -| Secret | `RELEASE_NAME-redis-auth-password` | Datastore password, wired into the SearXNG connection URL | `redis.redis.auth.password.enabled: true` | No volume set is created anywhere in this template. Everything durable is either your signing-key secret or a preference cookie in a user's own browser, so an uninstall leaves no storage behind. @@ -72,13 +71,24 @@ Use `get-deployments` — plain `cpln workload get` has no `versions` key. Creat Treat the key as write-once and keep a copy outside Control Plane: replacing it discards every saved preference in every browser, and reinstalling against the same secret loses nothing. See [Rotating the Signing Key](#rotating-the-signing-key). Nothing else is required — no cloud account, no domain, no external database. +**Optional secrets**, each needed only for the feature it turns on, and each created before the install or upgrade that uses it: + +```bash +# Redis password (redis.redis.auth.fromSecret) — a dictionary secret with the key `password`. +# Use only letters, digits, - and _: SearXNG puts it in its connection URL as-is. +cpln secret create-dictionary --name my-searxng-redis --entry password="$(openssl rand -hex 24)" + +# /metrics basic-auth password (metrics.secretName) — an opaque secret. +printf '%s' "$(openssl rand -hex 24)" | cpln secret create-opaque --name my-searxng-metrics-password --encoding plain -f - +``` + ## Installation Install the chart from the marketplace registry, pointing it at the secret you created: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/searxng \ - --version 1.0.0 --gvc GVC_NAME \ + --version 1.1.0 --gvc GVC_NAME \ --set secretKey.secretName=SECRET_NAME ``` @@ -133,7 +143,7 @@ imageProxy: true # proxy result images through this instance inste baseUrl: "" # set only behind a custom domain, e.g. https://search.example.com (empty = derive from request) limiter: enabled: false # bot-detection rate limiter; requires redis.enabled. NOTE: it 429s non-browser clients, including the JSON API -extraSettings: {} # merged into settings.yml verbatim, e.g. {general: {enable_metrics: false}} +extraSettings: {} # merged into settings.yml verbatim, e.g. {general: {enable_metrics: false}}; general.open_metrics is refused — use metrics.secretName ``` - `formats` — a format that is not listed is refused, so removing `json` disables the API. @@ -158,6 +168,15 @@ secretKey: The opaque secret from [Prerequisites](#prerequisites). The chart grants the workload `reveal` on exactly that secret and injects it as an environment variable, so the key never appears in the rendered configuration. +### Metrics Password + +```yaml +metrics: + secretName: "" # e.g. my-searxng-metrics-password; must exist BEFORE install +``` + +Empty keeps `/metrics` closed (404). Name the opaque secret from [Prerequisites](#prerequisites) and `/metrics` serves Prometheus metrics behind HTTP basic auth — any user name, the secret's payload as the password. The password is written into `settings.yml` inside the container when it starts, so it never appears in the Helm release. + ### Access ```yaml @@ -190,9 +209,9 @@ redis: image: redis:8 replicas: 1 # MUST be 1 (validated) — SearXNG must reach the master directly auth: - password: - enabled: false # true = require AUTH; password is wired into the connection URL - value: change-me-searxng-redis + fromSecret: + enabled: false # true = require AUTH, password read from the secret below + name: "" # prerequisite dictionary secret (key: password); URL-safe characters only resources: minCpu: 80m maxCpu: 200m @@ -210,7 +229,7 @@ redis: - `redis.enabled: false` runs SearXNG standalone; `limiter.enabled` must then stay `false`, and the chart refuses to render the combination. - `redis.redis.replicas` must stay `1`, and the chart rejects anything else: SearXNG's client is not Sentinel-aware and has to reach the master directly, so extra replicas would be read-only and would silently break the limiter. One Sentinel is required by the redis template and cannot be scaled to zero. -- Change `redis.redis.auth.password.value` from the placeholder if you enable `AUTH`. `redis.redis.auth.fromSecret` is not supported by this template. +- `AUTH` is off by default; the same-GVC firewall is the boundary. To require it, create the Redis password secret from [Prerequisites](#prerequisites) before installing, then set `redis.redis.auth.fromSecret.enabled: true` and `redis.redis.auth.fromSecret.name` to its name. Sentinel authentication must stay off; the chart refuses it, because SearXNG never talks to the Sentinel. ## Connecting @@ -222,7 +241,8 @@ SearXNG has no login, so there are no credentials anywhere in this table. A defa | JSON API for in-GVC agents | `http://RELEASE_NAME-searxng.GVC_NAME.cpln.local:8080/search?q=QUERY&format=json` | Always use the fully-qualified name; the bare short name does not resolve to this workload. | | Web UI and API, public | `status.canonicalEndpoint` of `RELEASE_NAME-searxng` (`https://...cpln.app`) | Only when `publicAccess.enabled: true`. Unauthenticated. | | Health | `GET /healthz` on port `8080` | Unaffected by the limiter. | -| Redis | `RELEASE_NAME-redis.GVC_NAME.cpln.local:6379` | Only when `redis.enabled: true`; `AUTH` only when `redis.redis.auth.password.enabled: true`. | +| Redis | `RELEASE_NAME-redis.GVC_NAME.cpln.local:6379` | Only when `redis.enabled: true`; `AUTH` only when `redis.redis.auth.fromSecret.enabled: true`, with the `password` key of that secret. | +| Metrics | `/metrics` on any of the addresses above | Only when `metrics.secretName` is set; HTTP basic auth, any user name, the password in that secret. | To verify from your own machine, open a tunnel and call the API through it: @@ -309,6 +329,23 @@ cpln apply -f rotate.yaml cpln workload force-redeployment RELEASE_NAME-searxng --gvc GVC_NAME ``` +### Upgrading from 1.0.0 + +1.1.0 takes no password as a plaintext value. Carrying either old key forward fails the render with a message naming the replacement, before anything is changed. + +**If you ran 1.0.0 with Redis auth on** (`redis.redis.auth.password.enabled: true`), the chart-created `RELEASE_NAME-redis-auth-password` secret is deleted by the upgrade. Copy your current password into a secret under a **new** name first: + +```bash +cpln secret reveal RELEASE_NAME-redis-auth-password -o yaml +cpln secret create-dictionary --name my-searxng-redis --entry password='YOUR-CURRENT-REDIS-PASSWORD' +``` + +Then upgrade with `redis.redis.auth.fromSecret.enabled: true` and `redis.redis.auth.fromSecret.name: my-searxng-redis`, and remove `redis.redis.auth.password` (and any `redis.sentinel.auth.password`). With Redis auth off in 1.0.0 (the default), just remove `redis.redis.auth.password` from your values if you set it. + +**If you opened `/metrics`** with `extraSettings.general.open_metrics`, create the opaque secret from [Prerequisites](#prerequisites) holding that password, set `metrics.secretName` to its name, and remove `open_metrics` from `extraSettings`. + +The new version of a workload can wait several minutes with `is not allowed to reveal the secret` in `get-deployments` while access to a newly named secret propagates; the previous version keeps serving meanwhile, and it clears by itself (or run `cpln workload force-redeployment` on that workload to retry). + ### Scaling and Availability - `replicas` scales the SearXNG tier horizontally. Instances share nothing on disk and there is no server-side session to pin, so any replica can serve any request; with `redis.enabled: true` the limiter's counters are shared across replicas. @@ -362,7 +399,7 @@ cpln workload force-redeployment RELEASE_NAME-searxng --gvc GVC_NAME **Cause:** the start-up script could not reach the bundled datastore within its wait window, and with the limiter on it refuses to serve an instance that believes it is rate-limited and is not. - **Fix:** check `RELEASE_NAME-redis` with `cpln workload get-deployments RELEASE_NAME-redis --gvc GVC_NAME -o yaml` and its logs. If `redis.redis.auth.password.enabled` is on, make sure `redis.redis.auth.password.value` is set. Once the datastore is healthy, SearXNG starts on its next restart. + **Fix:** check `RELEASE_NAME-redis` with `cpln workload get-deployments RELEASE_NAME-redis --gvc GVC_NAME -o yaml` and its logs. If `redis.redis.auth.fromSecret.enabled` is on, make sure the secret named by `redis.redis.auth.fromSecret.name` exists and holds the key `password`. Once the datastore is healthy, SearXNG starts on its next restart. **Symptom:** `cpln helm install` or `cpln helm upgrade` stops before touching any resource: @@ -406,7 +443,9 @@ cpln workload force-redeployment RELEASE_NAME-searxng --gvc GVC_NAME - **The limiter blocks the JSON API for non-browser clients.** Pick one of the two shapes in [Choosing an Access Shape](#choosing-an-access-shape) deliberately; never run public with the limiter off. - **Settings changes need a forced redeployment.** `settings.yml` is a mounted secret and running replicas do not re-read it — see [Changing Settings](#changing-settings). - **`redis.redis.replicas` must stay `1`, and `limiter.enabled` requires `redis.enabled`.** Both are enforced at render time, so a bad combination fails the install rather than shipping a silently unlimited instance. -- **`/metrics` is off as shipped and needs a password to open.** Upstream gates it on `general.open_metrics`, which is empty by default; setting it through `extraSettings` enables the route behind HTTP basic auth with that password. `extraSettings: {general: {enable_metrics: false}}` stops collecting the data behind `/stats`. +- **`/metrics` is off as shipped and needs a password to open.** It returns 404 unless `metrics.secretName` names a password secret, after which it requires that password via HTTP basic auth; see [Metrics Password](#metrics-password). With `limiter.enabled`, non-browser scrapers are rate-limited on `/metrics` too. `extraSettings: {general: {enable_metrics: false}}` stops collecting the data behind `/stats`. +- **Changing the `/metrics` password secret does not reach a running workload.** Run `cpln workload force-redeployment RELEASE_NAME-searxng --gvc GVC_NAME` afterwards; until then the old password keeps working. +- **Do not change the Redis password in place.** The Redis node, the Sentinel and SearXNG each read it only when they start, and the Sentinel keeps the one it started with. Redis holds nothing durable here, so to change it uninstall and reinstall the release with a new secret (reuse the same signing-key secret and nothing else is lost). - **Uninstalling removes everything the template created** — workloads, secrets, identities and policies, with no volume sets left behind. Your own signing-key secret is yours and is left alone. ## External References diff --git a/template-catalog/templates/sftpgo.mdx b/template-catalog/templates/sftpgo.mdx index f24d3fd1..d26e4af3 100644 --- a/template-catalog/templates/sftpgo.mdx +++ b/template-catalog/templates/sftpgo.mdx @@ -22,17 +22,17 @@ This template deploys into an **existing GVC** that you already have. It does no | Workload | `RELEASE_NAME-sftpgo` | SFTPGo server (stateful, 1 replica); SFTP on port `2022`, web admin and REST API on port `8080` when `webAdmin.enabled` | — | | Workload | `RELEASE_NAME-sftpgo-proxy` | Always-on proxy that holds client connections, wakes SFTPGo and suspends it again after `scaleToZero.idleHold` (standard, 1 replica) | `mode: scale_to_zero` (default) | | Volume set | `RELEASE_NAME-sftpgo-vs` | SFTPGo's embedded user database and SSH host keys (10 GiB); a final snapshot is kept for 7 days on deletion | — | -| Secret | `RELEASE_NAME-sftpgo-users` | The declared-users file built from `users[]` and the storage block | — | +| Secret | `RELEASE_NAME-sftpgo-users` | The declared-users template built from `users[]` and the storage block; it holds no credentials — passwords and S3-compatible keys are filled in from your secrets at start | — | | Identity | `RELEASE_NAME-sftpgo-identity` | SFTPGo's identity; carries the AWS or GCP cloud account binding for keyless bucket access | — | | Identity | `RELEASE_NAME-sftpgo-proxy-identity` | The proxy's identity | `mode: scale_to_zero` (default) | -| Policy | `RELEASE_NAME-sftpgo-policy` | Grants SFTPGo's identity `reveal` on your admin secret and the users secret | — | +| Policy | `RELEASE_NAME-sftpgo-policy` | Grants SFTPGo's identity `reveal` on the secrets you create (admin, user passwords and, with `minio`, the S3-compatible credentials) and the users secret | — | | Policy | `RELEASE_NAME-sftpgo-proxy-policy` | Lets the proxy's identity view and edit exactly the `RELEASE_NAME-sftpgo` workload, so it can suspend and wake it | `mode: scale_to_zero` (default) | The files your users upload are not stored on the volume set — they go to your bucket, which the template never creates or deletes. ## Prerequisites -Two things must exist before you install: the administrator secret, and a bucket in one of the supported backends with its access setup. +These must exist before you install: the administrator secret, the SFTP user passwords secret (while any user logs in with a password, as the default `sftpuser` does), and a bucket in one of the supported backends with its access setup — for an S3-compatible server that includes a credentials secret. ### Admin Credentials Secret @@ -80,11 +80,24 @@ Use **`get-deployments`** — plain `cpln workload get` has no `versions` key. O ```bash cpln workload force-redeployment RELEASE_NAME-sftpgo --gvc GVC_NAME ``` + +The same applies to the user passwords and S3-compatible credentials secrets below. +### SFTP User Passwords + +Required while any `users[]` entry logs in with a password (the default `sftpuser` does). A [dictionary secret](/guides/create-secret/dictionary) with one key per such username; the value is that user's password. Users with only `publicKeys` need no entry. + +```bash +cpln secret create-dictionary --name my-sftpgo-user-passwords \ + --entry sftpuser="$(openssl rand -hex 16)" +``` + +Set `userPasswords.secretName` to that name. Declared users are re-applied on every start, so the value in this secret **is** the password: to change one, update the secret and force a redeployment. If a password user has no key in the secret, the deployment wedges with nothing in the log: `cpln workload get-deployments RELEASE_NAME-sftpgo --gvc GVC_NAME -o yaml` shows `couldn't find key USERNAME in Secret` under `status.versions[].message`. Add the key, then force a redeployment. An empty value stops SFTPGo with `USERPW_ is empty` in its log. + ### Object Storage -Choose a backend with `storage.type` and complete its setup. AWS and GCP are keyless: SFTPGo's workload identity receives temporary credentials through a Control Plane cloud account, and no keys are stored. An S3-compatible server takes static keys. +Choose a backend with `storage.type` and complete its setup. AWS and GCP are keyless: SFTPGo's workload identity receives temporary credentials through a Control Plane cloud account, and no keys are stored. An S3-compatible server takes static keys from a secret you create. @@ -144,10 +157,16 @@ Choose a backend with `storage.type` and complete its setup. AWS and GCP are key Set `storage.minio.endpoint` to the S3 API address including the scheme and port. For an S3 server deployed from the catalog in the same GVC, use its fully qualified internal name, for example `http://WORKLOAD_NAME.GVC_NAME.cpln.local:9000` for [MinIO](/template-catalog/templates/minio). - - Set `storage.minio.accessKey` and `storage.minio.accessSecret` to a key pair with access to the bucket. For the MinIO template, these are the `username` and `password` in the dictionary secret named by its `admin.credentialsSecretName`. + + A [dictionary secret](/guides/create-secret/dictionary) with keys `accessKey` and `secretKey` that can access the bucket. For the MinIO template, these are the `username` and `password` in the dictionary secret named by its `admin.credentialsSecretName`. - These two keys are **Helm values**, not a secret reference: SFTPGo reads them from its declared-users file, which cannot resolve a secret. They are stored in the Helm release. + ```bash + cpln secret create-dictionary --name my-sftpgo-minio-credentials \ + --entry accessKey=YOUR_ACCESS_KEY \ + --entry secretKey=YOUR_SECRET_KEY + ``` + + Set `storage.minio.credentialsSecretName` to that name. @@ -155,18 +174,17 @@ Choose a backend with `storage.type` and complete its setup. AWS and GCP are key ## Installation -With the admin secret and your bucket in place, install into your existing GVC. This example uses the default AWS backend and replaces the default SFTP user, whose shipped password is a placeholder: +With the admin secret, the user passwords secret and your bucket in place, install into your existing GVC. This example uses the default AWS backend and the default `sftpuser`, whose password comes from the user passwords secret: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/sftpgo \ - --version 1.1.0 --gvc GVC_NAME \ + --version 1.2.0 --gvc GVC_NAME \ --set admin.secretName=SECRET_NAME \ + --set userPasswords.secretName=my-sftpgo-user-passwords \ --set storage.aws.bucket=BUCKET_NAME \ --set storage.aws.region=REGION \ --set storage.aws.cloudAccountName=CLOUD_ACCOUNT_NAME \ - --set storage.aws.policyName=IAM_POLICY_NAME \ - --set 'users[0].username=sftpuser' \ - --set 'users[0].password=YOUR-STRONG-PASSWORD' + --set storage.aws.policyName=IAM_POLICY_NAME ``` Setting any `users[...]` entry with `--set` replaces the whole default list, so always pass `username` with it. For more than one user, or for public-key users, put `users` in a values file instead. @@ -223,7 +241,7 @@ scaleToZero: # only used when mode is scale_to_zero ### SFTPGo Server ```yaml -image: drakkan/sftpgo:v2.7.4-distroless-slim +image: drakkan/sftpgo:v2.7.4-alpine-slim # needs /bin/sh: the chart runs a start script resources: # SFTPGo container; CPU governs transfer throughput and wake speed minCpu: 250m @@ -235,7 +253,7 @@ volumeset: capacity: 10 # GiB (minimum 10) — stores the embedded database and SSH host keys ``` -The volume set keeps SFTPGo's SSH host keys stable across restarts and wakes, so clients do not see a changed host key. The template prepares the volume for the image's non-root user, so if you change `image`, stay on the same `drakkan/sftpgo` distroless image family. +The volume set keeps SFTPGo's SSH host keys stable across restarts and wakes, so clients do not see a changed host key. The template prepares the volume for the image's non-root user, so if you change `image`, stay on the same `drakkan/sftpgo` image family. A custom `image` must contain `/bin/sh` and `awk`: the chart builds the users file, with the passwords from your secret, in a start script (the default `alpine-slim` tag has both; `distroless` tags do not). ### Admin @@ -275,8 +293,9 @@ storage: bucket: my-sftp-bucket # bucket must already exist region: us-east-1 keyPrefix: "" # optional bucket-wide folder prefix, e.g. sftp/ - accessKey: my-minio-username - accessSecret: my-minio-password + # REQUIRED with type minio — CREATE IT BEFORE YOU INSTALL: a dictionary secret + # with keys accessKey and secretKey (see Prerequisites). + credentialsSecretName: my-sftpgo-minio-credentials ``` Only the block matching `storage.type` is used. The `my-…` defaults are placeholders for resources you own and must replace. For `minio`, use the fully qualified internal hostname shown in [Prerequisites](#prerequisites) rather than the short name in the default. @@ -284,16 +303,21 @@ Only the block matching `storage.type` is used. The `my-…` defaults are placeh ### Users ```yaml +# REQUIRED while any user logs in with a password — CREATE IT BEFORE YOU INSTALL. +# A dictionary secret with one key per such username; the value is its password. +# If it does not exist the deployment wedges silently. See Prerequisites. +userPasswords: + secretName: my-sftpgo-user-passwords + users: - username: sftpuser - password: change-me-sftp-user # omit when publicKeys is set publicKeys: [] # SSH public keys, e.g. ["ssh-ed25519 AAAA... user@laptop"] + # passwordLogin: true # password from userPasswords; default true only when publicKeys is empty # keyPrefix: custom/folder/ # optional per-user bucket folder override ``` -- At least one user is required, usernames must be unique, and every user needs a `password`, at least one entry in `publicKeys`, or both. +- At least one user is required and usernames must be unique. Passwords never go in values: they come from the `userPasswords.secretName` secret (see [SFTP User Passwords](#sftp-user-passwords)), keyed by username. A user logs in with a password when it has no `publicKeys`, or when `passwordLogin: true` is set alongside its keys; a user with `passwordLogin: false` needs at least one public key. - Each user is confined to the bucket folder `{keyPrefix}{username}/`, where `keyPrefix` is the backend's bucket-wide prefix; a per-user `keyPrefix` replaces that folder entirely. -- **`users[].password` is a Helm value used exactly as written** — it cannot be moved into a secret, because SFTPGo reads it from a file that does not resolve secret references. Change the `change-me-sftp-user` default, or better, omit `password` and use `publicKeys`, which needs no credential in your values at all. - The list is re-applied every time SFTPGo starts or wakes. Edits made to a declared user through the web admin or REST API are overwritten; users created there are left alone. ### Access @@ -332,7 +356,7 @@ webAdmin: | SFTP, inside the GVC (`scale_to_zero`) | `RELEASE_NAME-sftpgo-proxy.GVC_NAME.cpln.local:2022` | Reachable from any workload in the same GVC. | | SFTP, inside the GVC (`always_warm`) | `RELEASE_NAME-sftpgo.GVC_NAME.cpln.local:2022` | Scoped by `internalAccess`. | | Web admin and REST API | `http://localhost:8080` through a port-forward | Requires `webAdmin.enabled: true`. | -| SFTP logins | Your `users[]` values | Password or SSH key. | +| SFTP logins | Usernames from your `users[]` values; passwords from your `userPasswords.secretName` secret | Password or SSH key. | | Administrator login | `cpln secret reveal SECRET_NAME -o yaml` | `username` / `password` from your admin secret. | Find the public endpoint on the client-facing workload (the proxy in the default mode): @@ -422,15 +446,41 @@ sftpgo: admin.password was removed in 1.1.0. Put it in the prerequisite dictiona Remove `admin.username` and `admin.password`, and set `admin.secretName` instead. Leave `users` and the storage block as they are. - Run `cpln helm upgrade` with `--version 1.1.0` and your updated values, as described in the [CLI guide](/template-catalog/install-manage/cli). In `scale_to_zero` mode an upgrade wakes a suspended server; it suspends again after the next connection comes and goes. + Run `cpln helm upgrade` with `--version 1.1.0` and your updated values, as described in the [CLI guide](/template-catalog/install-manage/cli). In `scale_to_zero` mode an upgrade wakes a suspended server; it suspends again after the next connection comes and goes. When upgrading straight to 1.2.0 or later, also follow [Upgrading from 1.1.0](#upgrading-from-1-1-0). +### Upgrading from 1.1.0 + +Version `1.2.0` removes `users[].password`, `storage.minio.accessKey` and `storage.minio.accessSecret`; the render fails naming the replacement while any of them is still in your values. Before upgrading, read the values the release runs with today. The 1.1.0 users file holds each user's `password` and, with `minio`, the `access_key` and the `access_secret` `payload`: + +```bash +cpln secret reveal RELEASE_NAME-sftpgo-users -o yaml +``` + +Create the new secrets with those **current** values, so every login keeps working: + +```bash +cpln secret create-dictionary --name my-sftpgo-user-passwords \ + --entry 'sftpuser=YOUR_CURRENT_SFTPUSER_PASSWORD' + +# storage.type minio only +cpln secret create-dictionary --name my-sftpgo-minio-credentials \ + --entry 'accessKey=YOUR_CURRENT_ACCESS_KEY' \ + --entry 'secretKey=YOUR_CURRENT_ACCESS_SECRET' +``` + +Then delete those keys from your values, set `userPasswords.secretName` (and `storage.minio.credentialsSecretName`), and upgrade. A user that had both a password and `publicKeys` needs `passwordLogin: true` to keep its password. The upgrade grants the workload access to the new secrets; if `status.versions[].message` reports it is not allowed to reveal one, wait a few minutes, then run `cpln workload force-redeployment RELEASE_NAME-sftpgo --gvc GVC_NAME`. + ### Changing Credentials - **Administrator password** — change it in the web admin or through the REST API. Editing the admin secret has no effect on an install that has already booted. -- **SFTP user passwords and keys** — edit `users` in your values and run `cpln helm upgrade`, then force a redeployment of `RELEASE_NAME-sftpgo` (in `scale_to_zero` mode the change applies at the next wake). The declared-users file is re-applied when SFTPGo restarts, and the upgrade alone does not restart it. -- **S3-compatible access keys** — edit `storage.minio.accessKey` and `storage.minio.accessSecret` and run `cpln helm upgrade`, then force a redeployment of `RELEASE_NAME-sftpgo` (in `scale_to_zero` mode the change applies at the next wake). +- **SFTP user passwords** — update the user's key in the `userPasswords.secretName` secret (`cpln secret edit SECRET_NAME`), then force a redeployment of `RELEASE_NAME-sftpgo` (in `scale_to_zero` mode the change applies at the next wake). The declared-users file is re-applied when SFTPGo restarts, and a secret change alone does not restart it. Public keys are edited in `users` and applied with `cpln helm upgrade` plus the same redeployment. +- **S3-compatible access keys** — update the `storage.minio.credentialsSecretName` secret, then force a redeployment of `RELEASE_NAME-sftpgo` (in `scale_to_zero` mode the change applies at the next wake). + +```bash +cpln workload force-redeployment RELEASE_NAME-sftpgo --gvc GVC_NAME +``` ### Switching Modes @@ -446,7 +496,7 @@ The template runs **one SFTPGo replica** and exposes no replica count. A restart **Symptom:** `cpln helm install` succeeded, but SFTPGo never starts and `cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-sftpgo"}' --limit 50 --since 10m` returns no lines at all. - **Cause:** The admin secret named by `admin.secretName` does not exist, so the container is never started. The deployment message names it: + **Cause:** A prerequisite secret — the admin secret named by `admin.secretName`, the user passwords secret named by `userPasswords.secretName` (or a password user's key in it), or the S3-compatible credentials secret — does not exist, so the container is never started. The deployment message names it: ```bash cpln workload get-deployments RELEASE_NAME-sftpgo --gvc GVC_NAME -o yaml @@ -455,11 +505,11 @@ The template runs **one SFTPGo replica** and exposes no replica count. A restart **Fix:** Create the secret as shown in [Prerequisites](#prerequisites). The deployment recovers on its own after several minutes, or immediately with `cpln workload force-redeployment RELEASE_NAME-sftpgo --gvc GVC_NAME`. - **Symptom:** The install stops at render time with `sftpgo: every users[] entry requires a username`, or with `sftpgo: user 'NAME' needs a password or at least one entry in publicKeys`. + **Symptom:** The install stops at render time with `sftpgo: every users[] entry requires a username`, or with `sftpgo: user 'NAME' has passwordLogin: false and no publicKeys, so it could never log in`. - **Cause:** Setting `users[0].password` alone with `--set` replaces the default user list, dropping its `username`. Or a user has neither a password nor a public key. + **Cause:** Setting a `users[0]` field alone with `--set` replaces the default user list, dropping its `username`. Or a user has `passwordLogin: false` and no public key. - **Fix:** Pass `--set 'users[0].username=...'` alongside the password, or define `users` in a values file with a `password` or `publicKeys` for every entry. + **Fix:** Pass `--set 'users[0].username=...'` alongside the other field, or define `users` in a values file; give every user with `passwordLogin: false` at least one entry in `publicKeys`. **Symptom:** The upgrade stops at render time naming `admin.username` or `admin.password`, and no revision is created. @@ -468,6 +518,13 @@ The template runs **one SFTPGo replica** and exposes no replica count. A restart **Fix:** Follow the steps under Upgrading from 1.0.0 above. + + **Symptom:** The upgrade stops at render time with `sftpgo: users[].password was removed in 1.2.0`, `sftpgo: storage.minio.accessKey was removed in 1.2.0` or the `accessSecret` equivalent. + + **Cause:** Your values still carry a plaintext credential that `1.2.0` moved into a prerequisite secret. + + **Fix:** Create the secrets holding your current values and delete the keys, as described in [Upgrading from 1.1.0](#upgrading-from-1-1-0). + **Symptom:** The install or upgrade stops with `sftpgo: mode 'scale_to_zero' requires internalAccess.type other than 'none'`. @@ -500,8 +557,8 @@ The template runs **one SFTPGo replica** and exposes no replica count. A restart ## Important Notes -- **Create the admin secret before installing.** A missing secret wedges the deployment silently; `cpln workload get-deployments` names it. -- **Change or remove the default `users[].password`.** It is used exactly as written and stays a Helm value; prefer `publicKeys`. +- **Create the admin secret, the user passwords secret and, with `minio`, the S3-compatible credentials secret before installing.** A missing secret wedges the deployment silently; `cpln workload get-deployments` names it. +- **Rotating a secret needs a forced redeployment** — changing the user passwords or the S3-compatible credentials does nothing until `cpln workload force-redeployment RELEASE_NAME-sftpgo --gvc GVC_NAME`. - **The admin secret seeds the first boot only.** Change an existing administrator's password in the web admin or REST API. - **Declared users are authoritative.** Web admin edits to them are overwritten on the next restart or wake. - **`scale_to_zero` clients need generous timeouts.** Use 120-second timeouts or retries, or choose `always_warm`. diff --git a/template-catalog/templates/temporal.mdx b/template-catalog/templates/temporal.mdx index d7d25a96..7c150d7b 100644 --- a/template-catalog/templates/temporal.mdx +++ b/template-catalog/templates/temporal.mdx @@ -1,7 +1,7 @@ --- title: Temporal icon: /template-catalog/templates/icons/temporal.png -description: Deploy Temporal on Control Plane using the Template Catalog. An internal durable-execution server of one to five replicas, a Web UI you can open to listed IP ranges, optional S3 archival, and a bundled HA or single-instance PostgreSQL. Needs only an existing GVC. +description: Deploy Temporal on Control Plane using the Template Catalog. An internal durable-execution server of one to five replicas, a Web UI you can open to listed IP ranges, optional S3 archival, and a bundled HA or single-instance PostgreSQL. Needs an existing GVC and a database credentials secret. keywords: ['workflow orchestration', 'workflow engine', 'cadence', 'step functions alternative', 'conductor alternative', 'camunda alternative', 'background jobs', 'task queue', 'saga pattern', 'automatic retries', 'grpc', 'history shards', 'namespace retention', 'self-hosted'] --- @@ -20,7 +20,6 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-temporal` | Temporal server (standard, `replicas` replicas, default 1, no volume). Each replica runs a `temporal` container (frontend, history, matching and worker services in one process; gRPC on `7233`, HTTP API on `7243`) and an `admin` sidecar that migrates the schema and serves the `temporal` CLI | — | | Workload | `RELEASE_NAME-temporal-ui` | Temporal Web UI on port `8080` (standard, 1 replica) | `ui.enabled: false` removes it | | Identity & Policy | `RELEASE_NAME-temporal-identity`, `RELEASE_NAME-temporal-policy` | Server identity with `reveal` on exactly the database credentials secret. With archival on, the identity is also bound to your AWS cloud account and bucket-scoped IAM policy. The UI has no identity because it mounts no secrets | — | -| Secret | `my-temporal-db-credentials` | Dictionary secret **created by this chart** from `postgres.credentials.*`, holding `username`, `password` and `database`. Its name is the active database block's `config.credentialsSecretName` | — | | Workload | `RELEASE_NAME-postgres-ha` | Patroni PostgreSQL cluster (stateful, 3 replicas) | HA mode (default) | | Workload | `RELEASE_NAME-etcd` | etcd consensus store for Patroni (stateful, 3 replicas) | HA mode | | Workload | `RELEASE_NAME-postgres-ha-proxy` | HAProxy leader endpoint Temporal connects through (standard, 2 replicas) | HA mode | @@ -38,7 +37,28 @@ In HA mode with `postgresHA.backup.mode: wal-g`, no cron workload is created; a ## Prerequisites -None for a default install. The database credentials are **not** a prerequisite: this chart creates the credentials secret for you from `postgres.credentials.*` and hands it to the bundled database by name. You only have to change the password before installing. +**A database credentials secret must exist before you install.** It is a [dictionary secret](/guides/create-secret/dictionary) holding `username`, `password` and `database`; the bundled PostgreSQL initializes with it and Temporal connects with it. Secrets are org-level, so no GVC flag is involved: + +```bash +cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=temporal \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=temporal +``` + +- **`database` must be `temporal`.** The server reads its main store from the database named `temporal`, while PostgreSQL creates the one the secret names. With any other value both Temporal containers stop at boot and log `CPLN_TEMPORAL_DATABASE_GUARD`. +- Set its name in the key of the database mode you use: `postgresHA.config.credentialsSecretName` (HA, the default) or `postgres.config.credentialsSecretName` (single-instance). Secret names are org-wide, so give each Temporal release its own secret. +- PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password). + + +**If the secret does not exist at install time the deployment wedges silently.** `cpln logs` returns zero lines — the containers never start, so there is nothing to log. The only place the missing secret is named is `status.versions[].message`: + +```bash +cpln workload get-deployments RELEASE_NAME-temporal --gvc GVC_NAME -o yaml +``` + +Use `get-deployments`; plain `cpln workload get` has no `versions` key. Create the missing secret and the deployment recovers on its own after several minutes. + Optional database backups need a bucket and, for AWS or GCP, a Control Plane Cloud Account. Backups to MinIO or another S3-compatible endpoint additionally need a dictionary secret holding `accessKey` and `secretKey`, which must exist before you enable them. See [Backing Up](#backing-up). @@ -46,14 +66,16 @@ Optional archival needs an S3 bucket, an AWS Cloud Account and a bucket-scoped I ## Installation -Install with the database password set to something strong. Nothing else is required for a default install: +Install with the name of the database credentials secret you created. Nothing else is required for a default install: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/temporal \ - --version 2.0.0 --gvc GVC_NAME \ - --set postgres.credentials.password='YOUR-STRONG-PASSWORD' + --version 2.1.0 --gvc GVC_NAME \ + --set postgresHA.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME ``` +In single-instance mode set `postgres.config.credentialsSecretName` instead. + Browse, install, and manage templates visually @@ -227,8 +249,13 @@ Exactly one of `postgresHA` and `postgres` must be enabled; the chart refuses to postgresHA: enabled: true config: - # Name of the dictionary secret this chart creates (from postgres.credentials.*) - # and the bundled postgres-highly-available reads. Org-wide: unique per release. + # REQUIRED PREREQUISITE SECRET (HA mode) — CREATE IT BEFORE YOU INSTALL. If + # it does not exist the deployment WEDGES silently and looks like a platform + # fault. A `dictionary` secret holding exactly three keys: username, password, + # database — and `database` MUST be `temporal` (the server's main store; any + # other value stops the admin container at boot). PostgreSQL reads it once, when + # its data is first initialized, and Temporal connects with the same keys. See + # README Prerequisites for the command. credentialsSecretName: my-temporal-db-credentials replicas: 3 volumeset: @@ -265,7 +292,7 @@ postgresHA: prefix: postgres/backups ``` -- `postgresHA.config.credentialsSecretName` — Name of the dictionary secret this chart creates and the HA cluster reads. Its contents come from `postgres.credentials.*` below, in both database modes. Secret names are organization-wide, so give each Temporal release its own name: a second release left on the default is refused at install and creates nothing, and the first release is unaffected. +- `postgresHA.config.credentialsSecretName` — Name of the prerequisite database credentials secret the HA cluster and Temporal read (see [Prerequisites](#prerequisites)). Secret names are organization-wide, so give each Temporal release its own. - `postgresHA.replicas` / `postgresHA.volumeset.capacity` — PostgreSQL replica count and volume size per replica. The remaining cluster settings take the defaults of the [Postgres Highly Available](/template-catalog/templates/postgres-highly-available) template; the HAProxy leader endpoint must stay enabled because it is the address Temporal connects to. - `postgresHA.backup.*` — Optional backups of both Temporal databases, as scheduled logical dumps or continuous WAL-G archiving. Keep `resources.memory` at `512Mi` or more for WAL-G. See [Backing Up](#backing-up). @@ -276,17 +303,11 @@ Enable this block and disable `postgresHA` for a lighter non-HA deployment. ```yaml postgres: enabled: false - # Credentials for the bundled database, used in BOTH database modes: this chart - # creates the secret from them, so there is nothing to create before installing. - # The password is set once, when the database volume is initialised. - # `database` MUST stay `temporal` (render-enforced in both modes). - credentials: - username: temporal - password: change-me-temporal-db-password # change before installing - database: temporal config: - # Name of the dictionary secret this chart creates and the bundled postgres reads. - # Org-wide: unique per release (a second release on this name is refused at install). + # REQUIRED PREREQUISITE SECRET (single-instance mode) — the same kind of + # `dictionary` secret as postgresHA.config.credentialsSecretName above + # (keys: username, password, database; database MUST be `temporal`). + # Create it BEFORE you install. credentialsSecretName: my-temporal-db-credentials volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) @@ -318,12 +339,10 @@ postgres: prefix: postgres/backups ``` -- `postgres.credentials.username` / `postgres.credentials.password` — The bundled database's credentials. **Change the password before installing.** These two values populate the chart-created credentials secret in **both** database modes, even though they sit under the `postgres` block. -- `postgres.credentials.database` — Must remain `temporal`. The server reads its main store as the literal database `temporal`, so the chart refuses to render any other value. -- `postgres.config.credentialsSecretName` — Name of the credentials secret in single-instance mode (the HA counterpart is `postgresHA.config.credentialsSecretName`). +- `postgres.config.credentialsSecretName` — Name of the prerequisite credentials secret in single-instance mode (the HA counterpart is `postgresHA.config.credentialsSecretName`). Its `database` key must be `temporal`. -The credentials are applied once, when the database volume is first initialized. Changing `postgres.credentials.password` on an existing release only rewrites the secret, not the password PostgreSQL enforces, so the server can no longer connect. +The credentials are applied once, when the database volume is first initialized. Changing the secret on an existing release does not change the password PostgreSQL enforces, so the server can no longer connect. To rotate, follow [Rotating the Database Password](#rotating-the-database-password). ## Connecting @@ -339,7 +358,7 @@ The server exposes no public endpoint: everything except the browser UI is reach | `temporal` CLI | The `admin` container of `RELEASE_NAME-temporal` | Already pointed at the server; see the command below. | | PostgreSQL (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | The HAProxy leader endpoint Temporal itself uses. | | PostgreSQL (single mode) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Only when `postgres.enabled: true`. | -| Database credentials | `cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml` | `DB_CREDENTIALS_SECRET_NAME` is the active database block's `config.credentialsSecretName` (default `my-temporal-db-credentials`). | +| Database credentials | `cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml` | `DB_CREDENTIALS_SECRET_NAME` is your secret named by the active database block's `config.credentialsSecretName`. | Confirm the server is serving by listing workflows with the `temporal` CLI in the `admin` sidecar. On a fresh install the list is empty and the command exits without an error: @@ -571,16 +590,14 @@ For GCS replace the first line of the pipeline with `gsutil cp "gs://BUCKET_NAME Apply every subsection from your version onward, in order, ending with [Upgrading from 1.x](#upgrading-from-1-x). Each removed key is rejected at render, so an upgrade that still carries one fails before anything is applied and the running release is left untouched. -Version `1.1.0` moved the bundled single-instance database to the `postgres` 3.4.1 template, which no longer takes credentials or MinIO keys as values. This template absorbed the credentials change rather than passing it on: it creates the credentials secret itself, so there is **no new prerequisite for the database**, only renames on the single-instance path: +Version `1.1.0` moved the bundled single-instance database to the `postgres` 3.4.1 template, which no longer takes credentials or MinIO keys as values: | Removed key | Replacement | |---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` (must be `temporal`) | +| `postgres.config.username` / `.password` / `.database` | the database credentials secret under [Prerequisites](#prerequisites) (`database` must be `temporal`), named by `postgres.config.credentialsSecretName` | | `postgres.backup.minio.accessKey`, `postgres.backup.minio.secretKey` | `postgres.backup.minio.credentialsSecretName`, a dictionary secret you create (MinIO backups only, see [Backing Up](#backing-up)) | -Carrying an old credentials key forward fails with the **bundled Postgres template's** message, which begins `config.username was REMOVED in postgres 3.4.0` and tells you to create a dictionary secret yourself. Ignore that advice for the three credentials keys: this template creates that secret from `postgres.credentials.*`. Use the username and password your database was initialized with, not new ones. +Carrying an old credentials key forward fails with the **bundled Postgres template's** message, which begins `config.username was REMOVED in postgres 3.4.0` and tells you to create a dictionary secret — that is correct here too. Use the username and password your database was initialized with, not new ones. **Template version `1.0.0` did not compact the etcd cluster inside the bundled HA database**, so its backend grew with time alone and eventually went read-only, taking PostgreSQL failover with it. Only HA-mode installs are affected. Upgrading to `1.0.1` or later turns compaction on, which stops further growth but cannot shrink a backend that has already grown; a cluster that has already raised a `NOSPACE` alarm needs operator recovery rather than an upgrade. The mechanism and the inspection commands are on the [Postgres Highly Available](/template-catalog/templates/postgres-highly-available) page. @@ -588,20 +605,13 @@ Carrying an old credentials key forward fails with the **bundled Postgres templa ### Upgrading from 1.1.0 -Version `1.2.0` moved the bundled HA database to `postgres-highly-available` 2.5.0, which removed the `postgresHA.postgres` block. The HA cluster now reads the same chart-created credentials secret as the single-instance path: - -| Removed key | Replacement | -|---|---| -| `postgresHA.postgres.username` | `postgres.credentials.username` | -| `postgresHA.postgres.password` | `postgres.credentials.password` | -| `postgresHA.postgres.database` | `postgres.credentials.database` (must be `temporal`) | -| — | `postgresHA.config.credentialsSecretName` names the secret this chart creates (default `my-temporal-db-credentials`) | +Version `1.2.0` moved the bundled HA database to `postgres-highly-available` 2.5.0, which removed the `postgresHA.postgres` block. The HA cluster now reads the database credentials secret under [Prerequisites](#prerequisites) (`database` must be `temporal`), named by `postgresHA.config.credentialsSecretName`. -**Use the credentials your cluster already has.** The HA cluster was initialized with the `postgresHA.postgres.*` values you installed with, and PostgreSQL still enforces them. Copy that username and password into `postgres.credentials.username` and `postgres.credentials.password` before upgrading. Putting new values there updates the secret but not the database, and Temporal can no longer authenticate. +**Use the credentials your cluster already has.** The HA cluster was initialized with the `postgresHA.postgres.*` values you installed with, and PostgreSQL still enforces them. Put that username and password into the credentials secret before upgrading. Putting new values there does not change the database, and Temporal can no longer authenticate. -An upgrade that still carries the `postgresHA.postgres` block fails with the HA template's message, which begins ``postgres-highly-available: the `postgres` block was REMOVED in 2.5.0`` and tells you to create a dictionary secret. Ignore that part: delete the block and set `postgres.credentials.*` instead. +An upgrade that still carries the `postgresHA.postgres` block fails with the HA template's message, which begins ``postgres-highly-available: the `postgres` block was REMOVED in 2.5.0`` and tells you to create a dictionary secret — that is correct here: delete the block and set `postgresHA.config.credentialsSecretName` to the secret's name. ### Upgrading from 1.2.x @@ -656,6 +666,9 @@ Temporal supports upgrading **one minor version at a time**. Every 1.x release o Run the same command without the two `--set` values; the chart defaults are 1.32.1. + + Follow [Upgrading from 2.0.0](#upgrading-from-2-0-0). + Each step migrates the schema before the new server starts. A step that skips a minor version is refused before the database is touched: the new replica never becomes ready while the old one keeps serving, and its `admin` container logs `CPLN_TEMPORAL_SKIP_GUARD` with the tag to use next. Set that tag and upgrade again. @@ -666,6 +679,68 @@ Also in this version: - **Temporal 1.32 visibility queries are stricter:** a list filter that compares a search attribute with the wrong type, or an empty `Text` filter, now returns an error instead of results. - **New optional features:** [archival](#archival) to S3, which needs Temporal 1.31 or later, so turn it on after the step to 1.31; and browser access to the Web UI from listed IP ranges with `ui.allowedCIDRs` (see [Web UI](#web-ui)). +### Upgrading from 2.0.0 + +Version `2.1.0` removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the database templates require on their own. Carrying the old key forward fails the render. + +2.0.0 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Temporal loses its database connection. `database` stays `temporal`: + +```bash +cpln secret reveal SECRET_NAME -o yaml # the 2.0.0 secret: its CURRENT values, until the upgrade deletes it +cpln secret create-dictionary --name RELEASE_NAME-postgres-credentials \ + --entry username='YOUR_CURRENT_USERNAME' --entry password='YOUR_CURRENT_PASSWORD' --entry database=temporal +``` + +`SECRET_NAME` is the `credentialsSecretName` of the mode you use in the 2.0.0 release (default `my-temporal-db-credentials`). Then upgrade with `postgresHA.config.credentialsSecretName` (HA mode) or `postgres.config.credentialsSecretName` (single-instance mode) set to the new name, and `postgres.credentials` removed. Your current values are also in `postgres.credentials` of the 2.0.0 release. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Temporal can briefly lose its database; it reconnects by itself. While access to the new secret propagates, the new version of a workload can wait in `get-deployments` reporting that it is not allowed to reveal the secret; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating the Database Password + +A running workload keeps the credential values it started with, and PostgreSQL reads the secret only when its data is first initialized, so editing the secret alone does not change the database password. Change one key and keep the others with export, edit, apply — keep `database` at `temporal`: + +```bash +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only the key you are rotating in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +``` + +**Single-instance mode** takes three steps in order: + +1. Change it in the database: `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` (connected as that user, from inside the GVC). +2. Update the secret as above. +3. Right away, redeploy Temporal, and wait until `get-deployments` shows a higher version number, alone and ready (right after the command the status may not show the restart yet, and `ready: true` alone can still describe the old replica). Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. + +```bash +cpln workload force-redeployment RELEASE_NAME-temporal --gvc GVC_NAME +``` + +Temporal keeps serving on the database connections it already has open. Do step 3 right away: if those connections drop before Temporal restarts, it cannot open new ones. + +**HA mode** (the default) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3. Back to back: change BOTH roles on -0, put the same password in the secret (export, edit, apply above), restart the members +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 4. Once the members are back, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 5. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-temporal`, which keeps the old password until it restarts. The app cannot open new database connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming`, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + ### Scaling and Availability - **`replicas` sets the server count, from 1 to 5.** Every replica runs the full server and they form one Temporal cluster. `3` survives the loss of a replica with capacity to spare; `1` is fine when a short gap per upgrade or restart is acceptable. @@ -676,7 +751,7 @@ Also in this version: - **Losing a replica:** the remaining replicas take over its workflows and the platform replaces it. Timers fire on time and nothing is lost; clients may see a few timed-out calls right after the loss. - **Workers scale independently.** Run as many worker workloads and replicas as your task queues need. - **Database availability is the HA cluster's job.** In the default `postgresHA` mode a PostgreSQL leader failure is handled by Patroni and the HAProxy endpoint keeps the same address. In `postgres` mode there is no failover. -- **Durability is in the database, not the server.** The server workload has no volume. `cpln helm uninstall` deletes the database volume sets and the chart-created credentials secret, taking every workflow history with it. +- **Durability is in the database, not the server.** The server workload has no volume. `cpln helm uninstall` deletes the database volume sets, taking every workflow history with it. ### Troubleshooting @@ -729,12 +804,19 @@ Also in this version: **Fix:** Set exactly one of `postgresHA.enabled` and `postgres.enabled` to `true`. Switching an existing release between modes is a new database, not a migration; the data stays on the old mode's volume set. - - **Symptom:** Render fails with `temporal: postgres.credentials.database must be "temporal"`. + + **Symptom:** The server never becomes ready, and its `admin` container logs `CPLN_TEMPORAL_DATABASE_GUARD`. + + **Cause:** The database credentials secret's `database` key is not `temporal`. The server reads its main store as the literal database `temporal`. + + **Fix:** Set `database` to `temporal` in the secret, then force a redeployment of `RELEASE_NAME-temporal`. The database PostgreSQL created under the wrong name is unused and can be dropped. + + + **Symptom:** `helm upgrade` stops at render with `temporal: postgres.credentials was REMOVED in 2.1.0`. - **Cause:** The server reads its main store as the literal database `temporal`, so the chart refuses any other value. + **Cause:** Your values still carry the 2.0.0 `postgres.credentials` block. - **Fix:** Leave `postgres.credentials.database` at `temporal`. + **Fix:** Follow [Upgrading from 2.0.0](#upgrading-from-2-0-0). **Symptom:** `helm upgrade` stops at render with `temporal: resources.cpu was RENAMED to resources.maxCpu` (or the same message for `resources.memory`, `ui.resources.cpu`, `ui.resources.memory`). @@ -748,22 +830,16 @@ Also in this version: **Cause:** Your values still carry `postgresHA.postgres.username`, `postgresHA.postgres.password` or `postgresHA.postgres.database` from template version `1.1.0` or earlier. - **Fix:** Delete the `postgresHA.postgres` block and copy the same username and password into `postgres.credentials.*`. Do not create a secret yourself; this chart creates it. See [Upgrading from 1.1.0](#upgrading-from-1-1-0). + **Fix:** Delete the `postgresHA.postgres` block and put the same username and password into the database credentials secret. See [Upgrading from 1.1.0](#upgrading-from-1-1-0). **Symptom:** `helm upgrade` stops at render with `config.username was REMOVED in postgres 3.4.0` (or the same for `config.password` / `config.database`). **Cause:** Your single-instance values still use `postgres.config.username`, `postgres.config.password` or `postgres.config.database` from template version `1.0.x`. - **Fix:** Move the three values to `postgres.credentials.*`. Do not create a secret yourself. See [Upgrading from 1.0.x](#upgrading-from-1-0-x). + **Fix:** Put the three values into the database credentials secret. See [Upgrading from 1.0.x](#upgrading-from-1-0-x). - - **Symptom:** Installing a second Temporal release in the same org fails with `The resource 'my-temporal-db-credentials' cannot be updated because it is being managed by a different release`, and nothing is created. - - **Cause:** Secret names are organization-wide, and both releases are trying to create the credentials secret under the default name. - **Fix:** Give the new release its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName` in single mode). The first release is unaffected. - **Symptom:** Render fails with `temporal: archival needs Temporal server 1.31 or later, and this release runs 1.30.x`. @@ -817,15 +893,16 @@ Also in this version: ## Important Notes -- **Change `postgres.credentials.password` before installing.** The default `change-me-temporal-db-password` is published in the template repository, and it is the database password in both database modes. It is set once, when the database volume is first initialized. +- **Create the database credentials secret before installing**, with `database` set to `temporal`. A missing secret wedges the deployment with zero log lines; `status.versions[].message` in `cpln workload get-deployments RELEASE_NAME-temporal --gvc GVC_NAME -o yaml` names it. +- **Editing the credentials secret does not change the database password** — run `ALTER ROLE` first, then force-redeploy Temporal; in HA mode move the leader first ([Rotating the Database Password](#rotating-the-database-password)). - **Upgrade Temporal one minor version at a time**, and back up through the database backup first. A skipped minor is refused; see [Upgrading from 1.x](#upgrading-from-1-x). - **`historyShards` is fixed at first install.** A later change is ignored, so size it before installing. - **The Web UI has no login.** List only addresses you trust in `ui.allowedCIDRs`; ranges broader than `/8` (IPv4) or `/32` (IPv6) are refused, and a change takes up to a few minutes to apply. - **To stop archiving, set `archival.defaultNamespace: false`.** Never set `archival.enabled: false` once archival has been used. - **Use the full `.cpln.local` hostname** in every worker and client; short workload names do not resolve. -- **Give each release its own `config.credentialsSecretName`.** Secret names are organization-wide; a second release on the default name is refused at install. +- **Give each release its own database credentials secret.** Secret names are organization-wide, so two releases pointed at one secret share a password. - **Keep the GVC to one location.** Each location would run its own servers against its own database. -- **Uninstall deletes the database volume sets** and the chart-created credentials secret, taking every workflow history with it. Enable backups if the data matters. +- **Uninstall deletes the database volume sets**, taking every workflow history with it. Enable backups if the data matters. ## External References diff --git a/template-catalog/templates/tooljet.mdx b/template-catalog/templates/tooljet.mdx index d4e3939f..ed769aa6 100644 --- a/template-catalog/templates/tooljet.mdx +++ b/template-catalog/templates/tooljet.mdx @@ -1,7 +1,7 @@ --- title: ToolJet icon: /template-catalog/templates/icons/tooljet.png -description: Deploy ToolJet, the open-source low-code platform for internal tools, on Control Plane. Single server that scales out with Redis, with bundled PostgreSQL for the app and the ToolJet Database. Needs an existing GVC and a pre-created dictionary secret holding its three keys. +description: Deploy ToolJet, the open-source low-code platform for internal tools, on Control Plane. Single server that scales out with Redis, with bundled PostgreSQL for the app and the ToolJet Database. Needs an existing GVC and pre-created key and database credentials secrets. keywords: ['tooljet', 'low-code', 'no-code', 'internal tools', 'admin panel', 'app builder', 'retool alternative', 'appsmith alternative', 'budibase alternative', 'crud dashboard', 'postgrest', 'self-hosted', 'agpl'] --- @@ -23,24 +23,33 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-redis` | Redis data node from the [redis](/template-catalog/templates/redis) template — the shared job queue and multiplayer coordination for more than one ToolJet replica (1 replica) | `redis.enabled: true` | | Workload | `RELEASE_NAME-sentinel` | Redis Sentinel, monitor-only (1 replica) | `redis.enabled: true` | | Volume set | `RELEASE_NAME-pg-vs` | PostgreSQL data directory | — | -| Secret | `RELEASE_NAME-tooljet-db` | Dictionary secret the chart creates with the database username and password ToolJet uses for its `PG_*`, `TOOLJET_DB_*`, and `PGRST_DB_URI` settings | — | -| Secret | the name in `postgres.config.credentialsSecretName` | Dictionary secret the chart creates (`username`, `password`, `database`) and hands to the bundled PostgreSQL | — | | Secret | `RELEASE_NAME-tooljet-startup` | Startup script that waits for PostgreSQL to accept connections before handing off to the image entrypoint | — | -| Secret | `RELEASE_NAME-redis-auth-password` | Redis password, wired into ToolJet as `REDIS_PASSWORD` | `redis.redis.auth.password.enabled: true` | | Secret | `RELEASE_NAME-redis-config`, `RELEASE_NAME-sentinel-config` | Redis and Sentinel configuration files | `redis.enabled: true` | | Identity | `RELEASE_NAME-tooljet-identity` | Bound to the ToolJet workload | — | -| Policy | `RELEASE_NAME-tooljet-policy` | Grants that identity `reveal` on exactly the secrets it reads: the two chart-created ToolJet secrets, your key secret, and the SMTP and Redis password secrets when configured | — | +| Policy | `RELEASE_NAME-tooljet-policy` | Grants that identity `reveal` on exactly the secrets it reads: the chart's startup script, your key secret and database credentials secret, and the SMTP and Redis password secrets when configured | — | | Identity | `RELEASE_NAME-pg-identity` | Bound to the bundled PostgreSQL and its backup job; carries your cloud account link when backups are on | — | | Policy | `RELEASE_NAME-pg-policy` | Grants the PostgreSQL identity `reveal` on its credentials secret only | — | | Identity / Policy | `RELEASE_NAME-redis-identity`, `RELEASE_NAME-redis-policy`, `RELEASE_NAME-sentinel-identity`, `RELEASE_NAME-sentinel-policy` | Identities and secret policies for the Redis and Sentinel workloads | `redis.enabled: true` | -Your key secret is **not** created by the template — see [Prerequisites](#prerequisites). +Your prerequisite secrets are **not** created by the template — see [Prerequisites](#prerequisites). ## Prerequisites -ToolJet reads its three root-of-trust keys from a [dictionary secret](/guides/create-secret/dictionary) that you create **before** installing. The key values never pass through Helm values. +Two [dictionary secrets](/guides/create-secret/dictionary) must exist **before** installing: the bundled database's credentials, and ToolJet's three root-of-trust keys. Their values never pass through Helm values. + + The bundled PostgreSQL initializes with these and ToolJet connects with them (it creates its `tooljet_db` database alongside): + + ```bash + cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \ + --entry username=tooljet \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=tooljet + ``` + + Set `postgres.config.credentialsSecretName` to this name. Keep the password URL-safe (letters, digits, `-`, `_`) — ToolJet places it in a connection URL as-is. PostgreSQL reads the secret only when its data directory is first initialized; see [Rotating the Database Password](#rotating-the-database-password) for changing it later. + Generate the three keys the same way ToolJet's own setup scripts do, straight into the secret: @@ -64,8 +73,15 @@ ToolJet reads its three root-of-trust keys from a [dictionary secret](/guides/cr +Only if you turn Redis auth on, create one more dictionary secret with the key `password` and name it in `redis.redis.auth.fromSecret.name` (with `redis.redis.auth.fromSecret.enabled: true`). Redis, its Sentinel and ToolJet all read it: + +```bash +cpln secret create-dictionary --name REDIS_SECRET_NAME \ + --entry password="$(openssl rand -hex 24)" +``` + -Create the secret **before** installing. Installing without it reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message`: +Create the secrets **before** installing. Installing without one reports success, but the deployment then wedges silently — `cpln logs` returns nothing, and the only place the missing secret is named is `status.versions[].message` (also check `RELEASE_NAME-postgres` for the database secret and `RELEASE_NAME-redis` for the Redis secret): ```bash cpln workload get-deployments RELEASE_NAME-tooljet --gvc GVC_NAME -o yaml @@ -81,16 +97,16 @@ Everything else works with the defaults. Two optional features need their own se ## Installation -Once your key secret exists, install from the marketplace registry, pointing `secrets.name` at it and replacing the bundled database password: +Once your key and database credentials secrets exist, install from the marketplace registry, pointing the chart at them: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/tooljet \ - --version 1.2.0 --gvc GVC_NAME \ + --version 1.3.0 --gvc GVC_NAME \ --set secrets.name=KEYS_SECRET_NAME \ - --set postgres.credentials.password='YOUR-DB-PASSWORD' + --set postgres.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME ``` -Installing a second ToolJet release in the same org? Also set `postgres.config.credentialsSecretName` to a name no other release uses — secret names are org-wide. If you want email invitations, add the `smtp.*` values to this first install as well (see [Email](#email)). +Installing a second ToolJet release in the same org? Give it its own database credentials secret — secret names are org-wide. If you want email invitations, add the `smtp.*` values to this first install as well (see [Email](#email)). Other ways to install and manage the template: @@ -197,12 +213,8 @@ With `publicAccess.enabled: false` there is no public endpoint; external request ```yaml postgres: image: postgres:16 # upstream-proven with the bundled PostgREST 12.2.0 - credentials: # the chart writes these into the secret named below — nothing for you to create - username: tooljet - password: change-me-tooljet-pg # change before installing — seeds the database on first boot - database: tooljet # application database; tooljet_db is auto-created alongside it config: - credentialsSecretName: my-tooljet-db-credentials # secret this chart CREATES; names are org-wide, so one per release + credentialsSecretName: my-tooljet-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install resources: minCpu: 250m maxCpu: 1000m @@ -212,7 +224,7 @@ postgres: capacity: 10 # GiB (minimum 10) ``` -One PostgreSQL instance backs the whole deployment, and ToolJet's boot scripts create every database it needs using the credentials above: +One PostgreSQL instance backs the whole deployment, and ToolJet's boot scripts create every database it needs using the credentials from that secret: | Database | Contents | |---|---| @@ -220,7 +232,7 @@ One PostgreSQL instance backs the whole deployment, and ToolJet's boot scripts c | `tooljet_db` | The ToolJet Database — user tables created in the UI, served through the in-image PostgREST. | | `sample_db` | Sample data created by ToolJet's boot scripts. | -`postgres.credentials.password` seeds the database on first boot and is not updated by later value edits — set it before you install. It is bundled plumbing, so the chart creates the credentials secret for you; it is not a prerequisite. Scheduled backups of this database are configured under [Backing Up](#backing-up). +The credentials secret seeds the database on first boot; editing it later does not change the database password (see [Rotating the Database Password](#rotating-the-database-password)). Scheduled backups of this database are configured under [Backing Up](#backing-up). The ToolJet Database is a core part of ToolJet 3.x and cannot be turned off. @@ -235,9 +247,9 @@ redis: image: redis:7.4 replicas: 1 # keep at 1 — ToolJet is not Sentinel-aware auth: - password: - enabled: false # true = require AUTH; the password is wired into ToolJet as REDIS_PASSWORD - value: change-me-tooljet-redis + fromSecret: + enabled: false # true = require AUTH (Redis, Sentinel and ToolJet read the secret below) + name: "" # prerequisite dictionary secret (key: password); used only when enabled resources: cpu: 200m memory: 256Mi @@ -253,7 +265,7 @@ redis: minMemory: 128Mi ``` -`redis.enabled: true` deploys the [redis](/template-catalog/templates/redis) template and points ToolJet at it; it is required for more than one ToolJet replica (see [Scaling and Availability](#scaling-and-availability)). With `redis.redis.auth.password.enabled: true` the password is stored in the chart-created `RELEASE_NAME-redis-auth-password` secret, which the ToolJet identity is granted `reveal` on. +`redis.enabled: true` deploys the [redis](/template-catalog/templates/redis) template and points ToolJet at it; it is required for more than one ToolJet replica (see [Scaling and Availability](#scaling-and-availability)). With `redis.redis.auth.fromSecret.enabled: true` Redis, Sentinel and ToolJet read the password from the prerequisite secret named by `redis.redis.auth.fromSecret.name` (see [Prerequisites](#prerequisites)), which the ToolJet identity is granted `reveal` on. Keep `redis.redis.replicas` and `redis.sentinel.replicas` at `1`. ToolJet's Redis client is not Sentinel-aware and connects straight to the service DNS name, so additional data nodes would spread writes onto read-only replicas. The chart refuses to render any other value. @@ -268,7 +280,7 @@ Keep `redis.redis.replicas` and `redis.sentinel.replicas` at `1`. ToolJet's Redi | Internal (same GVC) | `http://RELEASE_NAME-tooljet.GVC_NAME.cpln.local:3000` | Reachable per `internalAccess.type`. | | Login | The Super Admin account created during first-run onboarding | ToolJet ships no default account. | | PostgreSQL | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | `username` / `password` / `database` from `cpln secret reveal SECRET_NAME -o yaml`, where `SECRET_NAME` is your `postgres.config.credentialsSecretName`. | -| Redis | `RELEASE_NAME-redis.GVC_NAME.cpln.local:6379` | Only with `redis.enabled: true`. | +| Redis | `RELEASE_NAME-redis.GVC_NAME.cpln.local:6379` | Only with `redis.enabled: true`. With Redis auth on, the password is the `password` key of your `redis.redis.auth.fromSecret.name` secret. | With `publicAccess.enabled: false` there is no public endpoint; reach the UI from your machine through a tunnel instead and open `http://localhost:3000`: @@ -341,24 +353,84 @@ This restore path has not been exercised against this template end to end. Rehea ### Upgrading from 1.0.0 -Version 1.1.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. ToolJet absorbed that change rather than passing it on — the chart now creates the credentials secret itself — so there is **no new prerequisite**, only a rename of three keys: - -| Removed key (1.0.0) | Replacement (1.1.0 and later) | -|---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | +Version 1.1.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. `postgres.config.username`, `.password` and `.database` were removed: put the values your database was initialized with into the database credentials secret under [Prerequisites](#prerequisites) (keys `username`, `password`, `database`), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. The key secret is unchanged. -Move the three keys, keep the same values so the running database still matches, and upgrade in place. A values file that still carries an old key fails the render with the **postgres template's** message — `config.username was REMOVED in postgres 3.4.0` — which tells you to create a dictionary secret yourself; ignore that advice here, this template creates it. Then continue with the section below. +Keep the same values so the running database still matches. A values file that still carries an old key fails the render with the **postgres template's** message — `config.username was REMOVED in postgres 3.4.0` — which tells you to create a dictionary secret; that is correct here too. Then continue with the sections below. ### Upgrading from 1.1.0 -Version 1.2.0 changes no values. The ToolJet container now starts through a chart-provided script that waits until PostgreSQL actually accepts connections before running the image's own entrypoint, which removes a failed first attempt that earlier versions logged on a fresh install. Upgrade in place. +Version 1.2.0 changes no values. The ToolJet container now starts through a chart-provided script that waits until PostgreSQL actually accepts connections before running the image's own entrypoint, which removes a failed first attempt that earlier versions logged on a fresh install. Upgrade in place, then continue with [Upgrading from 1.2.0](#upgrading-from-1-2-0). The first `helm upgrade` after an install can restart the bundled PostgreSQL even when nothing about it changed, which interrupts ToolJet briefly; later upgrades of the same release do not. +### Upgrading from 1.2.0 + +Version 1.3.0 removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the `postgres` template requires on its own. Carrying the old key forward fails the render. + +1.2.0 created the database secret itself (and a second copy, `RELEASE_NAME-tooljet-db`), so **this upgrade deletes both**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or ToolJet loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-postgres-credentials \ + --entry username='YOUR_CURRENT_USERNAME' --entry password='YOUR_CURRENT_PASSWORD' --entry database='YOUR_CURRENT_DATABASE' +``` + +Then upgrade with `postgres.config.credentialsSecretName` set to that name and `postgres.credentials` removed. Your current values are in `postgres.credentials` of the 1.2.0 release, and also in the old secret named by its `postgres.config.credentialsSecretName` (`cpln secret reveal SECRET_NAME -o yaml`) until the upgrade deletes it. + +**If you ran 1.2.0 with Redis auth on**, `redis.redis.auth.password` was removed too (carrying it forward fails the render), and the chart-created `RELEASE_NAME-redis-auth-password` secret is deleted by the upgrade. Copy your current Redis password into a secret under a **new** name first: + +```bash +cpln secret reveal RELEASE_NAME-redis-auth-password -o yaml +cpln secret create-dictionary --name RELEASE_NAME-tooljet-redis \ + --entry password='YOUR_CURRENT_REDIS_PASSWORD' +``` + +Then also set `redis.redis.auth.fromSecret.enabled: true` and `redis.redis.auth.fromSecret.name` to the new name, and remove `redis.redis.auth.password`. With Redis auth off in 1.2.0 (the default), just remove `redis.redis.auth.password` from your values. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so ToolJet briefly loses its database; it reconnects by itself. While access to the new secret propagates, the new version of a workload can wait in `get-deployments` reporting that it is not allowed to reveal the secret; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating the Database Password + +Editing the secret alone does nothing to the database, and a running workload keeps the values it started with. Change one key and keep the others with export, edit, apply: + +```bash +cpln secret reveal SECRET_NAME -o yaml > secret.yaml +# edit only the key you are rotating in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal SECRET_NAME -o yaml +``` + +Run `ALTER ROLE` for the secret's `username` in PostgreSQL first (URL-safe characters only), then update the secret, then right away force a redeployment of ToolJet, and wait until `get-deployments` shows a higher version number, alone and ready (right after the command the status may not show the restart yet, and `ready: true` alone can still describe the old replica). Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. Connections the app already has open keep working after `ALTER ROLE`; new ones fail until its redeploy finishes, so redeploy right away and rotate in a quiet window. + +```bash +cpln workload force-redeployment RELEASE_NAME-tooljet --gvc GVC_NAME +``` + +### Rotating the Redis Password + +Only with Redis auth on. Redis keeps no password on disk, but a running Sentinel holds the one it started with, so a new password is applied live in three places before ToolJet restarts. Each Redis node first accepts both passwords, which keeps replication and every connected client working throughout. `rx` and `sx` run a command on one Redis node or Sentinel (`LOCATION` is your GVC's location; nodes and Sentinels are numbered from 0 up to `redis.redis.replicas` and `redis.sentinel.replicas` minus one): + +```bash +rx() { cpln workload exec RELEASE_NAME-redis --gvc GVC_NAME --location LOCATION \ + --container redis --replica "RELEASE_NAME-redis-$1" -- "${@:2}"; } +sx() { cpln workload exec RELEASE_NAME-sentinel --gvc GVC_NAME --location LOCATION \ + --container sentinel --replica "RELEASE_NAME-sentinel-$1" -- "${@:2}"; } +# 1. On EVERY Redis node: accept the new password alongside the old one, and replicate with it +rx 0 redis-cli --no-auth-warning -a OLD_PASSWORD ACL SETUSER default ">NEW_PASSWORD" +rx 0 redis-cli --no-auth-warning -a OLD_PASSWORD CONFIG SET masterauth NEW_PASSWORD +# 2. On EVERY Sentinel +sx 0 redis-cli -p 26379 SENTINEL SET mymaster auth-pass NEW_PASSWORD +# 3. Put the new password in the secret (export, edit, apply above) and read it back +cpln secret reveal SECRET_NAME -o yaml +# 4. Restart ToolJet, which keeps the old password until it restarts +cpln workload force-redeployment RELEASE_NAME-tooljet --gvc GVC_NAME +# 5. Once get-deployments shows a higher version, alone and ready: retire the old password on EVERY Redis node +rx 0 redis-cli --no-auth-warning -a NEW_PASSWORD ACL SETUSER default " **Symptom:** `cpln helm install` reported success, `cpln logs` returns nothing, and `cpln workload get-deployments RELEASE_NAME-tooljet --gvc GVC_NAME -o yaml` shows `The secret KEYS_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` under `status.versions[].message`. - **Cause:** The key secret named by `secrets.name` does not exist. The same applies to an `smtp.auth.secretName` that was set but never created. + **Cause:** The key secret named by `secrets.name` or the database credentials secret named by `postgres.config.credentialsSecretName` does not exist. The same applies to an `smtp.auth.secretName` or `redis.redis.auth.fromSecret.name` that was set but never created. Check `RELEASE_NAME-postgres` and `RELEASE_NAME-redis` the same way. **Fix:** Create it (see [Prerequisites](#prerequisites)). The deployment recovers on its own once the secret exists; `cpln workload force-redeployment RELEASE_NAME-tooljet --gvc GVC_NAME` skips the wait. @@ -403,13 +475,7 @@ The chart refuses to render `tooljet.replicas` of `2` or more while `redis.enabl **Fix:** Set both back to `1`. ToolJet connects to the Redis service name directly, so extra data nodes would receive writes they cannot accept. - - **Symptom:** Installing a second ToolJet release fails with `cannot be updated because it is being managed by a different release` and creates nothing. - **Cause:** Both releases use the same `postgres.config.credentialsSecretName`. Secret names are org-wide, and the first release owns that one. Nothing is shared, overwritten, or deleted. - - **Fix:** Set `postgres.config.credentialsSecretName` to a distinct name for the second release and install again. - **Symptom:** `smtp.enabled: true` was applied with `helm upgrade`, the `SMTP_*` variables are on the workload, yet invitations and password resets send nothing and the instance still reports mail as disabled. @@ -422,7 +488,14 @@ The chart refuses to render `tooljet.replicas` of `2` or more while `redis.enabl **Cause:** Your values file still carries `postgres.config.username`, `postgres.config.password`, or `postgres.config.database`. - **Fix:** Rename them to `postgres.credentials.*` as described in [Upgrading from 1.0.0](#upgrading-from-1-0-0). Do not create a secret yourself — this template creates it. + **Fix:** Move their values into the database credentials secret as described in [Upgrading from 1.0.0](#upgrading-from-1-0-0). + + + **Symptom:** The render fails with `postgres.credentials was removed in 1.3.0` or `redis.redis.auth.password was removed in tooljet 1.3.0`. + + **Cause:** Your values file still carries the 1.2.0 database or Redis password keys. + + **Fix:** Follow [Upgrading from 1.2.0](#upgrading-from-1-2-0). **Symptom:** The ToolJet logs repeat `Invalid License Key:Parse error`, and `/api/health` reports the license as invalid or expired. @@ -435,15 +508,15 @@ The chart refuses to render `tooljet.replicas` of `2` or more while `redis.enabl ## Important Notes -- **Create the key secret before installing** — `secrets.name` must name an existing dictionary secret with `SECRET_KEY_BASE`, `LOCKBOX_MASTER_KEY`, and `PGRST_JWT_SECRET`. A missing secret wedges the deployment silently. +- **Create every prerequisite secret before installing** — the key secret (`secrets.name`, with `SECRET_KEY_BASE`, `LOCKBOX_MASTER_KEY`, and `PGRST_JWT_SECRET`), the database credentials secret (`postgres.config.credentialsSecretName`) and, with Redis auth on, the Redis password secret. A missing secret wedges the deployment silently; `status.versions[].message` in `cpln workload get-deployments RELEASE_NAME-tooljet --gvc GVC_NAME -o yaml` names it. - **`SECRET_KEY_BASE` and `LOCKBOX_MASTER_KEY` are write-once** — rotating `LOCKBOX_MASTER_KEY` makes every stored datasource credential undecryptable. Back the keys up outside Control Plane. - **Do not lower `tooljet.resources.maxMemory` below `4Gi`** — the first boot's migrations run out of memory below it and the workload never becomes ready. - **Claim the Super Admin account right after install** — the first visitor to complete onboarding owns the instance, and the endpoint is public by default. - **Configure SMTP at the initial install** — enabling it through a later upgrade does nothing. - **`tooljet.replicas` of `2` or more requires `redis.enabled: true`**, and both the Redis data node and the Sentinel must stay at one replica. -- **Change `postgres.credentials.password` before installing** — it seeds the database on first boot and is not updated by later value edits. -- **Give each release its own `postgres.config.credentialsSecretName`** — secret names are org-wide, and a second release on the same name is refused at install. -- **Uninstalling deletes the PostgreSQL volume set** and every app, user, and ToolJet Database table in it; your key secret is yours and survives. +- **Changing the database password is a database change first** — run `ALTER ROLE`, update the secret, then force-redeploy ToolJet ([Rotating the Database Password](#rotating-the-database-password)). +- **Give each release its own database credentials secret** — secret names are org-wide, so two releases pointed at one secret share a password. +- **Uninstalling deletes the PostgreSQL volume set** and every app, user, and ToolJet Database table in it; your prerequisite secrets are yours and survive. - **ToolJet is licensed under the AGPL** — the deployed image is the LTS build and runs the free tier without a license key. ## External References diff --git a/template-catalog/templates/twenty.mdx b/template-catalog/templates/twenty.mdx index 88173752..a2edf88f 100644 --- a/template-catalog/templates/twenty.mdx +++ b/template-catalog/templates/twenty.mdx @@ -1,7 +1,7 @@ --- title: Twenty icon: /template-catalog/templates/icons/twenty.png -description: Deploy Twenty, the open-source CRM with customizable objects, pipelines and views, a background worker, a bundled single-node Redis and PostgreSQL that is single-instance by default or highly available with one flag. Needs an existing GVC and an opaque secret holding the app key. +description: Deploy Twenty, the open-source CRM with customizable objects, pipelines and views, a background worker, a bundled single-node Redis and PostgreSQL that is single-instance by default or highly available with one flag. Needs an existing GVC plus app key, database and Redis secrets. keywords: ['salesforce alternative', 'hubspot alternative', 'pipedrive alternative', 'attio alternative', 'contact management', 'sales pipeline', 'customer records', 'graphql api', 'rest api', 'bullmq', 'workflows', 'agpl', 'twenty template'] --- @@ -30,8 +30,6 @@ This template does not create a GVC. It deploys into an existing [GVC](/referenc | Volume Set | `RELEASE_NAME-twenty-redis-vs` | Redis append-only file at `/data` | — | | Volume Set | `RELEASE_NAME-pg-vs` | Single-instance database data | `postgres.enabled: true` (default) | | Volume Set | `RELEASE_NAME-postgres-ha-vs`, `RELEASE_NAME-etcd-vs` | Database and etcd data, one volume per replica | `postgresHA.enabled: true` | -| Secret | `my-twenty-db-credentials` | Dictionary secret with the database `username`, `password` and `database`, built by this template from `postgres.credentials.*`; the name comes from `postgres.config.credentialsSecretName` or `postgresHA.config.credentialsSecretName` | — | -| Secret | `RELEASE_NAME-twenty-creds` | Dictionary secret holding the bundled Redis password | — | | Secret | `RELEASE_NAME-twenty-server-start`, `RELEASE_NAME-twenty-worker-start` | Start scripts for the server and worker containers | — | | Secret | `RELEASE_NAME-etcd-startup`, `RELEASE_NAME-patroni-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-wal-g-backup-script` | Start scripts of the highly available database | `postgresHA.enabled: true` | | Secret | `RELEASE_NAME-postgres-config` | Non-sensitive backup settings of the highly available database | `postgresHA.backup.enabled: true` | @@ -40,11 +38,13 @@ This template does not create a GVC. It deploys into an existing [GVC](/referenc | Identity, Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | Single-instance database identity and its secret access | `postgres.enabled: true` (default) | | Identity, Policy | `RELEASE_NAME-postgres-ha-identity`, `RELEASE_NAME-postgres-ha-policy`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy` | Database and etcd identities and their secret access | `postgresHA.enabled: true` | -Your app key secret (see [Prerequisites](#prerequisites)) is **not** created by the template. +Your app key, database credentials and Redis password secrets (see [Prerequisites](#prerequisites)) are **not** created by the template. ## Prerequisites -Twenty encrypts stored OAuth tokens, TOTP secrets and app variables, and signs its auth tokens, with one app key that you supply through an [opaque secret](/guides/create-secret/opaque) created **before** installing. The value never passes through Helm values; the template hands it to Twenty as both `APP_SECRET` and `ENCRYPTION_KEY`. +Three secrets must exist **before** installing. They hold credentials, so they are prerequisite secrets rather than values — a value would sit in plaintext in the Helm release for the life of the install. + +Twenty encrypts stored OAuth tokens, TOTP secrets and app variables, and signs its auth tokens, with one app key that you supply through an [opaque secret](/guides/create-secret/opaque). The value never passes through Helm values; the template hands it to Twenty as both `APP_SECRET` and `ENCRYPTION_KEY`. The bundled PostgreSQL and Redis take their credentials from two [dictionary secrets](/guides/create-secret/dictionary). @@ -59,13 +59,32 @@ Twenty encrypts stored OAuth tokens, TOTP secrets and app variables, and signs i Store a copy outside Control Plane. The key is effectively write-once: changing `secrets.name` without pointing `secrets.fallbackName` at the previous key makes stored OAuth tokens, TOTP secrets and app variables undecryptable and logs everyone out. See [Rotating the App Key](#rotating-the-app-key). - - The database password (`postgres.credentials.password`) and the Redis password (`redis.auth.password`) ship as `change-me-…` placeholders that the template uses as-is. Pick strong values and pass them at install time. Both are embedded in connection URLs, so use only letters, digits, `-` and `_`; the chart refuses to render anything else. The database password seeds the database on first boot and is not changed by later value edits. + + A dictionary secret; the bundled PostgreSQL initializes with these values and Twenty connects with them, in either database mode: + + ```bash + cpln secret create-dictionary --name my-twenty-db-credentials \ + --entry username=twenty \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=twenty + ``` + + Set its name in the key of the database mode you use: `postgres.config.credentialsSecretName` (single-instance, the default) or `postgresHA.config.credentialsSecretName` (HA). Use only letters, digits, `-` and `_` in all three values — Twenty puts them in its database connection URL, and other characters break it (the `openssl rand -hex` password above is safe). Secret names are org-wide, so give each Twenty release its own secret. PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password). + + + A dictionary secret with one key, `password`; the bundled Redis requires it and the server and worker connect with it: + + ```bash + cpln secret create-dictionary --name my-twenty-redis \ + --entry password="$(openssl rand -hex 24)" + ``` + + Set its name in `redis.credentialsSecretName`. Use only letters, digits, `-` and `_`: Twenty receives the password inside its `REDIS_URL`, unencoded. Give each Twenty release its own secret. To change it later, see [Rotating the Redis Password](#rotating-the-redis-password). -**If the app key secret does not exist at install time, the deployment wedges silently.** `cpln logs` returns zero lines — the container never starts, so it has nothing to log. The one place the reason appears is `status.versions[].message`: +**If any of these secrets does not exist at install time, the deployment wedges silently.** `cpln logs` returns zero lines — the containers never start, so they have nothing to log. The one place the reason appears is `status.versions[].message` (for the Redis secret, also check `RELEASE_NAME-twenty-redis`): ```bash cpln workload get-deployments RELEASE_NAME-twenty --gvc GVC_NAME -o yaml @@ -82,14 +101,14 @@ Everything else works with the defaults. Three optional features need their own ## Installation -Once the app key secret exists, install into your existing GVC with the two bundled passwords set: +Once the three prerequisite secrets exist, install into your existing GVC with their names set: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/twenty \ - --version 1.3.0 --gvc GVC_NAME \ + --version 1.4.0 --gvc GVC_NAME \ --set secrets.name=APP_KEY_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' \ - --set redis.auth.password='YOUR-STRONG-REDIS-PASSWORD' + --set postgres.config.credentialsSecretName=my-twenty-db-credentials \ + --set redis.credentialsSecretName=my-twenty-redis ``` To install using another method, follow the instructions for it: @@ -305,8 +324,10 @@ The worker and the bundled Redis are never publicly reachable: the worker accept ```yaml redis: image: redis:8.10.0 - auth: - password: change-me-twenty-redis # template-managed; wired into REDIS_URL — letters/digits/-/_ only + # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL, or the deployment + # WEDGES silently. A `dictionary` secret with one key, `password` (letters, + # digits, - and _ only: it goes into REDIS_URL unencoded; openssl rand -hex 24). + credentialsSecretName: my-twenty-redis resources: minCpu: 100m maxCpu: 400m @@ -316,11 +337,11 @@ redis: capacity: 10 # GiB (minimum 10); AOF at /data ``` -Redis is required — Twenty does not boot without `REDIS_URL` — and it runs as a single node with persistence on and eviction disabled, so a queued job is never dropped. Twenty accepts only a plain `redis://` URL, which is why the Sentinel-based [redis](/template-catalog/templates/redis) template is not used here. Change `redis.auth.password` before installing; only letters, digits, `-` and `_` are accepted because the value is embedded in a connection URL. +Redis is required — Twenty does not boot without `REDIS_URL` — and it runs as a single node with persistence on and eviction disabled, so a queued job is never dropped. Twenty accepts only a plain `redis://` URL, which is why the Sentinel-based [redis](/template-catalog/templates/redis) template is not used here. Its password comes from the prerequisite secret named by `redis.credentialsSecretName`; use only letters, digits, `-` and `_`, because the value is embedded in a connection URL. ### Database -Enable exactly one of `postgres` (single instance, default) and `postgresHA` (highly available) — the chart refuses to render with both or neither. Twenty is wired to the active database automatically: the single instance directly, or the HAProxy leader endpoint in HA mode. In **both** modes the database credentials come from `postgres.credentials.*`: this template writes them into a dictionary secret named by `postgres.config.credentialsSecretName` or `postgresHA.config.credentialsSecretName` and hands that name to the bundled database. There is nothing for you to create. +Enable exactly one of `postgres` (single instance, default) and `postgresHA` (highly available) — the chart refuses to render with both or neither. Twenty is wired to the active database automatically: the single instance directly, or the HAProxy leader endpoint in HA mode. In **both** modes the database credentials come from the prerequisite dictionary secret you created (see [Prerequisites](#prerequisites)), named by `postgres.config.credentialsSecretName` or `postgresHA.config.credentialsSecretName`; the bundled database initializes from it and Twenty connects with it. ```yaml # ─── Database: Single-instance PostgreSQL 18 (default) ──────────── @@ -329,18 +350,13 @@ Enable exactly one of `postgres` (single instance, default) and `postgresHA` (hi postgres: enabled: true image: postgres:18 # PostgreSQL 18 (the HA path below runs 17 — the majors differ) - # Credentials for the bundled database — internal plumbing no human types elsewhere. - # Twenty CREATES the dictionary secret named by config.credentialsSecretName below out - # of these three values, so there is nothing for you to create before installing. - credentials: - username: twenty - password: change-me-twenty-db # change before install; letters/digits/-/_ only (it is embedded in a URL) - database: twenty config: - # Name of the dictionary secret this chart creates and the bundled postgres reads. - # Secret names are org-wide: give each twenty release its own name. A second - # release left on this name is REFUSED at install (it is owned by the first - # release) — nothing is shared, overwritten or deleted. + # REQUIRED PREREQUISITE SECRET (single-instance mode) — CREATE IT BEFORE YOU + # INSTALL. If it does not exist the deployment WEDGES silently and looks like a + # platform fault. A `dictionary` secret holding exactly three keys: username, + # password, database (letters/digits/-/_ only — they are embedded in a URL). + # PostgreSQL reads it once, when its data is first initialized, and Twenty + # connects with the same keys. See Prerequisites for the command. credentialsSecretName: my-twenty-db-credentials resources: minCpu: 250m @@ -383,10 +399,9 @@ postgres: postgresHA: enabled: false config: - # Name of the dictionary secret this chart creates and the bundled - # postgres-highly-available reads. Since pgHA 2.5.0 the subchart creates no - # credentials secret of its own, so both stores are fed the same one, built - # from postgres.credentials.* above. + # REQUIRED PREREQUISITE SECRET (HA mode) — the same kind of `dictionary` + # secret as postgres.config.credentialsSecretName above (keys: username, + # password, database). Create it BEFORE you install. credentialsSecretName: my-twenty-db-credentials replicas: 3 resources: @@ -435,7 +450,7 @@ Choose the database mode **before** you have data. The two paths run different P -Secret names are org-wide, and the credentials secret is created by this template in both modes. If you run more than one release of this template in the same org, give each its own `postgres.config.credentialsSecretName` (or `postgresHA.config.credentialsSecretName`). A second release left on the default name is **refused at install** with `cannot be updated because it is being managed by a different release` and creates nothing; the first release is unaffected. +Secret names are org-wide. If you run more than one release of this template in the same org, create a separate database credentials secret and Redis secret for each, and point each release's `postgres.config.credentialsSecretName` (or `postgresHA.config.credentialsSecretName`) and `redis.credentialsSecretName` at its own. ## Connecting @@ -450,14 +465,14 @@ Secret names are org-wide, and the credentials secret is created by this templat | Login | The first account created after install | See [First Run](#first-run) | | PostgreSQL (single-instance mode) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | Credentials in the secret named by `postgres.config.credentialsSecretName` | | PostgreSQL (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | Credentials in the secret named by `postgresHA.config.credentialsSecretName` | -| Redis | `RELEASE_NAME-twenty-redis.GVC_NAME.cpln.local:6379` | Password in the `redisPassword` key of `RELEASE_NAME-twenty-creds` | +| Redis | `RELEASE_NAME-twenty-redis.GVC_NAME.cpln.local:6379` | Password in the `password` key of the secret named by `redis.credentialsSecretName` | Read the public hostname and the stored credentials: ```bash cpln workload get RELEASE_NAME-twenty --gvc GVC_NAME -o yaml # status.canonicalEndpoint cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml # the name you set in *.config.credentialsSecretName -cpln secret reveal RELEASE_NAME-twenty-creds -o yaml # redisPassword +cpln secret reveal REDIS_SECRET_NAME -o yaml # the name you set in redis.credentialsSecretName ``` To verify the server from your machine without going through the public endpoint, forward port `3000` and query the health route: @@ -595,6 +610,62 @@ This restore path has **not** been exercised against a Twenty install. The datab Never change `secrets.name` without setting `secrets.fallbackName` to the previous key. Stored OAuth tokens, TOTP secrets and app variables are encrypted with the old value and become undecryptable, and every user is logged out. +### Rotating the Database Password + +PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with, so a rotation takes three steps in order (single-instance; in HA mode follow [Rotating the Database Password in HA Mode](#rotating-the-database-password-in-ha-mode) instead): + +1. Change it in the database: `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` (connected as that user, from inside the GVC; letters, digits, `-` and `_` only). +2. Update the secret: `cpln secret edit SECRET_NAME`. +3. Right away, redeploy Twenty — the worker first, then the server — and wait until `get-deployments` shows a higher version number, alone and ready (right after the command the status may not show the restart yet, and `ready: true` alone can still describe the old replica). Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. + + ```bash + cpln workload force-redeployment RELEASE_NAME-twenty-worker --gvc GVC_NAME + cpln workload force-redeployment RELEASE_NAME-twenty --gvc GVC_NAME + ``` + +Connections the app already has open keep working after `ALTER ROLE`; new ones fail until its redeploy finishes, so redeploy right away and rotate in a quiet window. + +### Rotating the Database Password in HA Mode + +The HA database (`postgresHA`) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3-5. Back to back: change BOTH roles on -0, put the same password in the secret, restart the members +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +cpln secret edit SECRET_NAME +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 6. Once the members have restarted, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 7. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-twenty-worker`, then `RELEASE_NAME-twenty`, which keep the old password until they restart. The database drops connections briefly at each switchover, and the app cannot open new connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming` for several minutes, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + +### Rotating the Redis Password + +Redis keeps no password on disk: it takes the password from the secret each time it starts, and the server and worker do the same. So rotation is the secret and then the restarts: + +1. Update the secret: `cpln secret edit SECRET_NAME` (letters, digits, `-` and `_` only), and check the new value with `cpln secret reveal SECRET_NAME -o yaml`. +2. Redeploy Redis, the worker and the server together. None of them picks up the change without its redeploy. + + ```bash + cpln workload force-redeployment RELEASE_NAME-twenty-redis --gvc GVC_NAME + cpln workload force-redeployment RELEASE_NAME-twenty-worker --gvc GVC_NAME + cpln workload force-redeployment RELEASE_NAME-twenty --gvc GVC_NAME + ``` + +Expect a short interruption: until all three have restarted, Redis and the app hold different passwords, so queued jobs wait and the server cannot reach its cache. + ### Upgrading From 1.0.0 Version `1.0.1` only updates the bundled highly available database to a release that compacts its etcd cluster. Without compaction, etcd's backend grows with time alone and eventually goes read-only, taking PostgreSQL failover with it. Only installs running the HA database (`postgresHA.enabled: true`, off by default here) are affected — see [etcd History Compaction](/template-catalog/templates/postgres-highly-available#etcd-history-compaction) for the mechanism and the symptoms. Compaction stops further growth but cannot shrink a backend that has already grown. @@ -603,20 +674,20 @@ Version `1.0.1` only updates the bundled highly available database to a release Version `1.1.0` changed the single-instance database path: -- `postgres.config.username`, `postgres.config.password` and `postgres.config.database` moved to `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database`. The new `postgres.config.credentialsSecretName` names the dictionary secret this template now creates from those three values. Carrying the old keys fails the render with `config.username was REMOVED in postgres 3.4.0` — move the three keys and you are done, and ignore that message's advice to create a secret yourself. +- `postgres.config.username`, `postgres.config.password` and `postgres.config.database` were removed. Carrying them fails the render with `config.username was REMOVED in postgres 3.4.0`. Put the values your single-instance database was initialized with into the database credentials secret under [Prerequisites](#prerequisites) (keys `username`, `password`, `database`), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. - `postgres.backup.minio.accessKey` and `postgres.backup.minio.secretKey` were replaced by `postgres.backup.minio.credentialsSecretName`, a prerequisite dictionary secret holding `accessKey` and `secretKey` — see [Backing Up](#backing-up). -Keep `postgres.credentials.*` equal to the values your database was initialised with; the running database still enforces them. +Keep the secret's values equal to the ones your database was initialized with; the running database still enforces them. When upgrading straight to 1.4.0 or later, also follow [Upgrading From 1.3.0 or Earlier](#upgrading-from-1-3-0-or-earlier) for the Redis password. ### Upgrading From 1.1.0 or Earlier Version `1.2.0` moved the highly available database to `postgres-highly-available` 2.5.0, which no longer creates its own credentials secret: - `postgresHA.postgres.username`, `postgresHA.postgres.password` and `postgresHA.postgres.database` were removed. Carrying them fails the render with `the postgres block was REMOVED in 2.5.0`. Delete the block. -- In HA mode the credentials now come from `postgres.credentials.*` — the same keys the single-instance path uses — and this template writes them into the secret named by the new `postgresHA.config.credentialsSecretName`. +- In HA mode the credentials now come from the database credentials secret under [Prerequisites](#prerequisites), named by `postgresHA.config.credentialsSecretName`. -**Use the credentials your cluster already has.** Set `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database` to exactly the values your `postgresHA.postgres.*` block held. The database was initialised with them and still enforces them; putting new values in the secret does not change the running database — it gives Twenty a password that no longer works. +**Use the credentials your cluster already has.** Put exactly the values your `postgresHA.postgres.*` block held into the secret's `username`, `password` and `database` keys. The database was initialized with them and still enforces them; putting new values in the secret does not change the running database — it gives Twenty a password that no longer works. ### Upgrading From 1.2.0 or Earlier @@ -625,6 +696,30 @@ Version `1.2.1` replaced `postgresHA.backup.minio.accessKey` and `postgresHA.bac Version `1.3.0` changes no values. It makes the server and worker wait for a live PostgreSQL before running boot migrations, which fixes a first-install failure where the containers could start against a database that was not yet accepting connections. +### Upgrading From 1.3.0 or Earlier + +Version `1.4.0` removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the database templates require on their own. `redis.auth.password` was removed the same way: the Redis password is now a prerequisite secret named by `redis.credentialsSecretName`. Carrying either old key forward fails the render. + +1.3.x created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Twenty loses its database connection. `OLD_SECRET_NAME` is the `credentialsSecretName` of the mode you use (default `my-twenty-db-credentials`): + +```bash +cpln secret reveal OLD_SECRET_NAME -o yaml # the CURRENT values, until the upgrade deletes this secret +cpln secret create-dictionary --name RELEASE_NAME-postgres-credentials \ + --entry username='YOUR_CURRENT_USERNAME' --entry password='YOUR_CURRENT_PASSWORD' --entry database='YOUR_CURRENT_DATABASE' +``` + +1.3.x also kept the Redis password in a secret it created, `RELEASE_NAME-twenty-creds` (key `redisPassword`), which this upgrade deletes too. Create the Redis secret under a different name, holding the **current** Redis password: + +```bash +cpln secret reveal RELEASE_NAME-twenty-creds -o yaml # redisPassword is the CURRENT Redis password +cpln secret create-dictionary --name RELEASE_NAME-twenty-redis \ + --entry password='YOUR_CURRENT_REDIS_PASSWORD' +``` + +Then upgrade with `postgres.config.credentialsSecretName` (single-instance mode) or `postgresHA.config.credentialsSecretName` (HA mode) set to the database secret's name, `redis.credentialsSecretName` set to the Redis secret's name, and `postgres.credentials` and `redis.auth` removed. Your current values are also in `postgres.credentials` and `redis.auth.password` of the 1.3.x release. + +The upgrade restarts the bundled PostgreSQL and Redis too (they now read the new secrets), so Twenty briefly loses its database and Redis and reconnects by itself. The new version of a workload can also wait for several minutes, with `get-deployments` reporting that it is not allowed to reveal the secret, while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. + ### Scaling and Availability - **Server tier** — `twenty.replicas` sets how many server replicas run behind the endpoint. Anything above `1` requires `storage.type: s3`; the chart refuses to render the combination with local storage. @@ -659,7 +754,7 @@ Version `1.3.0` changes no values. It makes the server and worker wait for a liv **Symptom:** `RELEASE_NAME-twenty` and `RELEASE_NAME-twenty-worker` never reach ready, and `cpln logs` returns zero lines. - **Cause:** a secret referenced by name does not exist — usually the app key secret (`secrets.name`), or the fallback or S3 key secret when those are set. The container never starts, so there is nothing to log. + **Cause:** a secret referenced by name does not exist — usually the app key secret (`secrets.name`), the database credentials secret (`postgres.config.credentialsSecretName` or `postgresHA.config.credentialsSecretName`) or the Redis secret (`redis.credentialsSecretName`), or the fallback or S3 key secret when those are set. The container never starts, so there is nothing to log. **Fix:** read `status.versions[].message`, which names the missing secret, then create it. The deployment recovers on its own after several minutes, or force a redeployment to skip the wait. @@ -712,45 +807,40 @@ Version `1.3.0` changes no values. It makes the server and worker wait for a liv **Cause:** your values still carry the `1.0.x` single-instance keys `postgres.config.username`, `postgres.config.password` or `postgres.config.database`. - **Fix:** move the three keys to `postgres.credentials.*` as described in [Upgrading From 1.0.1 or Earlier](#upgrading-from-1-0-1-or-earlier). Do not create a secret yourself — this template creates it. + **Fix:** put the three values into the database credentials secret and remove the keys, as described in [Upgrading From 1.0.1 or Earlier](#upgrading-from-1-0-1-or-earlier). **Symptom:** `helm upgrade` fails before anything is applied, citing `the postgres block was REMOVED in 2.5.0`. **Cause:** your values still carry `postgresHA.postgres.username`, `postgresHA.postgres.password` or `postgresHA.postgres.database` from version `1.1.0` or earlier. - **Fix:** delete the `postgresHA.postgres` block and put the same three values under `postgres.credentials.*` — see [Upgrading From 1.1.0 or Earlier](#upgrading-from-1-1-0-or-earlier). - - - **Symptom:** installing a second Twenty release in the same org fails with `cannot be updated because it is being managed by a different release`, and nothing is created. - - **Cause:** both releases use the same `credentialsSecretName`. Secret names are org-wide, and the first release owns that secret. - - **Fix:** give the second release its own `postgres.config.credentialsSecretName` (or `postgresHA.config.credentialsSecretName`). The first release is unaffected. + **Fix:** delete the `postgresHA.postgres` block and put the same three values into the database credentials secret — see [Upgrading From 1.1.0 or Earlier](#upgrading-from-1-1-0-or-earlier). - - **Symptom:** the render fails citing `the database password must contain only letters, digits, '-' and '_'` or the same message for `redis.auth.password`. + + **Symptom:** `helm upgrade` fails before anything is applied, citing `postgres.credentials was REMOVED in 1.4.0` or `redis.auth was REMOVED in 1.4.0`. - **Cause:** both passwords are embedded in connection URLs, and other characters break the URL. + **Cause:** your values still carry the 1.3.x plaintext database credentials or Redis password. - **Fix:** choose a password made of letters, digits, `-` and `_` only. For the database password this must be done before the first install — it seeds the database and is not changed by later value edits. + **Fix:** create the two prerequisite secrets holding your current values and remove the old keys — see [Upgrading From 1.3.0 or Earlier](#upgrading-from-1-3-0-or-earlier). ## Important Notes -- **Create the app key secret before installing** — `secrets.name` must point at an existing opaque secret (plain encoding). Without it the deployment wedges silently; see [Prerequisites](#prerequisites). +- **Create all three prerequisite secrets before installing** — the app key (`secrets.name`, opaque, plain encoding), the database credentials and the Redis password. A missing one wedges the deployment silently; see [Prerequisites](#prerequisites). - **The app key is effectively write-once** — rotate it only with `secrets.fallbackName` set to the previous key, or stored OAuth tokens, TOTP secrets and app variables become undecryptable and everyone is logged out. - **Sign up immediately after install** — there is no seeded account, and on a public endpoint the first visitor to sign up becomes the workspace administrator. Install with `publicAccess.enabled: false` until you are ready if that is a concern. - **Set `twenty.serverUrl` whenever browsers open Twenty at anything other than its canonical endpoint** — a custom domain or a port-forward — with the scheme. A mismatch breaks sign-in and CORS with opaque errors. -- **Change `postgres.credentials.password` and `redis.auth.password` before installing** — both are embedded in connection URLs, so use only letters, digits, `-` and `_`; the database password seeds the database on first boot and is not changed by later value edits. +- **Use only letters, digits, `-` and `_` in the database and Redis secrets** — both are embedded in connection URLs. +- **Changing the database credentials secret does not change the database password** — PostgreSQL keeps the password it was initialized with. Rotate as described in [Rotating the Database Password](#rotating-the-database-password); in HA mode a leader switchover comes first. +- **Rotating the Redis password needs a forced redeployment of Redis, the worker and the server** — see [Rotating the Redis Password](#rotating-the-redis-password). - **Choose the database mode before you have data** — the single-instance and HA paths run PostgreSQL 18 and 17 respectively on separate volume sets; switching is a dump and restore, not a values change. - **`twenty.replicas` above `1` requires `storage.type: s3`** — the shared local volume set cannot be snapshotted and lives in one location, so S3 is the production recommendation regardless of replica count. - **AWS S3 is keyless only** — use a cloud account plus a bucket-scoped IAM policy. Static keys are accepted only when `storage.s3.endpoint` points at an S3-compatible server. -- **Give each release its own `credentialsSecretName`** — secret names are org-wide, and a second release on the default name is refused at install. +- **Give each release its own database and Redis secrets** — secret names are org-wide. - **SMTP, AI provider keys, rate limits and OAuth/SSO are not values knobs** — configure them in **Settings → Admin Panel → Configuration Variables** inside the app. - **Foreign data wrappers ("remote objects") are unavailable** — that feature needs upstream's custom PostgreSQL image, which this template does not deploy. -- **Data survives restarts and upgrades; uninstalling deletes it** — records live in the database volume set and local attachments in the storage volume set, and both go with the release. Your prerequisite app key secret is yours and survives an uninstall. +- **Data survives restarts and upgrades; uninstalling deletes it** — records live in the database volume set and local attachments in the storage volume set, and both go with the release. Your prerequisite secrets are yours and survive an uninstall. ## External References diff --git a/template-catalog/templates/tyk.mdx b/template-catalog/templates/tyk.mdx index 49d1bbe6..13a01bc5 100644 --- a/template-catalog/templates/tyk.mdx +++ b/template-catalog/templates/tyk.mdx @@ -1,7 +1,7 @@ --- title: Tyk icon: /template-catalog/templates/icons/tyk.png -description: Tyk open-source API gateway (auth, rate limiting, quotas) as an autoscaling stateless workload over a bundled Redis and Sentinel store, deployed into an existing GVC. An admin API key secret must exist first, plus secrets for your API definitions and policies. +description: Tyk open-source API gateway (auth, rate limiting, quotas) as an autoscaling stateless workload over a bundled Redis and Sentinel store, deployed into an existing GVC. An admin API key secret and Redis and Sentinel password secrets must exist first, plus secrets for your API definitions and policies. keywords: ['api gateway', 'kong alternative', 'apigee alternative', 'oauth gateway', 'api auth', 'api throttle', 'rate limiting', 'api management', 'gateway proxy', 'master keys', 'prerequisite secret', 'redis sentinel'] --- @@ -21,16 +21,15 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-redis` | Redis token, rate-limit, quota and analytics store (stateful, 2 replicas, port `6379`) | — | | Workload | `RELEASE_NAME-sentinel` | Redis Sentinel for automatic Redis failover; the gateway connects through Sentinel (stateful, 3 replicas, port `26379`) | — | | Volume set | `RELEASE_NAME-redis-vs`, `RELEASE_NAME-sentinel-vs` | Redis data and Sentinel state (10 GiB per replica to start, daily snapshots kept 7 days) | `redis.redis.persistence.enabled`, `redis.sentinel.persistence.enabled` (both on by default) | -| Secret | `RELEASE_NAME-redis-auth-password`, `RELEASE_NAME-sentinel-auth-password` | Redis and Sentinel passwords, created by the chart from `redis.redis.auth.password.value` and `redis.sentinel.auth.password.value` | `auth.password.enabled` (on by default) | | Secret | `RELEASE_NAME-redis-config`, `RELEASE_NAME-sentinel-config` | Redis and Sentinel configuration | — | -| Identity / Policy | `RELEASE_NAME-tyk-identity`, `RELEASE_NAME-tyk-api-gateway-policy` | `reveal` on exactly the secrets the gateway reads: your admin, API and policy secrets, and the Redis and Sentinel password secrets | — | +| Identity / Policy | `RELEASE_NAME-tyk-identity`, `RELEASE_NAME-tyk-api-gateway-policy` | `reveal` on exactly the secrets the gateway reads: your admin, API and policy secrets, and your Redis and Sentinel password secrets | — | | Identity / Policy | `RELEASE_NAME-redis-identity`, `RELEASE_NAME-redis-policy`, `RELEASE_NAME-sentinel-identity`, `RELEASE_NAME-sentinel-policy` | The Redis tier's own identities and policies | — | -The admin API key, your API definitions and your policies are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. +The admin API key, your API definitions, your policies and the Redis and Sentinel passwords are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. ## Prerequisites -Three secrets are referenced by name, and each one named in your values must exist before you install. Secrets are org-level, so no `--gvc` flag is involved. +Five secrets are referenced by name, and each one named in your values must exist before you install. Secrets are org-level, so no `--gvc` flag is involved. @@ -104,8 +103,15 @@ Three secrets are referenced by name, and each one named in your values must exi Set `policySecretName` to this name, or set it to `""` to run with no policies. - - The bundled Redis serves only this gateway, so its passwords are Helm values the chart turns into secrets — nothing to create. They ship as the placeholders `change-me-tyk-redis` and `change-me-tyk-sentinel` and are used exactly as written, so set `redis.redis.auth.password.value` and `redis.sentinel.auth.password.value` to values of your own. + + Two [dictionary secrets](/guides/create-secret/dictionary), each with the key `password`: one read by Redis and the gateway, one read by Sentinel and the gateway. The names below are the defaults of `redis.redis.auth.fromSecret.name` and `redis.sentinel.auth.fromSecret.name`: + + ```bash + cpln secret create-dictionary --name my-tyk-redis-credentials --entry password="$(openssl rand -hex 24)" + cpln secret create-dictionary --name my-tyk-sentinel-credentials --entry password="$(openssl rand -hex 24)" + ``` + + Upgrading from 1.3.0? Do not generate new ones; follow [Upgrading from 1.3.0](#upgrading-from-1-3-0). @@ -121,20 +127,16 @@ The message reads `The secret no longer exists. Workload updates are paus ## Installation -Once the secrets exist, install the template into your GVC, naming the secrets and replacing the two password placeholders: +Once the secrets exist, install the template into your GVC, naming the secrets. If you named the Redis and Sentinel password secrets differently from the defaults, also set `redis.redis.auth.fromSecret.name` and `redis.sentinel.auth.fromSecret.name`: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/tyk \ - --version 1.3.0 --gvc GVC_NAME \ + --version 1.4.0 --gvc GVC_NAME \ --set adminSecretName=ADMIN_SECRET_NAME \ --set apiSecretName=APIS_SECRET_NAME \ - --set policySecretName=POLICIES_SECRET_NAME \ - --set redis.redis.auth.password.value='YOUR-STRONG-REDIS-PASSWORD' \ - --set redis.sentinel.auth.password.value='YOUR-STRONG-SENTINEL-PASSWORD' + --set policySecretName=POLICIES_SECRET_NAME ``` -Prefer a values file (`-f`) over `--set` for the passwords if you do not want them in your shell history. - Browse, install, and manage templates visually @@ -237,9 +239,10 @@ redis: memory: 256Mi replicas: 2 auth: - password: + fromSecret: enabled: true - value: change-me-tyk-redis + name: my-tyk-redis-credentials # REQUIRED - dictionary secret holding the Redis password + passwordKey: password # key inside that secret firewall: internal_inboundAllowType: same-gvc persistence: @@ -250,16 +253,17 @@ redis: memory: 256Mi replicas: 3 auth: - password: + fromSecret: enabled: true - value: change-me-tyk-sentinel + name: my-tyk-sentinel-credentials # REQUIRED - dictionary secret holding the Sentinel password + passwordKey: password # key inside that secret firewall: internal_inboundAllowType: same-gvc persistence: enabled: true ``` -Replace both `change-me-…` passwords before installing. Keys under `redis` are passed to the bundled [Redis template](/template-catalog/templates/redis); see its page for the full set. +Both password secrets must exist before installing; see [Prerequisites](#prerequisites). Keys under `redis` are passed to the bundled [Redis template](/template-catalog/templates/redis); see its page for the full set. ### Outbound Connectivity @@ -273,6 +277,7 @@ The gateway's outbound firewall is closed and there is no value to open it. Ever | Internal gateway | `http://RELEASE_NAME-tyk-api-gateway.GVC_NAME.cpln.local:8080` | Subject to `internalAccess.type`. Use your `listenPort` if you changed it. | | Proxied API | `GATEWAY_ADDRESS/LISTEN_PATH` | `listen_path` comes from each API definition, for example `/app1`. | | Gateway Control API | `GATEWAY_ADDRESS/tyk/keys`, `/tyk/apis`, `/tyk/reload` | Send the header `X-Tyk-Authorization` set to the payload of your admin secret. | +| Redis / Sentinel | `RELEASE_NAME-redis.GVC_NAME.cpln.local:6379`, `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` | Internal only; passwords are the `password` key in your Redis and Sentinel password secrets. | | Health check | `GATEWAY_ADDRESS/hello` | Reports gateway status and Redis connectivity. | | Redis and Sentinel | `RELEASE_NAME-redis.GVC_NAME.cpln.local:6379`, `RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379` | Internal only; the gateway reaches Redis through Sentinel. | @@ -318,6 +323,48 @@ cpln workload force-redeployment RELEASE_NAME-tyk-api-gateway --gvc GVC_NAME Keep the secret names unchanged; the release references them by name. +### Rotating the Redis Password + +Redis reads its password from the secret at every start, but Sentinel keeps the one it first started with on its volume, so a new password is applied live on Redis and Sentinel before the gateway restarts. `rx` and `sx` run a command on one Redis node or Sentinel (`LOCATION` is your GVC's location; nodes and Sentinels are numbered from 0 up to `redis.redis.replicas` and `redis.sentinel.replicas` minus one). + +1. Accept the new password alongside the old one on every Redis node, and give it to every Sentinel (`SENTINEL_PASSWORD` is the value in your Sentinel password secret): + +```bash +rx() { cpln workload exec RELEASE_NAME-redis --gvc GVC_NAME --location LOCATION \ + --container redis --replica "RELEASE_NAME-redis-$1" -- "${@:2}"; } +sx() { cpln workload exec RELEASE_NAME-sentinel --gvc GVC_NAME --location LOCATION \ + --container sentinel --replica "RELEASE_NAME-sentinel-$1" -- "${@:2}"; } +# The loops cover the default 2 Redis nodes and 3 Sentinels; list one number per node if you changed the counts +for n in 0 1; do + rx $n redis-cli --no-auth-warning -a OLD_PASSWORD ACL SETUSER default ">NEW_PASSWORD" + rx $n redis-cli --no-auth-warning -a OLD_PASSWORD CONFIG SET masterauth NEW_PASSWORD +done +for n in 0 1 2; do sx $n redis-cli -p 26379 --no-auth-warning -a SENTINEL_PASSWORD SENTINEL SET mymaster auth-pass NEW_PASSWORD; done +``` + +2. Put the new password in the Redis password secret: export it, change only `password`, apply it and read it back: + +```bash +cpln secret reveal my-tyk-redis-credentials -o yaml > secret.yaml +# edit only the password key in secret.yaml +cpln apply -f secret.yaml +cpln secret reveal my-tyk-redis-credentials -o yaml +``` + +3. Restart the gateway, which keeps the old password until it restarts: + +```bash +cpln workload force-redeployment RELEASE_NAME-tyk-api-gateway --gvc GVC_NAME +``` + +4. Once `get-deployments` shows a higher version, alone and ready, retire the old password on every Redis node: + +```bash +for n in 0 1; do rx $n redis-cli --no-auth-warning -a NEW_PASSWORD ACL SETUSER default " +Then follow [Upgrading from 1.3.0](#upgrading-from-1-3-0) to move the Redis and Sentinel passwords into prerequisite secrets. + +### Upgrading from 1.3.0 + +`redis.redis.auth.password` and `redis.sentinel.auth.password` were removed: the two passwords are now prerequisite secrets. Carrying the old keys forward fails the render. + +1.3.0 created `RELEASE_NAME-redis-auth-password` and `RELEASE_NAME-sentinel-auth-password` itself, so **this upgrade deletes them**. Before upgrading, copy the passwords your release runs with **today** into two secrets under **new** names. Do not generate new ones: Sentinel keeps the passwords it first started with on its volume, so a new value locks the gateway out of its store and stalls the Redis restart. To recover, upgrade again with the secrets holding the original passwords. + +```bash +cpln secret reveal RELEASE_NAME-redis-auth-password -o yaml +cpln secret reveal RELEASE_NAME-sentinel-auth-password -o yaml +cpln secret create-dictionary --name my-tyk-redis-credentials --entry password='YOUR-CURRENT-REDIS-PASSWORD' +cpln secret create-dictionary --name my-tyk-sentinel-credentials --entry password='YOUR-CURRENT-SENTINEL-PASSWORD' +``` + +**If you never overrode them, they are literally `change-me-tyk-redis` and `change-me-tyk-sentinel`.** Put those in the new secrets, then change the Redis one as described in [Rotating the Redis Password](#rotating-the-redis-password). + +Then upgrade with `redis.redis.auth.fromSecret.name` and `redis.sentinel.auth.fromSecret.name` set to the new names (the defaults above) and both `auth.password` blocks removed. If 1.3.0 ran with a password turned off, set the matching `auth.fromSecret.enabled: false` instead. + +```bash +cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/tyk \ + --version 1.4.0 --gvc GVC_NAME \ + --set adminSecretName=ADMIN_SECRET_NAME +``` + +The upgrade restarts Redis, Sentinel and the gateway, briefly interrupting key and rate-limit lookups. A new gateway version can also wait several minutes with `is not allowed to reveal the secret` in `get-deployments` while access to the new secrets propagates; the previous version keeps serving meanwhile. + ### Scaling and Availability The gateway is stateless and autoscales between `autoscaling.minScale` and `autoscaling.maxScale` replicas on CPU; raise `minScale` to `2` or more to keep a replica serving while another restarts. Redis runs 2 replicas behind 3 Sentinels, which promote a new Redis primary if the current one fails. Set `multiZone: true` to spread gateway replicas across availability zones where the location supports it. @@ -374,7 +448,7 @@ The gateway is stateless and autoscales between `autoscaling.minScale` and `auto **Symptom:** `cpln helm install` succeeded, but `RELEASE_NAME-tyk-api-gateway` has no running containers and `cpln logs` returns nothing. - **Cause:** A secret named in `adminSecretName`, `apiSecretName` or `policySecretName` does not exist. The defaults `my-tyk-apis` and `my-tyk-policies` count — they must exist unless set to `""`. + **Cause:** A secret named in `adminSecretName`, `apiSecretName`, `policySecretName`, `redis.redis.auth.fromSecret.name` or `redis.sentinel.auth.fromSecret.name` does not exist. The defaults `my-tyk-apis` and `my-tyk-policies` count — they must exist unless set to `""`. **Fix:** Read the message naming the secret, then create it: @@ -439,10 +513,10 @@ The gateway is stateless and autoscales between `autoscaling.minScale` and `auto ## Important Notes -- **Create the prerequisite secrets before installing**, including `apiSecretName` and `policySecretName` unless you set them to `""`. +- **Create the prerequisite secrets before installing**, including the Redis and Sentinel password secrets, and `apiSecretName` and `policySecretName` unless you set them to `""`. A missing secret is named only in `status.versions[].message` of `cpln workload get-deployments`. - **Leave `externalAccess` off unless you need it** — enabling it also publishes the Gateway Control API under `/tyk/`. - **Leave `allowMasterKeys` off** — with it on, any key created without `access_rights` reaches every API on the gateway. -- **Change the Redis and Sentinel passwords** — they ship as `change-me-…` placeholders and are used exactly as written. +- **Never change the Redis password by editing its secret alone.** Sentinel keeps the password it started with; follow [Rotating the Redis Password](#rotating-the-redis-password). - **Force a redeployment after changing a prerequisite secret**, including after rotating the admin key — the running gateway keeps the old value until it restarts. - **Your prerequisite secrets survive `cpln helm uninstall`** — delete them yourself if you no longer need them. diff --git a/template-catalog/templates/umami.mdx b/template-catalog/templates/umami.mdx index 30a1f96c..1318b8b0 100644 --- a/template-catalog/templates/umami.mdx +++ b/template-catalog/templates/umami.mdx @@ -1,7 +1,7 @@ --- title: Umami icon: /template-catalog/templates/icons/umami.png -description: Umami privacy-first, cookieless web analytics as a stateless app tier over a bundled PostgreSQL store (single-instance by default, highly available with one flag), deployed into an existing GVC. An app secret must exist first. Starts private until you change the admin password. +description: Umami privacy-first, cookieless web analytics as a stateless app tier over a bundled PostgreSQL store (single-instance by default, highly available with one flag), deployed into an existing GVC. App and database secrets must exist first. Starts private until you change the admin password. keywords: ['web analytics', 'product analytics', 'cookieless analytics', 'google analytics alternative', 'plausible alternative', 'matomo alternative', 'fathom alternative', 'self-hosted analytics', 'website tracking', 'tracking script', 'app secret', 'port forward', 'patroni'] --- @@ -28,18 +28,17 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-postgres-ha-backup` | Scheduled logical backup of the highly available store (cron) | `postgresHA.backup.enabled: true` with `mode: logical` | | Volume set | `RELEASE_NAME-pg-vs` | Single-instance database data (10 GiB by default; takes no scheduled snapshots; a final snapshot is taken when the volume set is deleted and kept 7 days) | Default; `postgres.enabled: true` | | Volume set | `RELEASE_NAME-postgres-ha-vs`, `RELEASE_NAME-etcd-vs` | Highly available database and consensus-store data (10 GiB per replica by default) | `postgresHA.enabled: true` | -| Secret | The name in `postgres.config.credentialsSecretName` (HA mode: `postgresHA.config.credentialsSecretName`), default `my-umami-db-credentials` | Database credentials (`username`, `password`, `database`), **created by this chart** from `postgres.credentials.*` and read by Umami and the active database | — | | Secret | `RELEASE_NAME-etcd-startup`, `RELEASE_NAME-patroni-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-wal-g-backup-script` | Startup scripts of the highly available database tier | `postgresHA.enabled: true` | | Secret | `RELEASE_NAME-postgres-config` | Non-sensitive backup settings of the highly available database | `postgresHA.backup.enabled: true` | | Identity / Policy | `RELEASE_NAME-umami-identity`, `RELEASE_NAME-umami-policy` | `reveal` on exactly two secrets: your app secret and the database credentials secret | — | | Identity / Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | The single-instance database's own identity and policy | Default; `postgres.enabled: true` | | Identity / Policy | `RELEASE_NAME-postgres-ha-identity`, `RELEASE_NAME-postgres-ha-policy`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy` | The highly available database tier's own identities and policies | `postgresHA.enabled: true` | -Umami itself has no volume set — every user, website, session and event lives in PostgreSQL. The app secret is a **prerequisite secret you create yourself**; the chart references it by name and never creates, modifies or deletes it. +Umami itself has no volume set — every user, website, session and event lives in PostgreSQL. The app secret and the database credentials secret are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. ## Prerequisites -One [opaque secret](/guides/create-secret/opaque) must exist before you install. Secrets are org-level, so no `--gvc` flag is involved. +Two secrets must exist before you install: an [opaque secret](/guides/create-secret/opaque) for the app secret and a [dictionary secret](/guides/create-secret/dictionary) for the database credentials. They hold credentials, so they are prerequisite secrets rather than values — a value would sit in plaintext in the Helm release for the life of the install. Secrets are org-level, so no `--gvc` flag is involved. @@ -51,13 +50,22 @@ One [opaque secret](/guides/create-secret/opaque) must exist before you install. Keep it for the life of the install: changing it logs every user out. Set `app.appSecretName` to this name. - - The bundled database's credentials are internal plumbing: this chart writes `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database` into a dictionary secret it creates, and both Umami and the database read it. Nothing to create — but the password ships as the placeholder `change-me-umami-db`, so set `postgres.credentials.password` to a value of your own before installing. This applies in **both** database modes. + + The bundled PostgreSQL initializes with these values and Umami connects with them, in **both** database modes: + + ```bash + cpln secret create-dictionary --name my-umami-db-credentials \ + --entry username=umami \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=umami + ``` + + Set its name in the key of the store you use: `postgres.config.credentialsSecretName` (single-instance, the default) or `postgresHA.config.credentialsSecretName` (HA). Umami builds its connection URL from the three keys, so keep them URL-safe (letters, digits, `-`, `_`). PostgreSQL reads the secret only when its data is first initialized — see [Rotating the Database Password](#rotating-the-database-password). -**A missing app secret wedges the install rather than failing it.** `cpln helm install` reports success and every resource is created, but the Umami container never starts — so `cpln logs` returns **zero lines**. The only place the missing secret is named is `status.versions[].message`: +**A missing app secret or database credentials secret wedges the install rather than failing it.** `cpln helm install` reports success and every resource is created, but the Umami container never starts — so `cpln logs` returns **zero lines**. The only place the missing secret is named is `status.versions[].message`: ```bash cpln workload get-deployments RELEASE_NAME-umami --gvc GVC_NAME -o yaml @@ -70,16 +78,16 @@ Optional database backups need a bucket and access setup on AWS S3, Google Cloud ## Installation -Once the app secret exists, install the template into your GVC, naming the secret and replacing the database password placeholder: +Once both secrets exist, install the template into your GVC, naming them: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/umami \ - --version 1.3.1 --gvc GVC_NAME \ + --version 1.4.0 --gvc GVC_NAME \ --set app.appSecretName=APP_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' + --set postgres.config.credentialsSecretName=my-umami-db-credentials ``` -Keep these settings in a values file (`--values`) rather than `--set` if you do not want the database password in your shell history — and because every later upgrade must pass them again. If you run more than one release of this template in the same org, also give each its own `postgres.config.credentialsSecretName` — see [Backing Store](#backing-store). +Keep these settings in a values file (`--values`) rather than `--set`, because every later upgrade must pass them again. If you run more than one release of this template in the same org, create a separate database credentials secret for each — see [Backing Store](#backing-store). @@ -145,12 +153,12 @@ Enable exactly one of the two stores — the chart refuses to render with both o postgres: # default: single-instance PostgreSQL — set postgresHA.enabled: true (and this false) for failover enabled: true image: postgres:18 - credentials: # this chart builds the database credentials secret from these - username: umami - password: change-me-umami-db # change before installing - database: umami config: - # name of the dictionary secret this chart CREATES; org-wide, so unique per release + # REQUIRED PREREQUISITE SECRET (single-instance mode) — CREATE IT BEFORE YOU + # INSTALL. If it does not exist the deployment WEDGES silently and looks like a + # platform fault. A `dictionary` secret holding exactly three keys: username, + # password, database. PostgreSQL reads it once, when its data is first + # initialized, and Umami connects with the same keys. See Prerequisites. credentialsSecretName: my-umami-db-credentials resources: minCpu: 250m @@ -163,7 +171,9 @@ postgres: # default: single-instance PostgreSQL — set postgresHA.enabled: true postgresHA: # durable HA: 3-replica Patroni store with an HAProxy leader endpoint enabled: false config: - # the secret name used in HA mode; still built from postgres.credentials.* above + # REQUIRED PREREQUISITE SECRET (HA mode) — the same kind of `dictionary` + # secret as postgres.config.credentialsSecretName above (keys: username, + # password, database). Create it BEFORE you install. credentialsSecretName: my-umami-db-credentials replicas: 3 volumeset: @@ -178,9 +188,9 @@ postgresHA: # durable HA: 3-replica Patroni store with an HAProxy leader endpoin | Database failover | None — a failed node is rescheduled onto the same volume | Automatic leader election | | Best for | Most installs — analytics can tolerate a few minutes of downtime | Installs where the dashboard and collection must stay up through a database failure | -To use HA mode on a **new** install, set `postgres.enabled: false` and `postgresHA.enabled: true`. The credentials still come from `postgres.credentials.*` in both modes. Switching an existing release between modes does not move data — the other store starts empty. +To use HA mode on a **new** install, set `postgres.enabled: false` and `postgresHA.enabled: true`. The credentials come from your prerequisite database credentials secret in both modes, named by `postgresHA.config.credentialsSecretName` in HA mode. Switching an existing release between modes does not move data — the other store starts empty. -Secret names are org-wide, so if you run more than one release of this template in the same org, give each its own credentials secret name — `postgres.config.credentialsSecretName`, or `postgresHA.config.credentialsSecretName` in HA mode. A second release left on the default name is **refused at install** and creates nothing; the first release is unaffected. +Secret names are org-wide, so if you run more than one release of this template in the same org, create a separate database credentials secret for each and point each release's `postgres.config.credentialsSecretName` (or `postgresHA.config.credentialsSecretName` in HA mode) at its own. Database backups are configured under `postgres.backup` / `postgresHA.backup` — see [Backing Up](#backing-up). @@ -216,7 +226,7 @@ internalAccess: | Dashboard (private) | `http://localhost:3000` through `cpln port-forward` | Works with public access off — see the command below | | Internal | `http://RELEASE_NAME-umami.GVC_NAME.cpln.local:3000` | Reachable per `internalAccess.type`; for server-side event tracking from your own workloads | | Admin login | `admin` / `umami` until you change it | Hardcoded by Umami — change it before publishing, see [First Run](#first-run) | -| Database credentials | The secret named by `postgres.config.credentialsSecretName` (HA mode: `postgresHA.config.credentialsSecretName`) | `cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml`; keys `username`, `password`, `database` | +| Database credentials | Your prerequisite secret, named by `postgres.config.credentialsSecretName` (HA mode: `postgresHA.config.credentialsSecretName`) | `cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml`; keys `username`, `password`, `database` | `CANONICAL_ENDPOINT` is the host of `status.canonicalEndpoint`. Open a tunnel to the dashboard from your machine — this works whether or not public access is on: @@ -249,13 +259,13 @@ Umami's bootstrap admin is a hardcoded `admin` / `umami`, seeded by its first da In the dashboard, open **Settings → Profile** and change the password. From then on the old password is rejected. - Upgrade the release with `publicAccess.enabled=true`, passing the **same** app secret name and database password you installed with (or the same values file): + Upgrade the release with `publicAccess.enabled=true`, passing the **same** app secret and database credentials secret names you installed with (or the same values file): ```bash cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/umami \ - --version 1.3.1 --gvc GVC_NAME \ + --version 1.4.0 --gvc GVC_NAME \ --set app.appSecretName=APP_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' \ + --set postgres.config.credentialsSecretName=my-umami-db-credentials \ --set publicAccess.enabled=true ``` @@ -429,27 +439,79 @@ Version 1.2.0 moved the single-instance database credentials. Carrying the old k | 1.1.0 | 1.2.0 and later | |---|---| -| `postgres.config.username`, `postgres.config.password`, `postgres.config.database` | `postgres.credentials.username`, `postgres.credentials.password`, `postgres.credentials.database` | +| `postgres.config.username`, `postgres.config.password`, `postgres.config.database` | The `username`, `password` and `database` keys of the database credentials secret, named by `postgres.config.credentialsSecretName` | | `postgres.backup.minio.accessKey`, `postgres.backup.minio.secretKey` | A dictionary secret holding `accessKey` and `secretKey`, named by `postgres.backup.minio.credentialsSecretName` | -Move the three credential values unchanged — the database was initialised with them. The error message comes from the bundled database (`config.username was REMOVED in postgres 3.4.0`) and tells you to create a credentials secret yourself; **ignore that part** — this template creates it from `postgres.credentials.*`. The MinIO keys, by contrast, are a real prerequisite secret: create it as shown in [Backing Up](#backing-up). +Put the values your single-instance database was initialized with into the database credentials secret under [Prerequisites](#prerequisites), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. The error message comes from the bundled database (`config.username was REMOVED in postgres 3.4.0`). The MinIO keys are a prerequisite secret too: create it as shown in [Backing Up](#backing-up). ### Upgrading From 1.2.0 -Version 1.3.0 moved the HA database credentials onto the same chart-created secret as single-instance mode. Carrying the old block forward fails at render with `the postgres block was REMOVED in 2.5.0`. The three `username`, `password` and `database` values that sat in a `postgres` block under `postgresHA` now come from `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database`, and this chart writes them into the secret named by `postgresHA.config.credentialsSecretName`. +Version 1.3.0 removed the HA database credentials block. Carrying the old block forward fails at render with `the postgres block was REMOVED in 2.5.0`. -Delete that block from your values and set `postgres.credentials.*` to the **same** values it held — the HA database was initialised with them. Do not create the credentials secret yourself. This path has not been rehearsed on an existing HA install; take a backup before upgrading. Single-instance installs need no action. +Put the `username`, `password` and `database` values that sat in a `postgres` block under `postgresHA` **unchanged** into the database credentials secret under [Prerequisites](#prerequisites) — the HA database was initialized with them — set `postgresHA.config.credentialsSecretName` to its name, and delete the block. Take a backup before upgrading. Single-instance installs need no action. ### Upgrading From 1.3.0 Version 1.3.1 moved the HA database's MinIO backup keys into a prerequisite secret. With HA-mode MinIO backups on, the old `accessKey` and `secretKey` values under `postgresHA.backup.minio` now fail at render: create a dictionary secret holding `accessKey` and `secretKey` as shown in [Backing Up](#backing-up) and set `postgresHA.backup.minio.credentialsSecretName` to its name. No other install needs action. +### Upgrading From 1.3.1 + +Version 1.4.0 removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the database templates require on their own. Carrying the old key forward fails the render. + +1.3.x created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Umami loses its database connection. `OLD_SECRET_NAME` is the `credentialsSecretName` of the mode you use (default `my-umami-db-credentials`): + +```bash +cpln secret reveal OLD_SECRET_NAME -o yaml # the CURRENT values, until the upgrade deletes this secret +cpln secret create-dictionary --name RELEASE_NAME-postgres-credentials \ + --entry username='YOUR_CURRENT_USERNAME' --entry password='YOUR_CURRENT_PASSWORD' --entry database='YOUR_CURRENT_DATABASE' +``` + +Then upgrade with `postgres.config.credentialsSecretName` (single-instance mode) or `postgresHA.config.credentialsSecretName` (HA mode) set to that name, and `postgres.credentials` removed. Your current values are also in `postgres.credentials` of the 1.3.x release. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Umami briefly loses its database and reconnects by itself. The new version of a workload can also wait for several minutes, with `get-deployments` reporting that it is not allowed to reveal the secret, while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating the Database Password + +PostgreSQL keeps the password it was initialized with, and a running workload keeps the values it started with, so a rotation takes three steps in order (single-instance; in HA mode follow [Rotating the Database Password in HA Mode](#rotating-the-database-password-in-ha-mode) instead): + +1. Change it in the database: `ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';` (connected as that user, from inside the GVC). +2. Update the secret: `cpln secret edit SECRET_NAME`. +3. Right away, redeploy Umami with `cpln workload force-redeployment RELEASE_NAME-umami --gvc GVC_NAME`, and wait until `get-deployments` shows a higher version number, alone and ready (right after the command the status may not show the restart yet, and `ready: true` alone can still describe the old replica). Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime. + +Umami keeps serving on the database connections it already has open, so logins fail only briefly while its replica is swapped. Do step 3 right away: if those connections drop before Umami restarts, it cannot open new ones. + +### Rotating the Database Password in HA Mode + +The HA database (`postgresHA`) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3-5. Back to back: change BOTH roles on -0, put the same password in the secret, restart the members +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +cpln secret edit SECRET_NAME +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 6. Once the members have restarted, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 7. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-umami`, which keeps the old password until it restarts. The database drops connections briefly at each switchover, and the app cannot open new connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming` for several minutes, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + ### Scaling and Availability - **The app tier scales with `replicas`.** It is stateless — replicas share only the database and the app secret — so `2` or more gives an always-on tier whose rolling restarts cycle one replica at a time. - **The database is the availability limit.** In single-instance mode a database restart makes the dashboard and collection briefly unavailable; set `postgresHA.enabled: true` on a new install for automatic failover. - **The first `helm upgrade` after an install can restart the bundled database** even when no database value changed. -- **Uninstall deletes every volume set the chart created**, so all analytics data is lost and does not survive a reinstall. The GVC is yours and is left alone, and your app secret survives. Enable backups if the data matters. +- **Uninstall deletes every volume set the chart created**, so all analytics data is lost and does not survive a reinstall. The GVC is yours and is left alone, and your prerequisite secrets survive. Enable backups if the data matters. ### Troubleshooting @@ -457,7 +519,7 @@ Version 1.3.1 moved the HA database's MinIO backup keys into a prerequisite secr **Symptom:** `cpln helm install` succeeded, but `RELEASE_NAME-umami` shows no containers and `cpln logs` returns nothing. - **Cause:** The secret named by `app.appSecretName` does not exist. The deployment pauses on the unresolved reference. + **Cause:** The secret named by `app.appSecretName`, or the database credentials secret named by `postgres.config.credentialsSecretName` (HA mode: `postgresHA.config.credentialsSecretName`), does not exist. The deployment pauses on the unresolved reference. **Fix:** Read the message naming the secret, then create it as shown in [Prerequisites](#prerequisites): @@ -481,13 +543,6 @@ Version 1.3.1 moved the HA database's MinIO backup keys into a prerequisite secr **Fix:** Enable exactly one — see [Choosing a Database Mode](#choosing-a-database-mode). For HA mode set `postgres.enabled: false` together with `postgresHA.enabled: true`. - - **Symptom:** Installing a second release of this template in the same org fails with `cannot be updated because it is being managed by a different release`, naming the database credentials secret, and creates nothing. - - **Cause:** Secret names are org-wide, and both releases are trying to create the same credentials secret (default `my-umami-db-credentials`). The first release owns it. - - **Fix:** Give the new release its own `postgres.config.credentialsSecretName` (or `postgresHA.config.credentialsSecretName` in HA mode) and install again. The first release is unaffected. - **Symptom:** All dashboard sessions end at once and everyone has to sign in again. @@ -496,22 +551,23 @@ Version 1.3.1 moved the HA database's MinIO backup keys into a prerequisite secr **Fix:** Nothing is lost — users sign in again. To avoid it, keep the app secret stable for the life of the install. - **Symptom:** `helm upgrade` stops at render with `app.appSecret was removed in 1.1.0`, `resources.cpu was renamed to resources.maxCpu`, `config.username was REMOVED in postgres 3.4.0`, `the postgres block was REMOVED in 2.5.0`, or `backup.minio.accessKey and backup.minio.secretKey were REMOVED`. + **Symptom:** `helm upgrade` stops at render with `app.appSecret was removed in 1.1.0`, `resources.cpu was renamed to resources.maxCpu`, `config.username was REMOVED in postgres 3.4.0`, `the postgres block was REMOVED in 2.5.0`, `backup.minio.accessKey and backup.minio.secretKey were REMOVED`, or `postgres.credentials was REMOVED in 1.4.0`. - **Cause:** Your values still carry a key that moved in template versions 1.1.0 to 1.3.1. The existing release is left untouched and running. + **Cause:** Your values still carry a key that moved in template versions 1.1.0 to 1.4.0. The existing release is left untouched and running. - **Fix:** Follow the matching migration note above — [Upgrading From 1.0.1](#upgrading-from-1-0-1), [Upgrading From 1.1.0](#upgrading-from-1-1-0), [Upgrading From 1.2.0](#upgrading-from-1-2-0) or [Upgrading From 1.3.0](#upgrading-from-1-3-0) — keeping the credentials the database already uses. Do not create a database credentials secret yourself; this template creates it. + **Fix:** Follow the matching migration note above — [Upgrading From 1.0.1](#upgrading-from-1-0-1), [Upgrading From 1.1.0](#upgrading-from-1-1-0), [Upgrading From 1.2.0](#upgrading-from-1-2-0), [Upgrading From 1.3.0](#upgrading-from-1-3-0) or [Upgrading From 1.3.1](#upgrading-from-1-3-1) — keeping the credentials the database already uses in your database credentials secret. ## Important Notes -- **Create the app secret before installing**, and change `postgres.credentials.password` from its `change-me` placeholder. A missing app secret does not fail the install — the workload waits on it with no logs. +- **Create the app secret and the database credentials secret before installing.** A missing one does not fail the install — the workload waits on it with no logs; see [Prerequisites](#prerequisites) for the diagnostic. +- **Editing the database credentials secret alone changes nothing** — rotate with `ALTER ROLE` plus a forced redeployment of Umami (not the database); see [Rotating the Database Password](#rotating-the-database-password). In HA mode a leader switchover comes first. - **Change the hardcoded `admin` / `umami` password before turning public access on** — follow [First Run](#first-run) in order. - **Tracking collects nothing until `publicAccess.enabled` is `true`**, and the tracking script must be loaded from the public address. - **Keep the app secret stable** — changing it logs every user out. It is not owned by the release: it survives `helm uninstall` and must be deleted manually. -- **Pass the same app secret name and database password on every upgrade** — a values file is the simplest way. -- **Give each release its own `credentialsSecretName`** — secret names are org-wide, and a second release on the default name is refused at install. +- **Pass the same app secret and database credentials secret names on every upgrade** — a values file is the simplest way. +- **Give each release its own database credentials secret** — secret names are org-wide. - **Uninstall deletes the database volume sets** — enable backups if the analytics data matters. ## External References diff --git a/template-catalog/templates/unleash.mdx b/template-catalog/templates/unleash.mdx index c7dcfb9a..94aa5d96 100644 --- a/template-catalog/templates/unleash.mdx +++ b/template-catalog/templates/unleash.mdx @@ -1,7 +1,7 @@ --- title: Unleash icon: /template-catalog/templates/icons/unleash.png -description: Unleash open-source feature-flag server (admin UI plus Client and Frontend SDK APIs) as a stateless workload over bundled PostgreSQL, highly available by default or single-instance by one flag, deployed into an existing GVC. An admin-login secret must exist first. +description: Unleash open-source feature-flag server (admin UI plus Client and Frontend SDK APIs) as a stateless workload over bundled PostgreSQL, highly available by default or single-instance by one flag, deployed into an existing GVC. Admin-login and database secrets must exist first. keywords: ['launchdarkly alternative', 'flagsmith alternative', 'split alternative', 'feature toggles', 'gradual rollout', 'kill switch', 'a/b testing', 'canary release', 'self-hosted', 'getunleash', 'client sdk', 'api token', 'patroni'] --- @@ -26,7 +26,6 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Workload | `RELEASE_NAME-postgres-backup` | Scheduled logical dump of the single-instance database (cron) | `postgres.backup.enabled: true` | | Volume set | `RELEASE_NAME-postgres-ha-vs`, `RELEASE_NAME-etcd-vs` | Highly available database and consensus-store data (10 GiB per replica by default; a final snapshot is taken on delete and kept 7 days) | Default; `postgresHA.enabled: true` | | Volume set | `RELEASE_NAME-pg-vs` | Single-instance database data (10 GiB by default; a final snapshot is taken on delete and kept 7 days) | `postgres.enabled: true` | -| Secret | The name in `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` (default `my-unleash-db-credentials`) | Database credentials, **created by this chart** from `postgres.credentials.*` and read by whichever database is enabled | — | | Secret | `RELEASE_NAME-unleash-start` | Start script that derives the public URL, waits until the database accepts queries, then starts Unleash | — | | Secret | `RELEASE_NAME-etcd-startup`, `RELEASE_NAME-patroni-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-wal-g-backup-script` | Startup scripts of the highly available database tier | Default; `postgresHA.enabled: true` | | Secret | `RELEASE_NAME-postgres-config` | Non-sensitive backup settings of the highly available database (bucket, region) | `postgresHA.backup.enabled: true` | @@ -34,11 +33,11 @@ This template deploys into an existing [GVC](/reference/gvc) that you already ha | Identity / Policy | `RELEASE_NAME-postgres-ha-identity`, `RELEASE_NAME-postgres-ha-policy`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy` | The highly available database tier's own identities and policies | Default; `postgresHA.enabled: true` | | Identity / Policy | `RELEASE_NAME-pg-identity`, `RELEASE_NAME-pg-policy` | The single-instance database's own identity and policy | `postgres.enabled: true` | -Unleash itself has no volume set — all of its state lives in the database. The admin login and the optional API-token seed are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. +Unleash itself has no volume set — all of its state lives in the database. The admin login, the database credentials and the optional API-token seed are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. ## Prerequisites -One secret must exist before you install, and a second is optional. Secrets are org-level, so no `--gvc` flag is involved. +Two secrets must exist before you install, and a third is optional. They hold credentials, so they are prerequisite secrets rather than values — a value would sit in plaintext in the Helm release for the life of the install. Secrets are org-level, so no `--gvc` flag is involved. @@ -63,8 +62,17 @@ One secret must exist before you install, and a second is optional. Secrets are Set `apiTokens.secretName` to this name. Leaving it empty (the default) turns seeding off entirely. - - The bundled database's credentials are internal plumbing: this chart writes `postgres.credentials.username`, `postgres.credentials.password` and `postgres.credentials.database` into a dictionary secret it creates, and the database reads that secret. Nothing to create — but the password ships as the placeholder `change-me-unleash-db-password`, so set `postgres.credentials.password` to a value of your own before installing. This applies in **both** database modes, even with `postgres.enabled: false`. + + A [dictionary secret](/guides/create-secret/dictionary) holding exactly the keys `username`, `password` and `database`. The bundled PostgreSQL initializes with these values and Unleash connects with them: + + ```bash + cpln secret create-dictionary --name my-unleash-db-credentials \ + --entry username=unleash \ + --entry password="$(openssl rand -hex 24)" \ + --entry database=unleash + ``` + + Set its name in the key of the database mode you use: `postgresHA.config.credentialsSecretName` (HA, the default) or `postgres.config.credentialsSecretName` (single-instance). PostgreSQL reads the secret only when its data is first initialized; to change the password later, change it in the database first, then update the secret and force a redeployment of the Unleash workload. @@ -82,16 +90,16 @@ Optional database backups need a bucket and access setup on AWS S3, Google Cloud ## Installation -Once the admin secret exists, install the template into your GVC, naming the secret and replacing the database password placeholder: +Once the admin and database credentials secrets exist, install the template into your GVC, naming them: ```bash cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/unleash \ - --version 1.4.0 --gvc GVC_NAME \ + --version 1.5.0 --gvc GVC_NAME \ --set admin.secretName=ADMIN_SECRET_NAME \ - --set postgres.credentials.password='YOUR-STRONG-DB-PASSWORD' + --set postgresHA.config.credentialsSecretName=my-unleash-db-credentials ``` -Add `--set apiTokens.secretName=API_TOKENS_SECRET_NAME` if you created the optional token seed. Prefer a values file (`--values`) over `--set` for the database password if you do not want it in your shell history. If you run more than one release of this template in the same org, also give each its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName` in single-instance mode) — see [Database](#database). +Add `--set apiTokens.secretName=API_TOKENS_SECRET_NAME` if you created the optional token seed. If you run more than one release of this template in the same org, create a separate database credentials secret for each — see [Database](#database). @@ -185,7 +193,7 @@ Exactly one of the two database modes must be enabled — the chart enforces thi postgresHA: enabled: true config: - credentialsSecretName: my-unleash-db-credentials # dictionary secret this chart CREATES from postgres.credentials.*; org-wide name — give each release its own + credentialsSecretName: my-unleash-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install replicas: 3 volumeset: capacity: 10 # initial capacity in GiB per replica (minimum is 10) @@ -196,19 +204,15 @@ postgresHA: enabled: false # set postgresHA.enabled: true (the default) for automatic database failover postgres: enabled: true - credentials: - username: unleash - password: change-me-unleash-db-password # change before installing - database: unleash config: - credentialsSecretName: my-unleash-db-credentials # secret this chart CREATES; org-wide name — give each release its own + credentialsSecretName: my-unleash-db-credentials # PREREQUISITE dictionary secret (username, password, database) — must exist BEFORE install volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) ``` -In **both** modes the database credentials come from `postgres.credentials.*` — the chart writes them into the dictionary secret named by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` (whichever database is enabled), and the database reads that secret. Change `postgres.credentials.password` before installing even when `postgres.enabled` is `false`. In HA mode the proxy must stay enabled — that endpoint is Unleash's stable database address, and the chart refuses to render with `postgresHA.proxy.enabled: false`. +In **both** modes the database credentials come from your prerequisite dictionary secret, named by `postgresHA.config.credentialsSecretName` or `postgres.config.credentialsSecretName` (whichever database is enabled); the database initializes from it and Unleash connects with it. In HA mode the proxy must stay enabled — that endpoint is Unleash's stable database address, and the chart refuses to render with `postgresHA.proxy.enabled: false`. -Secret names are org-wide, so a second release of this template left on the default `credentialsSecretName` is **refused at install** with `cannot be updated because it is being managed by a different release` and creates nothing — the first release is unaffected. Give each release its own name. +Secret names are org-wide, so if you run more than one release of this template in the same org, create a separate database credentials secret for each and point each release's `credentialsSecretName` at its own. Everything else the bundled databases can do — volume autoscaling, PgBouncer pooling, etcd tuning — is documented on the [postgres-highly-available](/template-catalog/templates/postgres-highly-available) and [postgres](/template-catalog/templates/postgres) pages and is set under the same `postgresHA.*` / `postgres.*` prefixes. Their backup settings are covered under [Backing Up](#backing-up). @@ -223,7 +227,7 @@ Everything else the bundled databases can do — volume autoscaling, PgBouncer p | Admin login | The `username` and `password` keys of your admin secret | `cpln secret reveal ADMIN_SECRET_NAME -o yaml` — or whatever you have since changed the password to in the UI | | Database (HA mode) | `RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432` | Always the current leader | | Database (single mode) | `RELEASE_NAME-postgres.GVC_NAME.cpln.local:5432` | — | -| Database credentials | The secret named by `postgresHA.config.credentialsSecretName` / `postgres.config.credentialsSecretName` | `cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml`; keys `username`, `password`, `database` | +| Database credentials | Your prerequisite secret, named by `postgresHA.config.credentialsSecretName` / `postgres.config.credentialsSecretName` | `cpln secret reveal DB_CREDENTIALS_SECRET_NAME -o yaml`; keys `username`, `password`, `database` | | Health | `/health` on either Unleash address | Unauthenticated; the same check the template's own probes use | Read the public address from `status.canonicalEndpoint`, then confirm the server answers — `/health` returns HTTP 200 once Unleash has finished its database migrations: @@ -446,29 +450,29 @@ Two changes landed between 1.0.0 and 1.1.0; apply both when upgrading from 1.0.0 ### Upgrading from 1.1.0 -Version 1.2.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. Unleash absorbed that change — the chart now creates the credentials secret itself — so there is **no new prerequisite**, only a rename of three keys in single-instance mode: +Version 1.2.0 adopted the `postgres` 3.4.1 template, which stopped taking database credentials as values. In single-instance mode three keys were removed: -| Removed key (1.1.0) | Replacement (1.2.0 and later) | +| Removed key (1.1.0) | Replacement (1.5.0 and later) | |---|---| -| `postgres.config.username` | `postgres.credentials.username` | -| `postgres.config.password` | `postgres.credentials.password` | -| `postgres.config.database` | `postgres.credentials.database` | +| `postgres.config.username` | `username` key in the database credentials secret named by `postgres.config.credentialsSecretName` | +| `postgres.config.password` | `password` key in the same secret | +| `postgres.config.database` | `database` key in the same secret | | `postgres.backup.minio.accessKey` / `postgres.backup.minio.secretKey` | a dictionary secret named by `postgres.backup.minio.credentialsSecretName` | -Move the keys **keeping the same values** — the database was initialized with them. A values file that still carries an old key fails the render with the **postgres template's** message (`config.username was REMOVED in postgres 3.4.0`), which tells you to create a dictionary secret yourself — ignore that advice here; this template creates it from `postgres.credentials.*`. If you back up to MinIO, create the dictionary secret described under [Backing Up](#backing-up) and set `postgres.backup.minio.credentialsSecretName`. The HA path is unchanged by this step. +Put the values your single-instance database was initialized with into the database credentials secret under [Prerequisites](#prerequisites), set `postgres.config.credentialsSecretName` to its name, and remove the three keys. A values file that still carries an old key fails the render with the **postgres template's** message (`config.username was REMOVED in postgres 3.4.0`). If you back up to MinIO, create the dictionary secret described under [Backing Up](#backing-up) and set `postgres.backup.minio.credentialsSecretName`. The HA path is unchanged by this step. ### Upgrading from 1.2.x -Version 1.3.0 adopted the `postgres-highly-available` 2.5.0 template, which made the same change on the HA path, and 1.3.1 did the same for its MinIO backup keys. The HA database credentials now come from the same `postgres.credentials.*` block as the single-instance ones, and the chart creates the secret named by `postgresHA.config.credentialsSecretName`. +Version 1.3.0 adopted the `postgres-highly-available` 2.5.0 template, which made the same change on the HA path, and 1.3.1 did the same for its MinIO backup keys. The HA database credentials now come from the database credentials secret named by `postgresHA.config.credentialsSecretName`. -| Removed key (1.2.x) | Replacement (1.3.x and later) | +| Removed key (1.2.x) | Replacement (1.5.0 and later) | |---|---| -| `postgresHA.postgres.username` | `postgres.credentials.username` | -| `postgresHA.postgres.password` | `postgres.credentials.password` | -| `postgresHA.postgres.database` | `postgres.credentials.database` | +| `postgresHA.postgres.username` | `username` key in the database credentials secret named by `postgresHA.config.credentialsSecretName` | +| `postgresHA.postgres.password` | `password` key in the same secret | +| `postgresHA.postgres.database` | `database` key in the same secret | | `postgresHA.backup.minio.accessKey` / `postgresHA.backup.minio.secretKey` | a dictionary secret named by `postgresHA.backup.minio.credentialsSecretName` (1.3.1) | -Move the three credential values **unchanged** into `postgres.credentials.*` — the running cluster was initialized with them — and delete the `postgresHA.postgres` block entirely. Leaving `postgres.enabled: false` is correct; the `postgres.credentials.*` values feed both database modes. A values file that still carries the old block stops the upgrade at render: +Put the three credential values **unchanged** into the database credentials secret under [Prerequisites](#prerequisites) — the running cluster was initialized with them — set `postgresHA.config.credentialsSecretName` to its name, and delete the `postgresHA.postgres` block entirely. A values file that still carries the old block stops the upgrade at render: ```text Error: ... postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres.username`, @@ -477,7 +481,7 @@ Error: ... postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres:` block from your values. See Prerequisites in the README. ``` -The message's instruction to create the secret yourself does not apply — this template creates it. If you back up the HA database to an S3-compatible server, create the credentials secret described under [Backing Up](#backing-up) with the same access key and secret key, set `postgresHA.backup.minio.credentialsSecretName`, and remove the two old keys; if you are already on 1.3.0, that MinIO change is the only part of this step that applies. +If you back up the HA database to an S3-compatible server, create the credentials secret described under [Backing Up](#backing-up) with the same access key and secret key, set `postgresHA.backup.minio.credentialsSecretName`, and remove the two old keys; if you are already on 1.3.0, that MinIO change is the only part of this step that applies. ### Upgrading from 1.3.x @@ -490,6 +494,55 @@ Version 1.4.0 renamed the Unleash container's resource limits: a block that expo An upgrade that still carries either old key is refused at render with `unleash: resources.cpu was RENAMED to resources.maxCpu` (or the `memory` equivalent); rename the two keys and upgrade. `resources.minCpu` and `resources.minMemory` keep their names, and the backup `resources` blocks, which expose only a limit, keep `cpu` and `memory`. +### Upgrading from 1.4.0 + +Version 1.5.0 removed `postgres.credentials`: the database credentials are now a prerequisite secret, the same one the database templates require on their own. Carrying the old key forward fails the render. + +1.4.0 created the database secret itself, so **this upgrade deletes it**. Before upgrading, create a secret under a **new** name holding the values your database was initialized with — not new ones, or Unleash loses its database connection: + +```bash +cpln secret create-dictionary --name RELEASE_NAME-unleash-db \ + --entry username=unleash --entry password='YOUR_CURRENT_PASSWORD' --entry database=unleash +``` + +Then upgrade with `postgresHA.config.credentialsSecretName` (HA mode) or `postgres.config.credentialsSecretName` (single-instance mode) set to that name, and `postgres.credentials` removed. Your current values are in `postgres.credentials` of the 1.4.0 release, and also in the old secret (`cpln secret reveal my-unleash-db-credentials -o yaml`, or the name your release set in `credentialsSecretName`) until the upgrade deletes it. + +The upgrade restarts the bundled PostgreSQL too (it now reads the new secret), so Unleash can briefly lose its database; it reconnects by itself. The new version of a workload can also wait for several minutes, with `get-deployments` reporting that it is not allowed to reveal the secret, while access to the new secret propagates; the previous version keeps serving meanwhile, and it clears by itself. + +### Rotating the Database Password + +PostgreSQL reads the database credentials secret only when its data is first initialized, and a running workload keeps the values it started with. In single-instance mode, change the password in the database first (`ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD';`), then update the secret with `cpln secret edit SECRET_NAME` and force a redeployment of the Unleash workload: + +```bash +cpln workload force-redeployment RELEASE_NAME-unleash --gvc GVC_NAME +``` + +#### Rotating the Database Password in HA Mode + +The HA database (`postgresHA`) needs its own order. The replication role `repl` shares the password, and a member that restarts with the old password can be promoted to leader, which splits the cluster. Move the leader to member `-0` first, so it restarts last, after every other member already has the new password. `pgx` runs a command in one member (`LOCATION` is your GVC's location, `DB_USERNAME` the secret's `username`): + +```bash +pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \ + --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; } +# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline" +pgx 0 curl -s localhost:8008/cluster +# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1 +pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover +# 3-5. Back to back: change BOTH roles on -0, put the same password in the secret, restart the members +pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';" +cpln secret edit SECRET_NAME +cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME +# 6. Once the members have restarted, confirm one leader and two "streaming" replicas on one timeline (repeat step 1) +# 7. Optional: remove the install-time password Patroni keeps in its cluster config +pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config +``` + +Then force-redeploy `RELEASE_NAME-unleash`, which keeps the old password until it restarts. The database drops connections briefly at each switchover, and the app cannot open new connections from the `ALTER ROLE` until its own redeploy finishes, so rotate in a quiet window. If a member stays `starting`, or `running` without `streaming` for several minutes, re-copy it from the leader. This copies the whole database, so it costs network traffic in proportion to its size: + +```bash +pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize +``` + ### Scaling and Availability - **The Unleash server is stateless** — set `replicas` to `2` or more so flags keep being served through a rolling restart or the loss of a replica. Every Helm upgrade rolls the Unleash replicas. @@ -505,7 +558,7 @@ An upgrade that still carries either old key is refused at render with `unleash: **Symptom:** `cpln helm install` succeeded, but `RELEASE_NAME-unleash` shows no running containers and `cpln logs` returns nothing. - **Cause:** A prerequisite secret named in your values does not exist — the admin secret, or the API-token secret when `apiTokens.secretName` is set. The deployment pauses on the unresolved reference. + **Cause:** A prerequisite secret named in your values does not exist — the admin secret, the database credentials secret, or the API-token secret when `apiTokens.secretName` is set. The deployment pauses on the unresolved reference. **Fix:** Read the message naming the secret, then create it: @@ -547,28 +600,22 @@ An upgrade that still carries either old key is refused at render with `unleash: **Fix:** Enable exactly one — see [Database](#database). To use the single-instance database, set `postgresHA.enabled: false` together with `postgres.enabled: true`. - - **Symptom:** Installing a second release of this template in the same org fails with `cannot be updated because it is being managed by a different release`, naming the database credentials secret, and creates nothing. - - **Cause:** Secret names are org-wide, and both releases are trying to create the same `credentialsSecretName` (default `my-unleash-db-credentials`). The first release owns it. - - **Fix:** Give the new release its own `postgresHA.config.credentialsSecretName` (or `postgres.config.credentialsSecretName` in single-instance mode) and install again. The first release is unaffected. - - **Symptom:** `helm upgrade` stops at render with `admin.password was REMOVED in 1.1.0`, `config.username was REMOVED in postgres 3.4.0`, `the postgres block was REMOVED in 2.5.0`, `backup.minio.accessKey and backup.minio.secretKey were REMOVED`, or `resources.cpu was RENAMED to resources.maxCpu`. + **Symptom:** `helm upgrade` stops at render with `admin.password was REMOVED in 1.1.0`, `config.username was REMOVED in postgres 3.4.0`, `the postgres block was REMOVED in 2.5.0`, `backup.minio.accessKey and backup.minio.secretKey were REMOVED`, `resources.cpu was RENAMED to resources.maxCpu`, or `postgres.credentials was REMOVED`. - **Cause:** Your values still carry a key that moved in template versions 1.1.0, 1.2.0, 1.3.0, 1.3.1 or 1.4.0. The existing release is left untouched and running. + **Cause:** Your values still carry a key that moved in template versions 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0 or 1.5.0. The existing release is left untouched and running. - **Fix:** Follow the matching migration note above — [Upgrading from 1.0.x](#upgrading-from-1-0-x), [Upgrading from 1.1.0](#upgrading-from-1-1-0), [Upgrading from 1.2.x](#upgrading-from-1-2-x) or [Upgrading from 1.3.x](#upgrading-from-1-3-x) — keeping the credentials the database already uses. Do not create a database credentials secret yourself; this template creates it. + **Fix:** Follow the matching migration note above — [Upgrading from 1.0.x](#upgrading-from-1-0-x), [Upgrading from 1.1.0](#upgrading-from-1-1-0), [Upgrading from 1.2.x](#upgrading-from-1-2-x), [Upgrading from 1.3.x](#upgrading-from-1-3-x) or [Upgrading from 1.4.0](#upgrading-from-1-4-0) — keeping the credentials the database already uses in your database credentials secret. ## Important Notes -- **Create the admin secret before installing**, and change `postgres.credentials.password` from its `change-me` placeholder. A missing prerequisite secret does not fail the install — the workload waits on it with zero log lines. +- **Create the admin secret and the database credentials secret before installing.** A missing prerequisite secret does not fail the install — the workload waits on it with zero log lines. +- **Rotating the database password needs a forced redeployment** — a running workload keeps the values it started with, and the database keeps its password until you change it there; in HA mode a leader switchover comes first. See [Rotating the Database Password](#rotating-the-database-password). - **The admin secret and the API-token seed apply on first boot only.** Change the password and manage tokens in the admin UI; editing the secrets later changes nothing, and an upgrade never resets your password. - **Backend tokens must stay secret** (server-side SDKs, `/api/client`); frontend tokens are safe to embed in browsers (`/api/frontend`). -- **Give each release its own `credentialsSecretName`** — secret names are org-wide, and a second release on the default name is refused at install. +- **Give each release its own database credentials secret** — secret names are org-wide. - **The free edition ships two environments** (`development`, `production`); SSO, role-based access control, multiple projects, change requests and audit logs need an Unleash Enterprise license and are not available in this template. - **Uninstall deletes the database volume sets** — all flags, users and tokens. Enable backups if the data matters; the prerequisite secrets you created survive and must be deleted manually.