From 7b3f7f53dc8d7c75b414e6a0daeea205b0fbfe4a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Anders=20=C3=85slund?= Date: Sun, 11 Oct 2026 17:20:03 +0000 Subject: [PATCH 1/2] fix(bootstrap): root CA with keyCertSign; tn-k3s-ca.sh setup, check and remediate k3s v1.34.12/v1.35.9/v1.36.5/v1.37.1 and later (k3s-io/k3s#14699) drop CA bundle certificates without keyCertSign. server.sh.tmpl copied the device certificate (rsa_public.pem, never a CA with keyCertSign) verbatim to server/tls/root-ca.pem, so a new device hangs on 'CA cert validation failed' and an existing one bricks on a k3s upgrade. pkg/templates/scripts/tn-k3s-ca.sh (POSIX sh, embedded in server.sh.tmpl, also usable stand-alone): - tn_k3s_ca_setup: re-issues the device certificate as the root from the SAME key, subject and subject key identifier (CA:TRUE, keyCertSign, 20 years), then issues the intermediate and the five leaf CAs as before. k3s never started: PKI created (a valid one is kept). k3s started and the root is fine: nothing is touched. k3s started and the root is old: repaired with rotate-ca before the unpinned k3s installer runs (TN_K3S_CA_AUTO_REMEDIATE=false stops with advice instead). - check: exit 0 ok / 2 needs rotation / 3 needs regeneration / 4 foreign CA. - remediate [--phase prepare|rotate|finalize|all]: stages the new bundles, runs k3s certificate rotate-ca against the running server (works on a post-fix server that refuses the old root, via SSL_CERT_FILE), copies the files, restarts k3s. A datastore that k3s never initialised (hung fresh bootstrap: no server/token, rotate-ca fails with 'token must not be empty') has nothing to rotate: the PKI is regenerated in place. rotate-ca gets --force only when the old root cannot vouch for the chain (expired, or a keyUsage without keyCertSign); the new bundle is verified with openssl first. - Does not change shell options, traps or the working directory of the caller (the old inline block overwrote the script's EXIT trap and cd'ed into server/tls), works under set -e and noglob, CN capped at 64 characters (long device names broke 'openssl req' before). Tests: pkg/templates/k3sca_test.go (openssl, fake k3s), docs (troubleshooting, bootstrap), a CI workflow (unit tests on PRs, docker matrix on dispatch). The docker harness in test/k3s-ca runs the library against real k3s releases. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/k3s-ca.yml | 64 +++ docs/commands/bootstrap.md | 21 + docs/troubleshooting.md | 73 ++++ pkg/templates/bootstrap_templates.go | 10 + pkg/templates/k3sca_test.go | 589 +++++++++++++++++++++++++ pkg/templates/scripts/tn-k3s-ca.sh | 628 +++++++++++++++++++++++++++ pkg/templates/server.sh.tmpl | 113 +---- 7 files changed, 1393 insertions(+), 105 deletions(-) create mode 100644 .github/workflows/k3s-ca.yml create mode 100644 pkg/templates/k3sca_test.go create mode 100644 pkg/templates/scripts/tn-k3s-ca.sh diff --git a/.github/workflows/k3s-ca.yml b/.github/workflows/k3s-ca.yml new file mode 100644 index 0000000..381bf76 --- /dev/null +++ b/.github/workflows/k3s-ca.yml @@ -0,0 +1,64 @@ +name: k3s-ca + +# The k3s custom-CA library (pkg/templates/scripts/tn-k3s-ca.sh) and its harness. +# Unit tests run on every change; the docker matrix against real rancher/k3s +# releases is started by hand (it needs privileged containers and several minutes). + +on: + pull_request: + paths: + - 'pkg/templates/**' + - 'test/k3s-ca/**' + - '.github/workflows/k3s-ca.yml' + workflow_dispatch: + inputs: + suites: + description: 'run.sh --suite value' + default: 'fresh,remediation' + versions: + description: 'k3s versions (space separated)' + default: 'v1.34.11 v1.36.5' + +permissions: + contents: read + +jobs: + unit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version: '1.25' + cache: true + - name: shellcheck + run: | + shellcheck -s sh -S warning pkg/templates/scripts/tn-k3s-ca.sh + - name: go test + run: go test ./pkg/templates/... ./test/k3s-ca/... + + docker-matrix: + if: github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version: '1.25' + cache: true + - name: run the harness + env: + TN_K3S_CA_LIB: ${{ github.workspace }}/pkg/templates/scripts/tn-k3s-ca.sh + SUITES: ${{ inputs.suites }} + VERSIONS: ${{ inputs.versions }} + run: | + test/k3s-ca/run.sh --suite "$SUITES" --versions "$VERSIONS" --gens bs-o111 -j 3 --out "$RUNNER_TEMP/k3sca" + - name: logs + if: always() + uses: actions/upload-artifact@v4 + with: + name: k3sca-results + path: | + ${{ runner.temp }}/k3sca/results.tsv + ${{ runner.temp }}/k3sca/logs diff --git a/docs/commands/bootstrap.md b/docs/commands/bootstrap.md index 03dcebc..02a3475 100644 --- a/docs/commands/bootstrap.md +++ b/docs/commands/bootstrap.md @@ -33,6 +33,27 @@ The scripts are rendered from templates and populated with the device's decoded (RSA keypair, username/password, public SSH key), the domain/namespace, the device ID, a freshly generated K3S token, and the optional Docker secret. +## The k3s CA + +The server scripts (`bootstrap_server_.sh`) use the device certificate (`rsa_public.pem`) +and key from the device's settings as the **root of the cluster CA** and issue the intermediate and +leaf CAs below it. k3s v1.34.12, v1.35.9, v1.36.5, v1.37.1 and later only accept a root with the +`keyCertSign` key usage, so the script **re-issues the root from the same key and subject** with +`keyCertSign` and CA:TRUE (valid 20 years; the device files are not modified). The code is +[`pkg/templates/scripts/tn-k3s-ca.sh`](https://github.com/teknoir/cli/blob/main/pkg/templates/scripts/tn-k3s-ca.sh), +embedded in the script and usable stand-alone (`sh tn-k3s-ca.sh check|remediate`). + +Re-running a server script is safe: + +| State of the device | What the script does with the CA | +| --- | --- | +| k3s never started | Creates the PKI (keeps a valid one) | +| k3s started, root is fine | Leaves it alone | +| k3s started, root has no `keyCertSign` (bootstrapped by an older script) | Repairs it first with `k3s certificate rotate-ca` and restarts k3s, so the k3s installer that follows cannot brick the device. `TN_K3S_CA_AUTO_REMEDIATE=false` stops with advice instead; a failed repair stops the script | + +See [Troubleshooting](../troubleshooting.md#bootstrapping-k3s) for devices that hang or were +bricked by a k3s upgrade. The agent script is unchanged: an agent has no CA of its own. + ## Requirements A domain and namespace must be set, and you must be authenticated. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index ffce9c6..9b2820f 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -53,6 +53,79 @@ including resolved URLs and the exact `ssh` invocation used. - Re-run with `--debug` to see the full `ssh` command and any `ncat`/proxy errors. - Verify network access to `deadend-.:2222`. +## Bootstrapping & k3s + +??? failure "k3s never becomes Ready: `Failed to validate connection to cluster ... CA cert validation failed ... x509: certificate signed by unknown authority`" + k3s v1.34.12, v1.35.9, v1.36.5, v1.37.1 and later (k3s-io/k3s#14699, a CVE fix) drop every + certificate without the `keyCertSign` key usage from the CA bundle. The device certificate of + the platform (`/etc/teknoir/rsa_public.pem`) is the **root** of the cluster CA, and the + certificates generated before this was fixed never had `keyCertSign`. The get.k3s.io `stable` + channel has been v1.36.5 since 2026-10-01, so a new device bootstrapped with an old script, or an + old device whose k3s was upgraded, hits this. + + The bootstrap scripts generated by a `tnctl` with this fix re-issue the root from the **same key + and subject** with `keyCertSign` (valid 20 years); the device files in `/etc/teknoir` are never + changed, and everything signed below the old root stays valid. Check that your `tnctl` has it: + `grep -c tn_k3s_ca_setup bootstrap_server_.sh` prints a number above 0. + + Find out where the device is with the CA tool (`tn-k3s-ca.sh`, run as root on the device; it is + [`pkg/templates/scripts/tn-k3s-ca.sh`](https://github.com/teknoir/cli/blob/main/pkg/templates/scripts/tn-k3s-ca.sh) + in this repository and is embedded in every generated `bootstrap_server_*.sh`): + + ```bash + sh tn-k3s-ca.sh check # exit 0 = ok, 2 = needs rotation, 3 = needs regeneration, 4 = foreign CA + ``` + + | Situation | How to tell | What to do | + | --- | --- | --- | + | **Fresh bootstrap that hangs**: k3s was never Ready on this device | `check` says `needs-regeneration`; `/var/lib/rancher/k3s/server/token` does not exist | Run the fixed `bootstrap_server_*.sh` again, or `sh tn-k3s-ca.sh remediate`. Nothing was ever stored in the cluster, so there is nothing to rotate: the PKI is regenerated (the old files are saved as `server/tls-backup-tn-*.tar`) and k3s is restarted. `rotate-ca` cannot be used here, it needs the token a started server writes | + | **Bricked by an upgrade**: the device worked, then k3s was upgraded and does not start | `check` says `needs-rotation`; k3s logs the error above | `sh tn-k3s-ca.sh remediate`. k3s must be running (a k3s that loops on the error is fine, it listens). The tool stages the new CA files, runs `k3s certificate rotate-ca` against the running server, copies the files and restarts k3s. Workloads and the datastore are kept. Details below | + | **Still works, not upgraded yet** | `check` says `needs-rotation`; `k3s --version` shows v1.34.11, v1.35.8, v1.36.4 or v1.37.0 or older | Do it now: `sh tn-k3s-ca.sh remediate`, or run the fixed bootstrap script again (it repairs the CA before it touches k3s; `TN_K3S_CA_AUTO_REMEDIATE=false` makes it stop with advice instead) | + | **`foreign`** | `check` says `foreign` | The root is not the Teknoir device CA (no matching key in `server/tls/root-ca.key` or `/etc/teknoir/rsa_private.pem`). The tool does not touch it | + + `remediate` runs in phases, which you can also run one at a time (`--phase`): `prepare` + (offline: checks, backs up `server/tls`, stages `server/.tn-ca-rotate-