tekn0ir · Oct 11, 2026 · Oct 11, 2026 · Oct 11, 2026
diff --git a/.github/workflows/k3s-ca.yml b/.github/workflows/k3s-ca.yml
@@ -0,0 +1,64 @@
+name: k3s-ca
+
+# The k3s custom-CA library (pkg/templates/scripts/tn-k3s-ca.sh) and its harness.
+# Unit tests run on every change; the docker matrix against real rancher/k3s
+# releases is started by hand (it needs privileged containers and several minutes).
+
+on:
+  pull_request:
+    paths:
+      - 'pkg/templates/**'
+      - 'test/k3s-ca/**'
+      - '.github/workflows/k3s-ca.yml'
+  workflow_dispatch:
+    inputs:
+      suites:
+        description: 'run.sh --suite value'
+        default: 'fresh,remediation'
+      versions:
+        description: 'k3s versions (space separated)'
+        default: 'v1.34.11 v1.36.5'
+
+permissions:
+  contents: read
+
+jobs:
+  unit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+          cache: true
+      - name: shellcheck
+        run: |
+          shellcheck -s sh -S warning pkg/templates/scripts/tn-k3s-ca.sh
+      - name: go test
+        run: go test ./pkg/templates/... ./test/k3s-ca/...
+
+  docker-matrix:
+    if: github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+          cache: true
+      - name: run the harness
+        env:
+          TN_K3S_CA_LIB: ${{ github.workspace }}/pkg/templates/scripts/tn-k3s-ca.sh
+          SUITES: ${{ inputs.suites }}
+          VERSIONS: ${{ inputs.versions }}
+        run: |
+          test/k3s-ca/run.sh --suite "$SUITES" --versions "$VERSIONS" --gens bs-o111 -j 3 --out "$RUNNER_TEMP/k3sca"
+      - name: logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: k3sca-results
+          path: |
+            ${{ runner.temp }}/k3sca/results.tsv
+            ${{ runner.temp }}/k3sca/logs
diff --git a/docs/commands/bootstrap.md b/docs/commands/bootstrap.md
@@ -33,6 +33,27 @@ The scripts are rendered from templates and populated with the device's decoded
 (RSA keypair, username/password, public SSH key), the domain/namespace, the device ID, a freshly
 generated K3S token, and the optional Docker secret.
 
+## The k3s CA
+
+The server scripts (`bootstrap_server_<device>.sh`) use the device certificate (`rsa_public.pem`)
+and key from the device's settings as the **root of the cluster CA** and issue the intermediate and
+leaf CAs below it. k3s v1.34.12, v1.35.9, v1.36.5, v1.37.1 and later only accept a root with the
+`keyCertSign` key usage, so the script **re-issues the root from the same key and subject** with
+`keyCertSign` and CA:TRUE (valid 20 years; the device files are not modified). The code is
+[`pkg/templates/scripts/tn-k3s-ca.sh`](https://github.com/teknoir/cli/blob/main/pkg/templates/scripts/tn-k3s-ca.sh),
+embedded in the script and usable stand-alone (`sh tn-k3s-ca.sh check|remediate`).
+
+Re-running a server script is safe:
+
+| State of the device | What the script does with the CA |
+| --- | --- |
+| k3s never started | Creates the PKI (keeps a valid one) |
+| k3s started, root is fine | Leaves it alone |
+| k3s started, root has no `keyCertSign` (bootstrapped by an older script) | Repairs it first with `k3s certificate rotate-ca` and restarts k3s, so the k3s installer that follows cannot brick the device. `TN_K3S_CA_AUTO_REMEDIATE=false` stops with advice instead; a failed repair stops the script |
+
+See [Troubleshooting](../troubleshooting.md#bootstrapping-k3s) for devices that hang or were
+bricked by a k3s upgrade. The agent script is unchanged: an agent has no CA of its own.
+
 ## Requirements
 
 A domain and namespace must be set, and you must be authenticated.

diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
@@ -53,6 +53,79 @@ including resolved URLs and the exact `ssh` invocation used.
     - Re-run with `--debug` to see the full `ssh` command and any `ncat`/proxy errors.
     - Verify network access to `deadend-<namespace>.<domain>:2222`.
 
+## Bootstrapping & k3s
+
+??? failure "k3s never becomes Ready: `Failed to validate connection to cluster ... CA cert validation failed ... x509: certificate signed by unknown authority`"
+    k3s v1.34.12, v1.35.9, v1.36.5, v1.37.1 and later (k3s-io/k3s#14699, a CVE fix) drop every
+    certificate without the `keyCertSign` key usage from the CA bundle. The device certificate of
+    the platform (`/etc/teknoir/rsa_public.pem`) is the **root** of the cluster CA, and the
+    certificates generated before this was fixed never had `keyCertSign`. The get.k3s.io `stable`
+    channel has been v1.36.5 since 2026-10-01, so a new device bootstrapped with an old script, or an
+    old device whose k3s was upgraded, hits this.
+
+    The bootstrap scripts generated by a `tnctl` with this fix re-issue the root from the **same key
+    and subject** with `keyCertSign` (valid 20 years); the device files in `/etc/teknoir` are never
+    changed, and everything signed below the old root stays valid. Check that your `tnctl` has it:
+    `grep -c tn_k3s_ca_setup bootstrap_server_<device>.sh` prints a number above 0.
+
+    Find out where the device is with the CA tool (`tn-k3s-ca.sh`, run as root on the device; it is
+    [`pkg/templates/scripts/tn-k3s-ca.sh`](https://github.com/teknoir/cli/blob/main/pkg/templates/scripts/tn-k3s-ca.sh)
+    in this repository and is embedded in every generated `bootstrap_server_*.sh`):
+
+    ```bash
+    sh tn-k3s-ca.sh check     # exit 0 = ok, 2 = needs rotation, 3 = needs regeneration, 4 = foreign CA
+    ```
+
+    | Situation | How to tell | What to do |
+    | --- | --- | --- |
+    | **Fresh bootstrap that hangs**: k3s was never Ready on this device | `check` says `needs-regeneration`; `/var/lib/rancher/k3s/server/token` does not exist | Run the fixed `bootstrap_server_*.sh` again, or `sh tn-k3s-ca.sh remediate`. Nothing was ever stored in the cluster, so there is nothing to rotate: the PKI is regenerated (the old files are saved as `server/tls-backup-tn-*.tar`) and k3s is restarted. `rotate-ca` cannot be used here, it needs the token a started server writes |
+    | **Bricked by an upgrade**: the device worked, then k3s was upgraded and does not start | `check` says `needs-rotation`; k3s logs the error above | `sh tn-k3s-ca.sh remediate`. k3s must be running (a k3s that loops on the error is fine, it listens). The tool stages the new CA files, runs `k3s certificate rotate-ca` against the running server, copies the files and restarts k3s. Workloads and the datastore are kept. Details below |
+    | **Still works, not upgraded yet** | `check` says `needs-rotation`; `k3s --version` shows v1.34.11, v1.35.8, v1.36.4 or v1.37.0 or older | Do it now: `sh tn-k3s-ca.sh remediate`, or run the fixed bootstrap script again (it repairs the CA before it touches k3s; `TN_K3S_CA_AUTO_REMEDIATE=false` makes it stop with advice instead) |
+    | **`foreign`** | `check` says `foreign` | The root is not the Teknoir device CA (no matching key in `server/tls/root-ca.key` or `/etc/teknoir/rsa_private.pem`). The tool does not touch it |
+
+    `remediate` runs in phases, which you can also run one at a time (`--phase`): `prepare`
+    (offline: checks, backs up `server/tls`, stages `server/.tn-ca-rotate-<time>/` with a `ROTATE_CMD`),
+    `rotate` (runs `ROTATE_CMD` against the running server), `finalize` (copies the files, only after
+    the rotation succeeded), `all` (default, plus a k3s restart). It is safe to run again: when the
+    root is fine it does nothing.
+
+    `rotate-ca` makes k3s verify the new CA bundle against the old one. An old root that has expired,
+    or that has a key usage without `keyCertSign` (the certificates the device-controller made), cannot
+    vouch for anything any more, so for those the tool verifies the new bundle itself with openssl and
+    adds `--force` to `rotate-ca` (it says so when it stages the rotation). A root without any key usage
+    (the Backstage certificates) is rotated without `--force`.
+
+    If `rotate-ca` fails, nothing was changed on disk. Use the fallback: put the last release before
+    the fix of **the same minor** in place, remediate, and upgrade again.
+
+    | Your minor | Last release without the check |
+    | --- | --- |
+    | v1.34 | v1.34.11 |
+    | v1.35 | v1.35.8 |
+    | v1.36 | v1.36.4 |
+    | v1.37 | v1.37.0 |
+
+    ```bash
+    systemctl stop k3s
+    # k3s-arm64 / k3s-armhf on ARM devices; replace the version with the row for your minor
+    curl -fL -o /usr/local/bin/k3s 'https://github.com/k3s-io/k3s/releases/download/v1.36.4%2Bk3s1/k3s'
+    chmod 755 /usr/local/bin/k3s
+    systemctl start k3s
+    sh tn-k3s-ca.sh remediate
+    ```
+
+    Then upgrade k3s as usual.
+
+    Devices with several k3s servers (embedded etcd) were not part of the tests; remediate the
+    servers one by one and read the [k3s documentation](https://docs.k3s.io/cli/certificate) for
+    `rotate-ca` first. Agent nodes need nothing: they have no CA of their own and pick up the new
+    root from the server.
+
+??? failure "Bootstrap fails with `string too long maxsize=64`"
+    The generated CA certificates carry the device ID and the domain in their common name, which is
+    limited to 64 characters. Fixed: the name is cut to 64 characters in scripts from a `tnctl` with
+    the CA fix. Regenerate the script.
+
 ## Configuration
 
 ??? question "Where is my config?"

diff --git a/pkg/templates/bootstrap_templates.go b/pkg/templates/bootstrap_templates.go
@@ -30,6 +30,15 @@ var DockerServerTemplateStr string
 //go:embed partials/*.tmpl
 var partialsFS embed.FS
 
+// TnK3sCAScript is the k3s CA library (setup, check, remediate). It is embedded in the server
+// bootstrap script as the template "tn-k3s-ca.sh" and is also usable stand-alone.
+//
+//go:embed scripts/tn-k3s-ca.sh
+var TnK3sCAScript string
+
+//go:embed scripts/*.sh
+var scriptsFS embed.FS
+
 var (
 	AgentTemplate        *template.Template
 	ServerTemplate       *template.Template
@@ -42,6 +51,7 @@ func init() {
 
 	ServerTemplate = template.Must(template.New("server").Parse(ServerTemplateStr))
 	template.Must(ServerTemplate.ParseFS(partialsFS, "partials/*.tmpl"))
+	template.Must(ServerTemplate.ParseFS(scriptsFS, "scripts/*.sh"))
 
 	DockerServerTemplate = template.Must(template.New("docker_server").Parse(DockerServerTemplateStr))
 	template.Must(DockerServerTemplate.ParseFS(partialsFS, "partials/*.tmpl"))