fix(bootstrap): root CA with keyCertSign; tn-k3s-ca.sh setup, check and remediate

[tekn0ir]

Problem

k3s v1.34.12, v1.35.9, v1.36.5, v1.37.1 and later (k3s-io/k3s#14699 + backport #14708, "Improve CA hash validation", CVE fix) prune every certificate without the keyCertSign key usage from the CA bundle. pkg/templates/server.sh.tmpl copied the device certificate (rsa_public.pem of the Device CR: Backstage openssl req -x509 without keyUsage, or the device-controller certificate with DigitalSignature|KeyEncipherment) verbatim to server/tls/root-ca.pem and issued the intermediate and the five leaf CAs below it. Result:

• every new device bootstrapped since stable became v1.36.5 (2026-10-01) hangs: Failed to validate connection to cluster ... CA cert validation failed ... x509: certificate signed by unknown authority;
• an existing device bricks when its k3s is upgraded (verified with a persistent data dir v1.34.11 -> v1.34.12), and the unpinned installer in every bootstrap script upgrades it;
• agents take the same path.

Stacked on #6 (the harness); base branch is test/k3s-ca-harness. Please merge #6 first, then retarget this PR to main.

Change

pkg/templates/scripts/tn-k3s-ca.sh (POSIX sh, ~630 lines): the CA logic as a library, embedded in server.sh.tmpl ({{template "tn-k3s-ca.sh" .}} + tn_k3s_ca_setup) and usable stand-alone (sh tn-k3s-ca.sh check|remediate).

The fix (root). The device certificate is re-issued from the same key, subject and subject key identifier as a self-signed root with CA:TRUE (critical), keyUsage = digitalSignature, keyEncipherment, keyCertSign, cRLSign (critical), no EKU, AKI == SKI, serial random, 7300 days. The device files in /etc/teknoir are never modified; everything signed below the old root stays valid (same key and name), which is why existing clusters can be repaired in place.

tn_k3s_ca_setup (what the bootstrap script runs):

state of the data dir	action
k3s never started (no server/token)	create the PKI below the re-issued root (a valid one that belongs to the current device key is kept)
k3s started, root fine	nothing is touched; re-running the script is safe
k3s started, old root	repaired first with rotate-ca (below), before the installer can upgrade k3s; TN_K3S_CA_AUTO_REMEDIATE=false stops with advice; a failed repair stops the script

check: exit 0 ok (also: an agent / no tls dir), 2 needs rotation, 3 needs regeneration, 4 foreign CA (the root is unusable but we do not hold its key), 1 error. Read-only.

remediate [--phase prepare|rotate|finalize|all]

• started datastore (S5 pre-fix k3s still running, S6 bricked by an upgrade): prepare backs up server/tls, stages server/.tn-ca-rotate-<ts>/ (mode 700) with the bundles in which the old root is replaced by the new one, checks that every leaf CA verifies through the new root, and writes the exact ROTATE_CMD; rotate runs k3s certificate rotate-ca --path <stage> against the running server, with SSL_CERT_FILE = the server's own leaf CA so that a post-fix k3s that refuses the old root can still be reached (it only has to be up); finalize copies the files only after ROTATED exists; all also restarts k3s. Safe to repeat (verified: prepare + rotate twice before finalize).
• never initialised datastore (S7, hung fresh bootstrap): there is no server/token, so rotate-ca fails with token must not be empty, and nothing was stored in the cluster that could need rotating. The PKI is regenerated in place (old files saved as server/tls-backup-tn-<ts>.tar) and k3s is restarted. Re-running the fixed bootstrap script does the same. This replaces the design in which S7 was remediated with rotate-ca.
• rotate-ca verifies the new bundle against the old chain, which Go refuses if the old root has expired or has a keyUsage without keyCertSign (the device-controller certificates): first run of the full matrix failed exactly there ("certificate error ID ... new CA cert cannot be verified using old CA chain ... parent certificate cannot sign this kind of certificate"). For those roots ROTATE_CMD gets --force (and says so); the new bundle was verified with openssl first. A root without any keyUsage (Backstage certificates) is rotated without --force.
• Fallback if rotate-ca fails (nothing was changed): put the last pre-fix patch of the same minor in place (v1.34.11 / v1.35.8 / v1.36.4 / v1.37.0), remediate, upgrade again. Documented in docs/troubleshooting.md and exercised as rem-s6-fallback.

Library hygiene. Under set -e and set -o noglob; does not set traps (the old inline block overwrote the script's EXIT trap, so $TMP was never cleaned) or change the working directory (it cd'ed into server/tls, so the installer was downloaded there); CNs are cut to 64 characters (long <device>.<domain> names failed with string too long maxsize=64 before). The agent and docker-server templates are unchanged: agents have no CA of their own.

Also: docs (docs/troubleshooting.md "Bootstrapping & k3s", docs/commands/bootstrap.md), .github/workflows/k3s-ca.yml (shellcheck + go test on PRs touching the library; the docker matrix on workflow_dispatch only), pkg/templates/k3sca_test.go.

Evidence

Unit tests (go test ./pkg/templates/, openssl 3.5 locally, fake k3s): setup produces a root that passes the k3s pruning rules and a chain that verifies; second setup keeps the PKI; initialised dir untouched; long device names; check states 0/2/3/4; prepare changes nothing on disk, stages a 700 dir, ROTATE_CMD without secrets; finalize refuses before ROTATED; a failing rotate-ca leaves the root as it was and names the fallback versions; full run with restart; idempotent second run; --force only for roots that cannot vouch; setup auto-repairs / opt-out / stops on a failed rotation; hung fresh bootstrap (prepare, all, re-run of setup); program mode vs. sourcing; the rendered server script has no cp rsa_public.pem ... root-ca.pem, passes sh -n, runs the CA step before the installer.

Docker harness against real rancher/k3s (stacked #6 at 740cc3d + this PR at b33c6a3, test/k3s-ca/run.sh --suite all -j 4; stable = v1.36.5): 294 / 294 PASS, rc=0:

suite	what	result
control (S1)	legacy block vs proper root, 5 generators x v1.34.11/.12, v1.35.9, v1.36.5, v1.37.1	50/50 (bug reproduced, harness sound)
fresh (S2)	tn_k3s_ca_setup x 5 generators x 5 versions	25/25 READY
legacy-with-cert (S3)	old scripts + fixed certs	6/6
agent (S4)	server + agent, 2 generators x 4 versions, with and without OS trust	16/16
remediation S5	legacy on v1.34.11 + agent + marker ConfigMap -> remediate (ubuntu:20.04/OpenSSL 1.1.1 and debian:bookworm/3.0 helpers) -> v1.34.12 -> v1.36.5, check = 0, second remediate no-op; 3 generators	36/36
remediation S6	bricked by upgrade (v1.34.11 -> v1.36.5 rejects), remediated through the post-fix server, marker kept; bs-o111, go-sha1 (--force), expired (--force)	12/12
remediation S7	hung fresh bootstrap on v1.36.5: PKI regenerated, k3s Ready, check = 0	3/3
remediation fallback	forced: pre-fix patch -> remediate -> upgrade	5/5
rerun (S8)	setup on an initialised dir: tls sha256 unchanged, restart Ready	3/3
host (S9, S10)	as root on the host: remediate --phase all; sourced tn_k3s_ca_setup repairing an old root, upgrade, no-op re-run	6/6
openssl-compat	setup twice + check -> remediate -> check, 7 generators / 5 for remediation, under dash on debian bullseye/bookworm/trixie, ubuntu 18.04/20.04/22.04/24.04, arm64 ubuntu 20.04/22.04 + debian bookworm, armv7 debian bookworm (qemu)	132/132

Real generator output (backstage #350 / device-controller #40 heads) is READY with the old scripts and with this library on v1.34.11/v1.36.5/v1.37.1 (see #6). No PRIVATE KEY in any log, no labelled container left, k3d-teknoir-dev untouched.

Review fixes

CI on this PR found a bug in the first version, fixed in b33c6a3: the library logged through the caller's info/warn/fatal and only defined its own if command -v info found nothing. On Debian/Ubuntu info is the GNU info reader, so a stand-alone sh tn-k3s-ca.sh check|remediate printed No menu item ... instead of its messages, and the test fixtures broke on the runner. The library now logs with its own tn_ca_info/warn/fatal; every unit test runs with a failing info binary first in PATH, and a regression test fails against the old library. The whole matrix below was re-run on the fixed commit.

This PR is the "core fix" the review found missing (WP2): tn-k3s-ca.sh, the server.sh.tmpl change and the check/remediate tool did not exist anywhere, so existing Device CRs still hung on v1.36.5, bootstrapped devices still bricked on an upgrade, and there was no check/remediate tool. The S7 stand-in result (token must not be empty) is handled as described above (regeneration, no rotate-ca), and the docs match the behaviour implemented.

Notes

• Not done / not tested: multi-server (embedded etcd) clusters, only a warning; docker_server.sh.tmpl has no CA code.
• vd1-templates-module (nested pkg/templates module) will need scripts/ in its embed; trivial rebase.
• bootstrap-device vendors the library and the harness byte for byte from this branch (Use the vendored k3s CA library: root CA with keyCertSign, repair of old devices bootstrap-device#11, pinned to commit ab148da).
• Nothing is merged, tagged, released or rolled out. A tnctl release is needed for operators to get the fixed scripts.

Re-review fixes (ab148da)

Three findings from the re-review, all reproduced first and fixed in pkg/templates/scripts/tn-k3s-ca.sh:

1. A failed openssl step was reported as success (tn_ca_generate_pki). Every caller runs it in an if/|| context, where POSIX shells (bash, dash) ignore errexit even if it is set again in the subshell, so the subshell status was that of its last command, rm -rf .ca. A failing openssl ca left a server/tls without intermediate or leaf certificates and tn_k3s_ca_setup returned 0. Now every step ends in || exit 1, the signing requests go through files instead of req | ca -in /dev/stdin (so a chroot without /dev/stdin works, and a failing req is not hidden by the pipeline status), and the new tn_ca_validate_pki checks the generated directory (all 22 files present, bundle sizes 2/3, every key matches its certificate, intermediate and all five leaf CAs chain to the root) in tn_ca_install_pki before the old server/tls is backed up, removed and replaced. A failure leaves the old directory and no backup behind.
2. A foreign CA that k3s refuses no longer lets the installer run (tn_k3s_ca_setup, initialised data dir). New TN_CA_ROOT_USABLE from tn_ca_state: 0 for a multi-certificate bundle whose root has no keyCertSign, or an expired root. State foreign with an unusable root now calls tn_ca_fatal ("... k3s must not be upgraded. Nothing was changed."), so the unpinned installer is not reached. A foreign root that has keyCertSign and only expires within 90 days warns and continues. ok and not-a-server are unchanged. check exit codes are unchanged (check is still 4 for foreign).
3. Serial of the re-issued root (tn_ca_reissue_root). OpenSSL 1.1.1 ignores -set_serial in x509 -in SRC -signkey KEY, so on Jetson Ubuntu 18.04/20.04 and Raspberry Pi bullseye the new root kept the serial of the device certificate (same issuer + serial for two certificates, RFC 5280 4.1.2.2; NSS rejects that). Now x509 -x509toreq -in SRC -signkey KEY followed by x509 -req -signkey KEY -extfile root.cnf -extensions v3_root -set_serial 0x<rand> -days 7300 -sha256: subject DER, public key and SKI are unchanged, AKI = SKI, no EKU, and the serial is random on 1.1.1w, 1.1.1 (18.04) and 3.x. The library also warns if the serial still equals the original. The "serial random" wording in this description is now true on all supported OpenSSL versions.

Tests added

• pkg/templates/k3sca_test.go: TestSetupFailsWhenAnOpensslStepFails (wrapper openssl that fails the Nth genrsa/req/ca/ecparam call: 9 cases; setup must fail, no partial server/tls, no work dir left), TestRemediateKeepsTheOldPKIWhenGenerationFails, TestValidatePKI (5 corruptions), TestSetupStopsOnAnUnusableForeignRoot, TestSetupStopsOnAnExpiredForeignRoot, TestSetupContinuesOnAForeignRootThatOnlyExpiresSoon, TestReissuedRootHasANewSerial (Backstage and device-controller style certificates, setup and remediate). Against the previous library the failing-openssl, validate and foreign-root tests fail (13 cases); the serial test only fails on OpenSSL 1.1.1 (see the compat run below).
• test/k3s-ca/cmd/k3sverify: CheckRoot fails when the root has the issuer and serial of the original certificate (unit test added); test/k3s-ca/compat/assert.sh asserts root serial != original for setup (run 1 and 2) and remediation.

Results (openssl 3.5.7 locally, docker for the rest)

• go test ./pkg/templates/... ./test/k3s-ca/...: all packages ok, 39 top-level tests / 57 including subtests passing (go vet, shellcheck -s sh -S warning clean).
• run.sh --suite openssl-compat (11 distro/arch images incl. debian bullseye 1.1.1w, ubuntu 18.04 1.1.1, ubuntu 20.04 1.1.1f, arm64, arm/v7): 132 / 132 PASS. Negative control: the previous library on debian:bullseye and ubuntu:18.04 fails the new serial assertion on every generator (12 PASS / 24 FAIL over bullseye, 18.04 and bookworm).
• run.sh --suite fresh,remediation --versions "v1.34.11 v1.36.5" --gens bs-o111 against real rancher/k3s: 26 / 26 PASS (fresh, remediation S5 with two helper images, bricked, fallback, hung).
• Not re-run: the full 294-case matrix above (agent, rerun, host and the other generators); the changed code paths are the PKI generation, the root re-issue and the setup state handling, which the two runs above exercise.

🤖 Generated with Claude Code

Final fixes (eca0ea1)

Reviewer should-fix items for a tool that runs on real devices, plus the approved k3s pin (Anders: "K3s can be pinned yes", "Cert param changes are ok").

1. Data dir detection (tn_ca_resolve_datadir). check, remediate and tn_k3s_ca_setup no longer assume /var/lib/rancher/k3s. Order: explicit DATA_DIR / --data-dir; --data-dir, --data-dir=, -d of the ExecStart of k3s.service (multi-line, quoted, drop-in resets); the last data-dir of config.yaml.d/*.yaml (lexical); data-dir of config.yaml; K3S_DATA_DIR of the unit; default. Deviation from the task text: it listed config.yaml before conf.d before the unit. I tested what k3s does (rancher/k3s v1.36.5, three containers): a command line flag beats config.yaml.d, which beats config.yaml (flag /var/lib/cli > drop-in /var/lib/drop > config /var/lib/cfg), so the library follows k3s, as the task intended ("the way k3s does"). check is never ok when a k3s server unit exists but the resolved dir has no server/tls/server-ca.crt: state no-server-data, exit 4, with the dir, where it came from and "pass --data-dir". Tests for each source (TestDataDirResolution, plus check/remediate/setup on a resolved dir).
2. Regeneration race. The regenerate path (hung fresh bootstrap, and setup) stops k3s (TN_K3S_STOP_CMD, else systemctl stop if the unit is active), backs up, moves the old server/tls aside by rename, asserts the target is absent, moves the new dir in, validates the installed result with tn_ca_validate_pki, restores the old dir on any failure, then starts k3s (TN_K3S_START_CMD). A failed stop changes nothing. Tests: order of stop/replace/start, a failing validation of the installed dir (old dir restored, k3s started), a failing stop.
3. rotate-ca --force also for an old root with no basicConstraints, CA:FALSE, or not yet valid, and generally whenever openssl cannot verify the old chain against the old root while the new chain verifies (reason is logged). Four fixtures.
4. Messages and cleanup. A restart or finalize failure after a successful rotate no longer says "the device was not changed": it says the datastore holds the new CA and how to recover (exit 5; the setup stops with "WAS changed"); a second remediate finalizes a completed rotation instead of rotating again; a leftover ROTATED staging dir (key copies) is removed when the CA is ok; the newest 3 tls-backup-tn-*.tar are kept; --phase/--data-dir/--config-dir without a value is a usage error (it used to loop); a k3s older than v1.24.12/v1.25.8/v1.26.3 (no certificate rotate-ca) fails safe before anything is changed, with matching advice (upgrade stepwise to a release with rotate-ca that is older than v1.34.12). Rollback limits are documented in the library header and docs/troubleshooting.md (the datastore is not backed up; restoring the tar after a rotation does not undo it).
5. k3s pin. The server and agent templates export INSTALL_K3S_VERSION (default v1.36.5+k3s1, confirmed as stable on https://update.k3s.io/v1-release/channels today, and the release the harness proves). tnctl bootstrap --k3s-version vX.Y.Z+k3sN changes it (validated, it ends up in a shell script), stable keeps the old behaviour (nothing exported), latest sets INSTALL_K3S_CHANNEL; INSTALL_K3S_VERSION=... sh script overrides at run time. Server and agent are rendered from the same data, so they get the same version. Known limitation: a pin older than what a device runs makes the installer downgrade it (documented). The server script also writes a non-default DATA_DIR to config.yaml and puts the manifests there. Docs: README, docs/commands/bootstrap.md, docs/troubleshooting.md.

Tests / results

• go test ./pkg/templates/... ./test/k3s-ca/... ./cmd/...: ok (also with mawk as awk); go vet, shellcheck -s sh -S warning clean.
• run.sh --suite fresh,remediation --versions "v1.34.11 v1.36.5" --gens "bs-o111 go-sha1" against real rancher/k3s: 44 / 44 PASS (fresh 4, remediation 40), no labelled container left, k3d-teknoir-dev untouched.
• Not re-run: agent, rerun, host and openssl-compat suites (the OpenSSL-facing code paths are unchanged apart from the extra --force probes).

Rebase note: cli PR #5 (vd1-templates-module, the nested pkg/templates module) must be rebased on this branch: scripts/*.sh, the new partial partials/k3s_version.sh.tmpl and TemplateData.K3SVersion have to survive the module move.

bootstrap-device is re-vendored from this head (teknoir/bootstrap-device#11, c014e40).