[tekn0ir]

Problem

k3s v1.34.12, v1.35.9, v1.36.5, v1.37.1 and later (k3s-io/k3s#14699, CVE fix) prune every CA-bundle certificate without keyCertSign. templates/05_tn_install_k3s.sh copied the device certificate (rsa_public.pem, never a CA with keyCertSign) verbatim to server/tls/root-ca.pem and issued the intermediate and leaf CAs below it, then installed k3s unpinned from get.k3s.io (stable = v1.36.5 since 2026-10-01). A new device hangs on CA cert validation failed ... x509: certificate signed by unknown authority; an existing device bricks when this script (or anything else) upgrades k3s.

Change

• templates/05_tn_install_k3s.sh calls tn_k3s_ca_setup. It re-issues the device certificate as the root from the same key and subject (CA:TRUE, keyCertSign, 20 years). k3s never started: the PKI is created (a valid one is kept). k3s started and the root is fine: nothing is touched, so re-running the script is safe. k3s started with the old root: repaired first with k3s certificate rotate-ca (and a restart), before the installer below can upgrade k3s; TN_K3S_CA_AUTO_REMEDIATE=false stops with advice instead; a failed repair stops the script.
• templates/04_tn_k3s_ca_lib.sh: the library, vendored byte for byte from teknoir/cli pkg/templates/scripts/tn-k3s-ca.sh (fix(bootstrap): root CA with keyCertSign; tn-k3s-ca.sh setup, check and remediate cli#7, branch fix/k3s-ca-root-keycertsign) with one header line; VENDORED_FROM pins cli commit b33c6a3. 04_tn_k3s_ca_guard.sh keeps the library's stand-alone mode (sh tn-k3s-ca.sh check|remediate) off in the assembled script.
• test/k3s-ca/: the docker harness (cli GPU Acceleration Addition #6/double quote issue when eval yaml templates #7), vendored the same way, plus check-vendored.sh (library and 16 harness files == cli at the pinned commit; --cli <checkout> or gh api), local go.mod for the harness unit tests. test/check-assembly.sh (assemble the server and agent scripts, sh -n/dash -n, library embedded byte for byte and ordered guard < library < call < installer, no verbatim copy, agent has no server CA), test/check-setup.sh (setup under dash with the bootstrap options, second run keeps the PKI, initialised dir untouched). docs/development.md explains how to re-vendor and run it all.
• Not changed: generate-custom-ca-certs.sh (the upstream k3s example script; not part of the bootstrap flow, and it already issues its root with keyCertSign), the agent templates (an agent has no CA of its own; it trusts the device certificate in the OS store as before).
• Side effects of dropping the inline block: the script's EXIT trap is no longer overwritten (the temp dir is cleaned) and the working directory is no longer server/tls.

Evidence

Run from this branch (head 80aebda, library from cli b33c6a3) with TN_K3S_CA_LIB=templates/04_tn_k3s_ca_lib.sh test/k3s-ca/run.sh --suite fresh,rerun,remediation,agent,host --gens "bs-o111 go-sha1 expired" -j 3 (real rancher/k3s containers, stable = v1.36.5): 104 / 104 PASS. agent 24 (3 certificate kinds x v1.34.11/.12/v1.36.5/v1.37.1, with and without OS trust), fresh 15, rerun 3, host 6 (remediate --phase all and sourced tn_k3s_ca_setup repairing an old root on the host), remediation 56: old scripts on v1.34.11 with an agent and a marker ConfigMap -> remediate -> v1.34.12 -> v1.36.5 with check = 0 and a no-op second run, for ubuntu:20.04 (OpenSSL 1.1.1) and debian:bookworm (3.0) helpers; bricked-by-upgrade devices repaired through the post-fix server (--force for the device-controller and expired roots); hung fresh bootstrap on v1.36.5 (PKI regenerated); the forced downgrade fallback. The 132-row distro/arch matrix (debian/ubuntu, arm64, armv7) ran against the same library in cli#7.

Offline: test/check-assembly.sh passes (server and agent assemble, sh -n/dash -n, library embedded byte for byte, order guard < library < call < installer); test/check-setup.sh passes; test/k3s-ca/check-vendored.sh passes (via gh api, i.e. against the pushed cli commit); (cd test/k3s-ca && go test ./...) passes. No PRIVATE KEY in any log, no labelled containers left, k3d-teknoir-dev untouched.

Review fixes

The WIP commit on this branch made 05_tn_install_k3s.sh call tn_k3s_ca_setup but 04_tn_k3s_ca_lib.sh did not exist, so any script assembled from it failed at the CA step. It had to wait for the cli library; this PR is opened only now that the library is vendored and check-assembly.sh, check-vendored.sh and the fresh and remediation suites pass. The WIP call convention (tn_k3s_ca_setup <dir> <cert> <key> <suffix>) was replaced by the library's environment-based one (CONFIG_PATH, DATA_DIR, PRODUCT, _DEVICE_ID, _DOMAIN), and check-assembly.sh/check-vendored.sh were adapted (guard variable TN_K3S_CA_NO_MAIN, one header line, the agent rule: the agent adds the device certificate to the OS trust store but prepares no server CA).

Second commit 80aebda re-vendors cli b33c6a3: CI on cli#7 found that the first library version logged through the caller's info, which on Debian/Ubuntu can be the GNU info reader when the library runs stand-alone; no change to the bootstrap flow. The suites above were run on the re-vendored state.

Notes

• Base is toev3; merge cli GPU Acceleration Addition #6 and double quote issue when eval yaml templates #7 first: the vendored commit must be reachable on cli main (it is on the PR branch now, so check-vendored.sh works today). After they merge, re-vendor from the merge commit if the SHA changes.
• Not run: a real device. Nothing is merged, tagged or rolled out.

Re-review fixes (76716eb)

Re-vendored from cli ab148da (byte for byte, test/k3s-ca/check-vendored.sh --cli ../cli passes; VENDORED_FROM updated; the library, k3sverify and compat/assert.sh changed). The cli fixes, see teknoir/cli#7 "Re-review fixes":

• tn_ca_generate_pki: a failed openssl step is a failure (errexit does not work inside the callers' if/||); the generated PKI is validated before server/tls is replaced.
• tn_k3s_ca_setup: an initialised data dir with a foreign CA whose root k3s refuses (no keyCertSign, or expired) stops the bootstrap with "k3s must not be upgraded", so the unpinned k3s installer in 05_tn_install_k3s.sh is not reached. A foreign root that only expires soon continues with a warning.
• tn_ca_reissue_root: random serial for the re-issued root on OpenSSL 1.1.1 too (it ignored -set_serial in x509 -signkey); this affects Jetson Ubuntu 18.04/20.04 and Raspberry Pi bullseye.

test/check-setup.sh (this repo, dash with the bootstrap shell options) now also asserts: new serial of the re-issued root, setup fails and leaves no server/tls when openssl ca fails, setup stops on an unusable foreign CA of an initialised data dir.

Results: test/check-assembly.sh, test/check-setup.sh (17 checks), test/k3s-ca/check-vendored.sh --cli ../cli (library + 16 harness files identical at ab148da), (cd test/k3s-ca && go test ./...) all pass. The docker suites ran in the cli worktree against the same library: openssl-compat 132 / 132 and fresh+remediation on v1.34.11 / v1.36.5 26 / 26 (details in cli#7). The 104-case run above was not repeated on this branch. The pinned SHA in VENDORED_FROM is now ab148dab5ea9c5b7272fe35aac64ec2ee0f2e265; re-vendor again from the merge commit if the SHA changes.

🤖 Generated with Claude Code

Final fixes (c014e40)

Re-vendored from teknoir/cli eca0ea1 (VENDORED_FROM, templates/04_tn_k3s_ca_lib.sh, test/k3s-ca/; check-vendored.sh --cli ../cli is identical). What changed in the library is described in teknoir/cli#7 ("Final fixes"): data dir resolved like k3s (not hard-coded), safe regeneration, more --force cases, clear messages after a rotation, k3s too old for rotate-ca fails safe.

In this repo:

• k3s pin. TN_K3S_VERSION="${INSTALL_K3S_VERSION:-v1.36.5+k3s1}" at the top of the server and agent header; INSTALL_K3S_VERSION is exported before the installer runs in both (stable unsets it: the installer's own choice). Agents must run the version of their server; k3s cannot be downgraded.
• 05_tn_install_k3s.sh. The data-dir: /opt/k3s line (and mkdir -p /opt/k3s) is removed: the config.yaml written a few lines later replaced that file before the installer ran, so it never took effect and devices run k3s from /var/lib/rancher/k3s (the default the library finds; documented in docs/development.md). A non-default DATA_DIR that the library resolved is kept in config.yaml, and the manifests go to ${DATA_DIR}. The duplicate info "Prepare K3s CA Certs" line is gone (the library prints it once).
• test/check-assembly.sh (pin in both scripts and before the installer, stable/override logic, no dead /opt/k3s line, one CA message, manifests in DATA_DIR) and test/check-setup.sh (data dir from config.yaml, config.yaml.d, unit ExecStart, under dash) pass; (cd test/k3s-ca && go test ./...) passes.
• Not changed: docker_server-style scripts do not exist here. The pin must be kept equal to DefaultK3SVersion in teknoir/cli.