Skip to content

k3s CA fix: tn-k3s-ca.sh, keyCertSign root, k3s pin v1.36.5 + cgroup v1, no implicit upgrade - #8

Merged
tekn0ir merged 10 commits into
mainfrom
fix/k3s-ca-root-keycertsign
Oct 12, 2026
Merged

k3s CA fix: tn-k3s-ca.sh, keyCertSign root, k3s pin v1.36.5 + cgroup v1, no implicit upgrade#8
tekn0ir merged 10 commits into
mainfrom
fix/k3s-ca-root-keycertsign

Conversation

@tekn0ir

@tekn0ir tekn0ir commented Oct 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Status: APPROVED by final review at 1d61656 (2026-10-12) — merge this commit. After merging, tag v0.0.8-beta on the merge commit.

Problem

k3s v1.34.12, v1.35.9, v1.36.5 and v1.37.1 (released 2026-09-30, k3s-io/k3s#14699/#14708, a CVE fix) drop CA certificates without keyCertSign. Teknoir devices use the device certificate (Device CR rsa_public/rsa_private) as the k3s root CA. Neither generator set keyCertSign, and both issued 1-year certificates, so many roots have already expired. The templates installed unpinned stable, which became v1.36.5 on 2026-10-01. Since then every new device bootstrap hangs, and existing devices brick on a k3s upgrade.

What this PR does

  • pkg/templates/scripts/tn-k3s-ca.sh (embedded in the server script; standalone check / remediate):
    • Re-issues the k3s root from the same device key, subject and SKI, with CA:TRUE, keyCertSign/cRLSign, a new serial (OpenSSL 1.1.1 included) and 20-year validity. /etc/teknoir and the Device CR are not touched.
    • Remediates legacy devices with k3s certificate rotate-ca, using --force only when the old root can't vouch for the chain.
    • Regenerates the PKI in place for hung, never-started datastores.
    • Validates before replacing, keeps 3 backups, resolves the data dir the way k3s does, and refuses unknown, foreign or unsafe states before changing anything.
  • pkg/templates/scripts/tn-k3s-install.sh:
    • k3s pin v1.36.5+k3s1 by default (tnctl bootstrap --k3s-version, --k3s-upgrade).
    • Re-run keeps the installed k3s. Downgrades are refused. A jump of more than one minor is refused unless forced.
    • cgroup v1 (Jetson L4T r32/r35, Ubuntu 18.04/20.04): k3s ≥ 1.35 refuses cgroup v1 by default. The script writes kubelet-arg+: fail-cgroupv1=false as a drop-in from v1.31, and puts it on the installer command line when an old (< 1.31) kubelet must not see it yet.
    • Agents follow their server: they install exactly the server's release, read from https://<server>:6443/static/teknoir/k3s-version, which the server script publishes. k3s denies anonymous /version.

Fleet behaviour change

A new agent for a server bootstrapped by an older script stops before installing k3s, because that server publishes no version. Re-run the server script (which also repairs a legacy CA: rotate-ca and a k3s restart), or pass INSTALL_K3S_VERSION.

Evidence

  • Docker harness (test/k3s-ca, merged in test: k3s custom-CA harness (reproduces k3s CA hash validation failure) #6): full matrix 294/294 on cli. Every later round re-ran the touched suites green: fresh, remediation, rerun, install, agent, host, and openssl-compat on 11 distro/arch images.
  • Two proofs on a real cgroup-v1 kernel (Ubuntu 20.04 KVM):
    • Plain k3s 1.36.5 crash-loops; the fixed script gives Ready.
    • A legacy device is repaired while staying on v1.34.11, then upgraded one minor at a time.
    • A hung Jetson-like v1.36.5 with a legacy CA ends Ready.
    • Agents install their server's exact release, both v1.34.11 and v1.36.5.
    • A re-run is a byte-identical no-op.
  • Staging fleet check: all 3 reachable toev3 devices report check exit 2 (needs rotation). They are healthy on 1.34.5/1.34.6 until a k3s upgrade.
  • Opus reviews over six rounds. Each round's findings (silent openssl failure, foreign CA, serial reuse, data dir, cgroup v1, agent skew, step order) are fixed and tested. The final review of the last changes approved 1d61656.

After merge

  1. Tag v0.0.8-beta on the merge commit (goreleaser). The libraries are embedded in the binary.
  2. Merge the bootstrap-device PR (fix/k3s-ca-root-keycertsign → toev3, vendored from 1d61656). Re-run test/k3s-ca/check-vendored.sh against the merge commit.
  3. Remediate devices one at a time, in a window: sudo sh tn-k3s-ca.sh check, then remediate.

Known follow-ups (not blocking): --k3s-version stable is passed to the installer as an exact version (the installer then fails safely before changing anything; it should use INSTALL_K3S_CHANNEL); the CA data-dir resolver doesn't read config.yaml.d/*.yml. cli #5 (vd1-templates-module) must be rebased on this.

🤖 Generated with Claude Code

tekn0ir and others added 10 commits October 11, 2026 17:20
@tekn0ir @claude
fix(bootstrap): root CA with keyCertSign; tn-k3s-ca.sh setup, check a…
7b3f7f5 
…nd remediate

k3s v1.34.12/v1.35.9/v1.36.5/v1.37.1 and later (k3s-io/k3s#14699) drop CA
bundle certificates without keyCertSign. server.sh.tmpl copied the device
certificate (rsa_public.pem, never a CA with keyCertSign) verbatim to
server/tls/root-ca.pem, so a new device hangs on 'CA cert validation failed'
and an existing one bricks on a k3s upgrade.

pkg/templates/scripts/tn-k3s-ca.sh (POSIX sh, embedded in server.sh.tmpl,
also usable stand-alone):

- tn_k3s_ca_setup: re-issues the device certificate as the root from the
  SAME key, subject and subject key identifier (CA:TRUE, keyCertSign, 20
  years), then issues the intermediate and the five leaf CAs as before. k3s
  never started: PKI created (a valid one is kept). k3s started and the root
  is fine: nothing is touched. k3s started and the root is old: repaired with
  rotate-ca before the unpinned k3s installer runs
  (TN_K3S_CA_AUTO_REMEDIATE=false stops with advice instead).
- check: exit 0 ok / 2 needs rotation / 3 needs regeneration / 4 foreign CA.
- remediate [--phase prepare|rotate|finalize|all]: stages the new bundles,
  runs k3s certificate rotate-ca against the running server (works on a
  post-fix server that refuses the old root, via SSL_CERT_FILE), copies the
  files, restarts k3s. A datastore that k3s never initialised (hung fresh
  bootstrap: no server/token, rotate-ca fails with 'token must not be empty')
  has nothing to rotate: the PKI is regenerated in place. rotate-ca gets
  --force only when the old root cannot vouch for the chain (expired, or a
  keyUsage without keyCertSign); the new bundle is verified with openssl first.
- Does not change shell options, traps or the working directory of the
  caller (the old inline block overwrote the script's EXIT trap and cd'ed into
  server/tls), works under set -e and noglob, CN capped at 64 characters
  (long device names broke 'openssl req' before).

Tests: pkg/templates/k3sca_test.go (openssl, fake k3s), docs (troubleshooting,
bootstrap), a CI workflow (unit tests on PRs, docker matrix on dispatch). The
docker harness in test/k3s-ca runs the library against real k3s releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir @claude
fix(tn-k3s-ca): do not depend on the caller's info/warn/fatal
b33c6a3 
CI found it: on Debian/Ubuntu `info` is the GNU info reader. The library fell
back to its own logging only if `command -v info` found nothing, so a
stand-alone `sh tn-k3s-ca.sh check|remediate` on a machine with the info
package printed "No menu item ..." instead of its messages. The library now
logs with tn_ca_info/tn_ca_warn/tn_ca_fatal (same format as header.sh.tmpl).
legacy-ca.sh (harness) asks for a function with `type` for the same reason.

Every unit test now runs with a failing `info` binary first in PATH; a
regression test covers the stand-alone and the setup path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir @claude
fix(tn-k3s-ca): fail on a failed openssl step, stop on an unusable fo…
ab148da 
…reign CA, new serial on OpenSSL 1.1.1

- tn_ca_generate_pki: errexit is ignored in the if/|| context of every caller, so the status of
  the subshell was that of its last command and a failed openssl step looked like success.
  Every step now ends in `|| exit 1`, requests go through files (no `-in /dev/stdin`, which
  chroots may lack), and tn_ca_validate_pki checks the new directory (all files, bundle sizes,
  key/cert pairs, chain to the root) before server/tls is backed up and replaced.
- tn_k3s_ca_setup: on an initialised data dir a foreign CA whose root k3s refuses (no keyCertSign
  in a bundle, or expired) now stops the bootstrap ("k3s must not be upgraded") instead of
  letting the unpinned installer run. A foreign root that only expires soon still continues.
- tn_ca_reissue_root: OpenSSL 1.1.1 ignores -set_serial in `x509 -signkey`, so the new root kept
  the serial of the device certificate. Re-issue through `x509 -x509toreq | x509 -req`.
- tests: failing-openssl wrapper (genrsa/req/ca/ecparam), validate_pki, foreign unusable/
  expired/expires-soon, serial of the re-issued root; k3sverify CheckRoot and compat/assert.sh
  assert serial != original (fails on the old library in debian:bullseye and ubuntu:18.04).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tekn0ir @claude
fix(tn-k3s-ca): data dir like k3s, safe regeneration, more --force ca…
eca0ea1 
…ses, k3s pin

Data dir: check, remediate and setup resolve the data dir the way k3s does
(explicit DATA_DIR/--data-dir, ExecStart --data-dir/-d of the k3s unit,
config.yaml.d/*.yaml, config.yaml, K3S_DATA_DIR of the unit, default). Verified
against rancher/k3s v1.36.5: a command line flag beats config.yaml.d, which
beats config.yaml. check is never ok when a k3s server unit exists but the
resolved dir has no server/tls/server-ca.crt (no-server-data, exit 4).

Regeneration: k3s is stopped (TN_K3S_STOP_CMD / systemctl) while server/tls is
replaced; the old dir is moved aside, the target asserted absent, the installed
result validated, the old dir restored on any failure, k3s started again.

rotate-ca --force also for an old root with no basicConstraints, CA:FALSE, not
yet valid, or whenever openssl cannot verify the old chain but the new one.

Messages: a restart or finalize failure after a successful rotate says the
datastore holds the new CA and how to recover (exit 5); a completed rotation is
finalized instead of rotated twice; leftover ROTATED staging is removed when the
CA is ok; the newest 3 tls backups are kept; '--phase' without a value is a
usage error; a k3s without rotate-ca (< v1.24.12/v1.25.8/v1.26.3) fails safe
with matching advice. Rollback limits documented.

k3s pin: the server and agent scripts install INSTALL_K3S_VERSION
v1.36.5+k3s1 by default; tnctl bootstrap --k3s-version changes it (stable keeps
the installer's own behaviour, latest follows that channel). The server script
keeps a non-default data dir in config.yaml and writes the manifests there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir @claude
feat(bootstrap): cgroup v1 kubelet flag, keep the installed k3s on a …
946f20f 
…re-run, final-review fixes

cgroup v1: Kubernetes >= 1.35 makes the kubelet refuse to start on a cgroup v1
host (failCgroupV1 defaults to true; Jetson L4T r32/r35, Ubuntu 18.04/20.04),
and k3s v1.36.5 does not change that. New library pkg/templates/scripts/
tn-k3s-install.sh (embedded in the server and agent script): on a host where
stat -fc %T /sys/fs/cgroup is not cgroup2fs (or cannot be told: OS builds) and
the k3s that will run is >= v1.35, it writes
/etc/rancher/k3s/config.yaml.d/99-teknoir-cgroupv1.yaml with
`kubelet-arg+: fail-cgroupv1=false` (appends to the kubelet-arg of config.yaml;
verified with rancher/k3s v1.36.5: the user's max-pods is kept) and warns that
cgroup v1 is deprecated. Idempotent, removed on cgroup v2, a fail-cgroupv1 set by
the device itself is respected. TN_K3S_CGROUPV1=true|false forces the answer.

Re-run policy: k3s installed -> its exact release is kept (a different pin is
reported and ignored). tnctl bootstrap --k3s-upgrade (TN_K3S_UPGRADE=true) lets
the pin replace it; a downgrade is always refused, an upgrade over more than one
minor is refused unless --force (TN_K3S_FORCE=true). The server script no longer
replaces an existing config.yaml (keys are added when missing).

tn-k3s-ca.sh: an 'activating' unit counts as running when it is stopped before a
regeneration; setup and remediate stop when the resolved data dir was never
initialised but the k3s unit is active and ready (k3s uses another data dir);
prepare checks every write of the bundle rebuild and verifies each rebuilt bundle
(same certificate count, new root last) before rotate-ca; leftover .tn-tls-old.*
and .tn-tls-new.* dirs (they hold keys) are removed once the CA is ok; stop,
start and restart of k3s are bounded (TN_K3S_SERVICE_TIMEOUT, 120 s) and a
timeout is reported as "the CA was changed but k3s did not restart".

Tests: Go unit tests for both libraries and the rendered scripts; harness suite
`install` (real k3s: flag passed and user kubelet-arg kept on >= v1.35, version
read from the real `k3s --version`), install-lib check in openssl-compat.
Docs: bootstrap.md, troubleshooting.md, harness README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir @claude
feat(bootstrap): agent follows its server, flag before the CA step, s…
24ed293 
…low-device waits, safer refusals

- An agent installs the k3s release of its server, never a newer kubelet. k3s answers /version with
  401 to anonymous callers (all releases), so the server script now publishes the release it runs as
  server/static/teknoir/k3s-version (served by k3s without authentication; tn_inst_publish_release)
  and the agent reads it (then /version, for a server that allows it). A release asked for
  (--k3s-version / INSTALL_K3S_VERSION) is installed if not newer than the server, refused if newer;
  an unknown server release stops the script unless a release was asked for or TN_K3S_FORCE=true.
- Server script order: release policy + cgroup flag, then the CA step, then the installer, so a k3s
  that a cgroup v1 host made exit has the kubelet flag when the CA step restarts it; rotate-ca is tried
  again while the unit starts.
- TN_K3S_SERVICE_TIMEOUT defaults to 600 s; after a timeout the unit is watched 120 s more.
- A k3s binary or server/db whose version cannot be read is not installed over (TN_K3S_FORCE=true).
- fail-cgroupv1=false from k3s v1.31 (was v1.35), written atomically.
- Refusals say what this run changed (TN_K3S_CA_CHANGED); docs fixed.
- A data dir that differs from config.yaml(.d) is refused before anything is changed.
- Harness: install suite covers v1.30/v1.31 and the agent-follows-server scenario with real k3s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir @claude
fix(bootstrap): INSTALL_K3S_EXEC is final before the cgroup step, dat…
e5fa8ad 
…a dir guard first, harness expects the drop-in from v1.31

The server script rebuilt INSTALL_K3S_EXEC after tn_inst_prepare. The library reads that variable
(a fail-cgroupv1 in it counts as the device's own setting and removes the drop-in), so the order
mattered: build it first, keeping the caller's value, so the library sees what the installer gets.

- server.sh.tmpl: INSTALL_K3S_EXEC is built before tn_ca_resolve_datadir / tn_inst_prepare
- new tn_ca_require_consistent_datadir (library): called right after tn_ca_resolve_datadir so a data dir
  that disagrees with config.yaml refuses before tn_inst_prepare changes anything
- test/k3s-ca/compat/assert.sh: the drop-in is expected for an installed k3s 1.34 (written from v1.31),
  and none for 1.30
- stale strings: agent header and troubleshooting say /static/teknoir/k3s-version first; cgroup flag
  from v1.31; fleet note: a new agent stops until the server script is re-run (it also repairs a
  legacy CA) or INSTALL_K3S_VERSION is given
- tests: the caller's fail-cgroupv1 reaches the installer, order, data dir guard first

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir @claude
fix(bootstrap): keep the cgroup v1 drop-in until the installer has pu…
3712031 
…t the flag in the unit

tn_inst_cgroup removed 99-teknoir-cgroupv1.yaml at tn_inst_prepare time whenever the
device's own fail-cgroupv1 was found, also when it was only in INSTALL_K3S_EXEC. That
reaches the unit only when the installer has run, so a k3s restart in the gap (the CA
step restarts k3s, or the device reboots after the script stopped) left a cgroup v1
k3s >= 1.35 without the flag.

- The drop-in is removed early only when the device's setting is in a persistent config
  file that k3s reads (config.yaml, config.yaml.d/*.yaml); files win over INSTALL_K3S_EXEC.
  With only INSTALL_K3S_EXEC the drop-in stays or is written (same value twice is harmless,
  the command line wins).
- Audit: writing the flag for an installed k3s older than v1.31 (about to be upgraded)
  would stop its kubelet on a restart before the installer. That write now waits for the
  new tn_inst_finish, called right after the installer in the server and agent scripts;
  an existing drop-in is removed at once in that case. The other removals (host on cgroup
  v2, release older than v1.31) are safe early.
- Library version 3. Tests: go, dash, and the openssl-compat image check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir @claude
docs(bootstrap): when the cgroup v1 drop-in is removed and when it wa…
c160448 
…its for the installer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir @claude
fix(bootstrap): a k3s upgrade across v1.31 gets the cgroup v1 flag on…
1d61656 
… its first start

When the drop-in has to wait for the installer (installed k3s older than v1.31) and the
release the installer will start is a channel or v1.35 or newer, --kubelet-arg=fail-cgroupv1=false
is appended to INSTALL_K3S_EXEC (once), so the new unit carries it from its first start and the
installer's systemctl restart does not fail before tn_inst_finish. The old kubelet never reads it.

Only an uncommented fail-cgroupv1=false (config.yaml, config.yaml.d/*.yaml and *.yml,
INSTALL_K3S_EXEC) counts as the device setting it itself; a comment or =true does not, and
does not remove the drop-in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tekn0ir tekn0ir mentioned this pull request Oct 11, 2026
Merged
@tekn0ir
tekn0ir merged commit 821c023 into main Oct 12, 2026
1 of 2 checks passed
tekn0ir added a commit that referenced this pull request Oct 12, 2026
@tekn0ir
Merge pull request #9 from teknoir/fix/k3s-version-channel
47d39cd 
k3s follow-up: channels via INSTALL_K3S_CHANNEL, *.yml data-dir (after #8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tekn0ir