Skip to content

[olcedar] d8 platform olcedar node add — ввод статической ноды одной командой - #450

Draft
borg-z wants to merge 6 commits into
mainfrom
olcedar-node-add
Draft

[olcedar] d8 platform olcedar node add — ввод статической ноды одной командой#450
borg-z wants to merge 6 commits into
mainfrom
olcedar-node-add

Conversation

@borg-z

@borg-z borg-z commented Aug 20, 2026

Copy link
Copy Markdown
Member

What

Adds d8 platform olcedar node add <address> --group <nodegroup> — one command in place of the five-step manual flow for introducing a static node running olcedar (systemType: Immutable):

# before
d8 k get nodeconfigtemplate worker -o yaml > template.yaml
curl http://<machine>:50000/inventory > machine.yaml
# uncomment the disk in machine.yaml
yq eval-all 'select(fi==0) * select(fi==1)' template.yaml machine.yaml > worker-0.yaml
yq -i '.metadata.name = "worker-0" | .spec.nodeName = "worker-0"' worker-0.yaml
curl -X PUT --data-binary @worker-0.yaml http://<machine>:50000/config

# after
d8 platform olcedar node add 10.12.0.40 --group worker

Neither side can do this alone: the cluster does not know the machine's disks, the machine does not know the cluster's bootstrap token. The CLI sees both — and, unlike a file, it can ask.

Everything it talks to is in main now: #21557 landed 25.08 (dhctl and node-controller, the immutable bootstrap) and #22212 on 08.09 (static bootstrap and immutable converge). So the aggregated nodeconfigtemplates API, the internal.deckhouse.io/v1alpha1 NodeConfig and the machine's :50000 contract are all upstream — this branch needs no unmerged deckhouse branch to run against.

It implements §6 "Следующий шаг — добавление узлов командой" of the static-bootstrap spec, with one deliberate difference: the spec sketched dhctl attach-node, and this is d8. The reasons are the same either way — one render on the node-controller side, the token never stored in an object, only in the body of the PUT.

Flow

  1. GET :50000/whoami — a machine already held by the node agent is refused by node name, resolved from its InternalIP. (A second push to an installed machine cannot work: the agent demands a maintenance token and answers 401.)
  2. Read nodeconfigtemplates.templates.internal.deckhouse.io/v1alpha1 for the group. A 404 is explained against the NodeGroup itself (missing / wrong nodeType / wrong systemType); a 503 is passed through as it is, because it means the cluster has not minted the group's bootstrap token yet — a "not yet", not a broken backend.
  3. GET :50000/inventory.json. An image too old to serve it is refused — there is nothing to pick a disk out of.
  4. Ask the disk, the network and the node name.
  5. Build the NodeConfig, PUT :50000/config, wait for the node to register.

Adopting a disk, and never erasing one by accident

init decides for itself whether anything is installed. resolveSystemDisk (images/init/src/0.1/disk.go) finds one disk carrying BOOT/CONFIG/DATA and answers install=false; initmode.go then computes provision := install || cfg.Spec.Storage.Wipe. So wipe: true is the one thing that can force a provision where none is needed — and that provision is destructive first and sourced second: provisionDisk erases, and only afterwards copy-boot-assets looks for the UKI and the rootfs under /run/media. A machine booted from its own disk (every disk cloned from an image, so every DVP node) has no such media: the erase succeeds, the copy fails, and nothing is left to boot from. This is defect B1 of the spec — "wipe: true на однодисковой установке фатален", to be refused at the entrance rather than inside dhctl.

So this command:

  • adopts a disk that already carries the layout — the document names no storage at all, because init identifies that disk itself, and a selector disagreeing with the pin writeDiskPin recorded at install is itself a reason for it to reinstall;
  • writes wipe only from an explicit --wipe, which reads as "reinstall, from installation media" and says what it costs without one;
  • calls out a second disk carrying the layout: the node identifies its disk by that layout before it reads any selector, so it would take that one and leave the named disk alone.

What those notes warn about cannot be settled from an inventory — diskState spells installer media and an installed system the same way (inventory.go:101), and only the machine knows which disk it booted from. Teaching the inventory that fact belongs to the initramfs repository, where bootedDisk() already computes it.

The three documents this produces:

# blank disk, no --wipe                 # layout already there (adopt)      # --wipe
storage:                                storage: {}                         storage:
  diskSelector:                                                               diskSelector:
    serial: S3Z8NB0K700002                                                      serial: ZA20ABCD
                                                                              wipe: true

The rest of what is never decided silently

  • The single blank disk is the default but still confirmed; a machine with one disk offers it as the default, since there is no choice to make. Two blank disks and no answer → refusal naming --disk-selector.
  • The disk is named by the most stable attribute it reports (wwidserialbusPathname), so the document survives sda becoming sdb.
  • The interface the CLI reached the machine on is the default, proven by the connection itself. An address belonging to no interface (a port forward, a NAT) is asked about, or named with --network-interface.
  • DHCP unless the operator pins the current address; --network dhcp|static decides it up front.
  • Without a TTY and without --yes the command refuses instead of choosing.

Secrets

The template carries four live secrets on every read: kubelet.bootstrapToken, registry.auth, registryPackagesProxyAccessTokenB64 and statusToken — the last minted per read as the bearer for the node's :50000 status port, and deliberately not marked sensitive upstream (the apiserver would answer <omitted> and the carry-over would write that back, destroying the token). Nothing else hides it, so this command does: the document is assembled in memory, is never logged or written to a file, and --dry-run prints it with all four redacted. The test counts the redactions off redactedPaths and asserts the fixture values are absent, so a fifth secret added upstream fails it rather than leaking quietly.

Non-interactive

d8 platform olcedar node add 10.12.0.40 --group worker \
  --name worker-1 --disk-selector serial=S3Z8NB0K700002 --network dhcp --yes

--group completes to Static+Immutable NodeGroups only — a group that cannot take a machine is never offered.

Bounded reads, and a visible wait

A live run hung with nothing on the screen: the machine had already answered /whoami, and the command was sitting in the template read, which the kube-apiserver proxies to node-controller and which client-go leaves unbounded. So every cluster read is now capped at 30s, each network step announces itself before entering it, and the wait reports every 30 seconds. The command closes with where the time went:

Timings
  read the cluster and the machine      1.2s
  pushed the configuration              0.3s
  node registered                      1m24s
  machine time in total                1m24s

The operator's own thinking time sits between the read and the push and is deliberately not counted.

On not importing dhctl

The machine-facing HTTP contract (/whoami, /inventory.json, /config) and the inventory types mirror dhctl/pkg/immutable rather than importing it. CheckDocumentAgainstInventory there reaches dhctl/pkg/config and dhctl/pkg/global, so importing it means pulling dhctl's whole dependency tree into the CLI for some 250 lines of net/http. The copies carry a mirror comment naming the original.

Its disk-vs-inventory check is not ported wholesale either: TemplateStorage.render blanks spec.storage and spec.network before serving, so both halves are filled in by this command out of the inventory it just read — checking them back against it checks the CLI against itself. What is ported is the part that catches a live mistake: a --disk-selector from a human must resolve to exactly one disk.

Tests

go test ./internal/olcedar/... ./internal/system/...226 passed in 44 packages; golangci-lint run ./internal/olcedar/... → 0 issues. Among them:

  • refusal on agent,
  • refusal without a TTY and without --yes,
  • worker-0 taken → worker-1 offered (and hole-filling, and other groups ignored),
  • a system-layout disk adopted with no storage in the document,
  • wipe present only when --wipe was passed,
  • the missing-media and second-layout-disk warnings,
  • a 503 template read passed through undecorated, a non-404 failure naming the aggregated API,
  • every fixture secret, statusToken included, absent from --dry-run output.

Still open

  • The live run on a stand is not finished: the machine took the configuration on 21.08, but no run has been carried through to a registered node with the current build. That is what the draft state is for.
  • Waiting is still "does a Node object exist". Now that every document carries a statusToken and the node serves :50000/status, the wait could ask the machine what it is doing instead of polling the cluster blindly — a follow-up, not this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_018aJFD4wEg66sX5yx4Sk6kd

borg-z and others added 5 commits September 9, 2026 14:44
Adding a static node running the immutable OS took five steps by hand: read
the NodeGroup template, curl the machine inventory, uncomment a disk, merge
the two documents with yq, and PUT the result back. Neither side could do it
alone — the cluster does not know the machine's disks, the machine does not
know the cluster's bootstrap token — but the CLI sees both, and unlike a file
it can ask.

The command reads the aggregated NodeConfigTemplate of the group and the
machine's /inventory.json, asks which disk, which network and which node name,
and pushes the assembled NodeConfig to the machine's maintenance port.

Nothing is decided silently. The single blank disk is offered as a default but
still confirmed; a disk that already holds a system is refused until the
operator confirms the erase (or passes --wipe); without a terminal and without
--yes the command refuses rather than choosing. A machine whose port is held by
the node agent is refused by node name: a second configuration would replace
the one a working node runs on.

The template carries a live bootstrap token, the registry auth and the
registry-packages-proxy token on every read, so the document is built in
memory, never logged, and --dry-run prints it with those three fields redacted.

The machine-facing HTTP contract (/whoami, /inventory.json, /config) mirrors
dhctl/pkg/immutable of the deckhouse repository rather than importing it: the
dhctl module would pull its whole dependency tree in for some 250 lines.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: borg-z <me@zykov.tatar>
…sing it

The first cut read `system-layout` as "an OS is here, the install will erase
it" and wrote `spec.storage.wipe: true` once the operator confirmed. On a
machine whose disk was cloned from an image — every DVP node — that is not a
confirmation, it is a self-destruct.

init decides for itself whether anything is installed:
`resolveSystemDisk` (images/init/src/0.1/disk.go) finds one disk carrying
BOOT/CONFIG/DATA and answers `install=false`, and initmode.go computes
`provision := install || cfg.Spec.Storage.Wipe`. So `wipe: true` is the one
thing that can force a provision where none is needed — and the provision is
destructive first and sourced second: `provisionDisk` erases, and only then
`copy-boot-assets` looks for the UKI and the rootfs under /run/media. A machine
booted from its own disk has no such media, so the copy fails after the erase
and there is nothing left to boot from.

So:

  - a disk that already carries the layout is adopted, not installed onto, and
    the document names no storage at all: init identifies that disk itself, and
    a selector disagreeing with the pin recorded at install (writeDiskPin) is
    itself a reason for it to reinstall;
  - `wipe` reaches the document only from an explicit --wipe, which now reads as
    "reinstall, from installation media", and says what it costs without it;
  - a second disk carrying the layout is called out: the node identifies its
    disk by that layout before it reads any selector, so it would take that one
    and leave the named disk alone.

What the notes warn about cannot be settled from an inventory: `diskState`
spells installer media and an installed system the same way (inventory.go:101),
and only the machine knows which disk it booted from. Adding that fact to the
inventory belongs to the initramfs repository, where bootedDisk() already
computes it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: borg-z <me@zykov.tatar>
A live run on zykov-st read as one paragraph: the answer to a question, the
note about the disk and the next question all ran together, and the wait that
followed said nothing for minutes.

So the prompt sets its blocks apart itself — a blank line after a list, before
a question, and around a note — rather than leaving each caller to remember a
newline. A machine with one disk now offers it as the default: there is no
choice to make, and the list still shows what the disk is.

The wait says it is still waiting every 30 seconds, and the command closes with
what the addition cost:

  Timings
    read the cluster and the machine      1.2s
    pushed the configuration              0.3s
    node registered                      1m24s
    machine time in total                1m24s

The operator's own thinking time sits between the read and the push and is
deliberately not counted: the number worth knowing is how long the machine
takes, not how long someone stared at a disk list.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: borg-z <me@zykov.tatar>
A run against 10.12.0.40 hung with nothing on the screen at all. The machine
was fine — it had already answered /whoami — and the command was sitting in the
template read: nodeconfigtemplates is served by an aggregated API, so the
kube-apiserver proxies it to node-controller, and client-go carries no timeout
of its own. A backend that never answers hangs the command forever.

So the rest config now bounds every cluster read at 30 seconds, and each
network step says what it is about to do before it does it. Two silent minutes
become two lines and, at worst, one bounded failure.

That failure names where the answer comes from, because the kube-apiserver is
almost never the part that is broken:

  read the node configuration template of worker: <err>. It is served by an
  aggregated API, so this read is proxied by the kube-apiserver to
  node-controller of node-manager: a node-controller that is down, unreachable
  or unregistered fails it. Check it with:
  d8 k get apiservice v1alpha1.templates.internal.deckhouse.io

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: borg-z <me@zykov.tatar>
Rebased onto main, where the whole immutable bootstrap now lives: #21557 landed
25.08 and #22212 on 08.09. Two things the template gained since this branch was
written:

  - spec.statusToken. The render mints one on every read; it is the bearer for
    the node's :50000 status port afterwards, and it is deliberately not marked
    sensitive — the apiserver would answer "<omitted>" and the carry-over would
    write that back, destroying the token. So nothing upstream hides it, and
    --dry-run printed it in full. It joins the redacted set, which the test now
    counts off redactedPaths rather than a literal 3.

  - 503 from the template read. The cluster mints the group's bootstrap token
    itself, and until it has, the render answers ServiceUnavailable with "read
    this again once the cluster has issued one". That is a "not yet", not a
    broken backend, so it is passed through as it is instead of being dressed
    up with the aggregated-API diagnosis a real failure gets.

Without the first fix TestRedactHidesEverySecretTheTemplateCarries fails on the
document it just built.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018aJFD4wEg66sX5yx4Sk6kd
Signed-off-by: borg-z <me@zykov.tatar>
@borg-z borg-z changed the title [olcedar] d8 platform olcedar node add for static immutable nodes [olcedar] d8 platform olcedar node add — ввод статической ноды одной командой Sep 9, 2026
CI runs golangci-lint v2.11.4 with the release build tags, and it caught five
things a different local version does not: the olcedar import sitting in a
group of its own (gci), the clientset SetupK8sClientSet returns being taken and
immediately thrown away (ineffassign), and three statements pressed against the
brace above them with no variable in common (wsl_v5).

The clientset is now not taken at all: the config it comes with carries no
timeout, so the client is built from the bounded config instead.

Checked the rest of the branch for the same three classes. The three remaining
defers each sit under the line declaring what they close, which is what wsl_v5
allows and why CI did not name them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018aJFD4wEg66sX5yx4Sk6kd
Signed-off-by: borg-z <me@zykov.tatar>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant