Installing a cluster

Everything happens in lock step: every node finishes a step before any node starts the next. Within a step the nodes work at once, so installing K3s on six agents takes about as long as installing it on one.

The barrier is the point. Without it a failure leaves the cluster smeared across the install — one machine with storage and a CNI, another three steps behind with half a package list. With it, a failure means every node is at the same place, which is a state you can look at, reason about, and resume from by fixing the cause and running the install again.

Before it connects to anything

The whole plan is worked out first, so a misspelled bundle at the end of a pipeline fails immediately rather than twenty minutes in. --dry-run never opens a connection at all, so it works against a cluster that doesn't exist yet.

Two nodes that turn out to be the same machine stop the run before it starts. That's an easy configuration mistake — a copied block with the address changed and the port left alone, a username that overrides what ~/.ssh/config would have resolved — and without the check the first sign of it is a cluster missing a node nobody can account for.

Running part of an install

all is every node in the configuration; a name is one node. A single node is a cluster of one rather than a separate code path — the interesting multi-node bugs are the ones that only appear with more than one machine, and a separate path is how those hide.

run-bundle is run-pipeline with one bundle in it, and it's the same code: the barrier between steps, the refusal to start when two nodes turn out to be one machine, and the per-node reporting are all there whichever you ask for. A pipeline can also be one named in the configuration's pipelines section.

Running a bundle against a single agent still works when something in it needs kubectl. The agent has no kubeconfig, so the run borrows a reachable server from the configuration to talk to the cluster with — no step runs on its behalf.

Adding a node

Add the node to nodes in the configuration first. That file is the record of what the cluster is, and a node added without being written down is one that vanishes from the next install.

This is the install with the cluster-wide half taken out. Cilium, the Gateway, Helm and the charts have already happened and would either be re-applied for nothing or re-applied differently. What's left is everything that makes one machine a member:

cluster-online → preflight → base → network → gpu
              → registries → lvm → k3s → nodeLabels → node-joined

cluster-online fails first if there's no cluster to join, which is worth catching early — every step after it would fail in a way that blames the new node for a problem the cluster already had. node-joined is the cluster's own word that it worked: the node is registered and its kubelet reports Ready, which it doesn't do until the CNI has set the node up.

Agents only, for now. Growing the control plane means bringing the cluster's datastore from one member to several, which isn't something to do halfway, so add-node refuses a node whose type is server.

Taking a node out

This is the install read backwards, and only the part of it that belongs to one machine:

cluster-online → drain → k3s-uninstall → node-removed

Nothing here touches Cilium, the Gateway, Helm or the charts. Those are the cluster rather than the node, and removing them because one machine is leaving would take the cluster down in order to shrink it.

It keeps the data

That's the whole distinction between uninstall and clean, and it's why K3s's own uninstall script isn't used: k3s-uninstall.sh ends by deleting /etc/rancher/k3s, /var/lib/rancher/k3s and /var/lib/kubelet, which on a server is the cluster's datastore and its certificate authority. Removing the software and removing what it was looking after are two decisions.

What's left behind:

A node uninstalled this way can be put back with add-node and finds its images, its certificates and its volumes where it left them.

What is used from K3s's own scripts is k3s-killall.sh, the half that changes no data: it stops the service, kills every container and shim, and unmounts everything under /run/k3s and /var/lib/kubelet.

Draining first

Stopping K3s on a node kills whatever is running there. The pods come back somewhere else a minute or two later, once the cluster notices — fine for something stateless, not fine for a database being killed mid-write. Draining moves the work first and waits for it to land.

Two flags are deliberately not passed to kubectl drain. --force deletes pods belonging to no controller, and since nothing recreates those it turns "move this workload" into "delete this workload" without saying so. --disable-eviction ignores every disruption budget in the cluster, which is the thing draining exists to respect. So a drain that can't finish stops and says what held it up, and --skip-drain is how you decide to stop the workloads instead.

--drain-timeout <seconds> bounds the wait, which matters because a pod held by a disruption budget that can't be satisfied is never evicted at all.

Servers

A server is refused without --server, because uninstalling one destroys the cluster: the datastore, the certificate authority and the token all live on it, nothing can put them back, and every agent loses its control plane at once. It's the one operation in the installer that no other command can undo.

With the flag, the pipeline is just k3s-uninstall. There's nothing to drain to and nothing to be removed from.

Agents are taken out one at a time; all is refused, since draining every node at once would mean waiting out every timeout and then failing, having cordoned the whole cluster on the way. Take the agents out one by one and the server last.

When the cluster is already gone

Everything above needs a reachable control plane, to drain to and to remove the node from. If the cluster isn't there any more, the node-local half is a bundle like any other:

Cleaning up

clean is the other half of the decision: removing what the uninstall kept. There are two levels, because there are two sizes of loss.

By default it removes the K3s directories — the datastore, the kubeconfig, the registry credentials, the image cache, and what's left of the pod network. Painful, and all of it rebuildable by installing again.

--full also destroys the volume groups the installer created, which is where the data is: TopoLVM carves every persistent volume out of them, so this is the contents of every database and every upload on the node. Nothing can put that back, so --full on its own prints what it would destroy and stops:

cc-ssd-vg           931GB  4 logical volumes
    pvc-8f2a…         30GB
    …
Nothing was destroyed. The volume groups listed above hold what the
cluster stored on this node — run the same command again with --yes to
remove them.

Ownership comes from an LVM tag the installer wrote rather than from the group's name. A group called cc-ssd-vg is probably one of ours, and "probably" isn't the standard to apply to something about to be destroyed.

The --yes gate lives in the storage-remove bundle rather than in the command, so reaching it another way — run-bundle, a pipeline in the configuration — is no shorter a path. --keep-going doesn't get past it either.

Why it won't clean a running node

Cleaning refuses while K3s is running, and while anything is still mounted under /var/lib/kubelet or /run/k3s. The second matters most: rm -rf /var/lib/kubelet with a volume still mounted underneath descends through the mount point and deletes what's in the volume — exactly the data every other decision here is arranged to protect. An uninstall unmounts those, so the usual answer is to run one first.

What's left alone

Cilium's datapath is removed by clean and reported by k3s-uninstall rather than removed. The cilium_* interfaces, the veth pairs and the eBPF programs belong to Cilium rather than K3s, and a node going back into a cluster may well want them. A reboot clears them either way.

A node's mesh membership — Nebula or Tailscale — is left alone by both. It's usually how you're reaching the machine.