Commands

Most commands that change things share a set of options:

Installing, adding nodes, and taking them out are covered in Cluster Operations.

Commands and bundles

A command is one step: a shell command on a node, or an HTTP call to an API. A bundle is an ordered list of them aimed at one node, sharing a context so that what one finds is available to the next. The GPU bundle detects hardware, and the labelling bundle puts the right role on the node because of it.

Every command declares two things about itself.

Where it runs. Most belong on the node the bundle is aimed at. Anything using kubectl doesn't, because only a server has a kubeconfig, so it says runOn: ControlPlane and the runner opens the connection it needs. That's what lets a bundle read the hardware on an agent and then label it — impossible from the agent itself.

What it's for, which is mostly whether it changes anything:

doctor and preflight are built on that: they ask a bundle for only the purposes that change nothing, so they're safe to point at a cluster in service.

Which nodes a step runs on

Two different questions look like one on a single node:

Labelling a node shows they're separate: the shell is kubectl so it has to run on a server, and it is nevertheless one command per node.

A command that says nothing gets cluster if it runs on the control plane and each-node otherwise.

Bundles

List them, or the commands inside one, with list-bundles:

Teardown bundles

These take a cluster apart rather than building one. Their checks assert the opposite of a working cluster — "K3s isn't running here", "this node isn't a member" — so a check-up that asks about everything leaves them out, and doctor --bundle <name> runs them when the question is meant.

Checking a cluster

preflight

Before anything is installed, preflight says what would stop an install — no server, two servers, a missing token, a node that can't be reached, a gateway with no addresses.

doctor

Once a cluster exists, doctor asks it how it's doing:

It runs on the same machinery as an install and asks every node at once. A cluster is several machines: asking them one after another makes the report a description of six different moments, and the first slow node delays every answer behind it.

Two things are relaxed, because the questions are read-only. A node that can't be reached doesn't stop the others being asked — a cluster with a machine down is exactly the case you'd run this for — and one failure doesn't stop the run, because one thing being broken is the most likely reason to want to know what else is.

doctor also compares the configuration recorded in the cluster against your file, and reports anywhere they've drifted apart.

Embedded manifests

The manifests and values files under k8s/ are compiled into the binary as a SQLite database, so a released executable can apply them on a machine that has never seen the repository. They're referred to with an embed:// URL, and the paths have no k8s/ prefix:

bun run build rebuilds src/generated/embedded.sqlite before compiling, so anything added under k8s/ is picked up automatically. To rebuild it on its own:

The database is built from scratch each time and the rows go in sorted, so the same tree always produces the same bytes. Only .yaml and .yml files are embedded.

Secrets in commands

Anything a command is given on its command line is readable by every user on that node through a process listing, and gets quoted back in error messages and logs. Credentials go over standard input instead, which only the process reading it ever sees.

A command can take a payload on stdin — a registry password, a rendered manifest — or declare secretEnv, which travels over standard input and is exported by a preamble on the far side. The K3s token, registry credentials, and Nebula enrolment codes all go that way.

{
  name: "install-something",
  description: "Run an installer that reads a token from its environment",
  env: { INSTALL_VERSION: "1.2.3" },
  // Read and exported by a preamble on the far side, so the value
  // never appears in an argument list
  secretEnv: (config) => ({ TOKEN: config.getToken() }),
  command: "sh ./install.sh",
}

A few rules follow from a command having only one standard input:

• stdin and secretEnv can't both be set on one command
• secretEnv values have to be single lines; a document belongs on stdin
• secretEnv can't be combined with sudo, which clears the environment it was given

Writing commands

A command is a string, an array of strings, or a function returning either. Nearly all of them are several lines of shell, so the array form is the usual one and the lines are joined for you:

{
  name: "wait-for-service",
  description: "Wait for the service to come up",
  command: [
    `for attempt in $(seq 60); do systemctl is-active --quiet ${SERVICE} && break; sleep 1; done`,
    `systemctl is-active --quiet ${SERVICE} || { echo "${SERVICE} never came up" >&2; exit 1; }`,
  ],
}

The lines are joined with a newline rather than "; ", which is what shell keywords taking a body straight after them require — for x in y; do; echo $x; done is a syntax error, and an easy one to write when a join is putting the separators in.

Source layout

installer/
├── scripts/
│   ├── build.ts               # Cross-compiles for eight targets
│   └── generateEmbedded.ts    # Compiles k8s/ into a SQLite database
└── src/
    ├── cli/
    │   ├── installer.ts       # Entry point, commander definitions
    │   ├── commands/          # What can be done to a node or cluster
    │   └── configure/         # The interactive configuration builder
    ├── pipeline/              # Planning and running steps in lock step
    ├── plugins/               # Finding, approving and loading plugins
    ├── runners/               # What drives the commands
    ├── remote/                # SSH, and ~/.ssh/config resolution
    ├── topology/              # The fixed region list
    ├── util/                  # Config, templating, shell, redaction
    └── generated/             # The embedded manifest database

Development

Tests live in src/test, use the node:test API with node:assert, and reach nothing outside the repository — no live node, no cluster, no cc.config.json. They run under bun test, which implements node:test faithfully; Node itself can't currently run them, since the codebase uses enum and reads the embedded manifests with bun:sqlite.

Two scripts in src/test/manual talk to a live machine, so they're run by hand: