Overview

Community Cloud offers two complementary storage layers:

Local storage is deliberately local rather than distributed: it's simpler, it's faster, and most applications would rather have speed than the ability to move a volume between machines. Anything that needs its data in more than one place — a database with replicas, object storage — handles that itself.

Local storage with LVM

Local storage is built on the Logical Volume Manager (LVM), which provides a great deal of flexibility:

• Grow volumes as data grows
• Encrypt all data
• RAID, for resilience even in single-node setups
• Live migration of volumes for zero-downtime upgrades

TopoLVM is a CSI driver that hands a pod a logical volume from a volume group on the node it's scheduled to.

Device classes

Storage performance should be predictable rather than a lottery. A database asking for fast storage shouldn't land on a spinning disk because that's what had room. So volumes are carved out of a volume group chosen for what the disks in it actually are:

k8s/storage/TopoLVM/TopoLVM.values.yaml is the source of truth. The installer reads the device classes out of it and creates exactly those volume groups, so adding a class there is all it takes.

Storage classes

Each device class is exposed through a Kubernetes StorageClass, so mapping a service to the right storage is a matter of naming a class:

Both use XFS, Retain, allowVolumeExpansion, and WaitForFirstConsumer — which matters for local storage, since the volume can only be made once the pod's node is known.

cc-local-ssd-fast is the cluster default, so a PVC that doesn't ask for a class gets fast local storage. Services in Community Cloud use sensible classes by default: databases land on solid state.

Retain means deleting a PVC leaves the volume behind. Reclaiming it is a manual step, on the grounds that losing data is worse than tidying up by hand.

How the volume groups get made

The lvm bundle looks at every block device on a node, works out which are free, sorts them by what they are, and builds the groups:

A disk is only touched if it is demonstrably unused: no partitions, no filesystem signature, not mounted, nothing holding it, not removable, not read-only. Anything else is reported with the reason and left alone. There's no flag to override that — if you meant to reuse a disk, wipe it yourself first.

Disks are sorted by hardware — NVMe to ssd, other solid state to ssd-sata, spinning to hdd — and which disks are considered can be narrowed or reclassified per node with the storage configuration.

Everything the installer creates carries LVM tags: community-cloud on both the physical volumes and the volume groups, plus cc-class-<name> saying which device class it serves. Ownership is a property of the object rather than a guess from its name:

Which nodes offer storage

TopoLVM's controller, lvmd, and node components all schedule only onto nodes labelled node-role.kubernetes.io/storage-local=storage-local, which the installer applies from the roles on a node. A node without that role isn't running out of storage — it simply doesn't offer any.

Nodes that can't serve every class

Not every machine has one of each kind of disk, and a Community Cloud is usually built from whatever hardware people have. A node with SSDs and nothing spinning is a perfectly good node for the classes it does have.

lvmd disagrees. It's handed a list of device classes and refuses to start unless every volume group in that list is on its node, so one missing group means the node serves none of them and crash-loops:

"msg":"volume group not found","volume_group":"cc-hdd-vg","error":"not found"

Until that's fixed upstream, the installer works around it. Once a node's volume groups exist, the lvm bundle records which device classes it can serve and nodeLabels puts them on the node as a single label:

storage.community-cloud.technology/classes=hdd.ssd

Installing the charts then reads those labels back and renders one lvmd configuration per distinct combination, each offering only the classes those nodes actually have. So run the bundles in that order, and re-run helm-charts after the storage on a node changes:

A cluster where nothing has been labelled yet — a first install — offers every node every class. When a node's shape does change it moves between lvmd DaemonSets, and topolvm-node may crash once or twice while the socket is absent before reconnecting on its own.

On ssd-cache

LVM can put an SSD in front of a spinning disk, and the ssd-cache device class is reserved for that. It isn't implemented because TopoLVM can't deliver it: a device class has no cache support of any kind, and an LVM cache attaches to exactly one logical volume, so it can't serve a class that creates volumes on demand.

Doing it properly means caching below LVM, where TopoLVM can't see it — bcache or dm-writecache over the spinning disk, with the volume group on top. Every volume in the group is then transparently cached. That's a kernel module and a device format, so it's a decision rather than a detail.

Object storage with Garage

Garage is S3-compatible object storage built for exactly this shape of cluster — a handful of machines in different places, on ordinary connections. Its Helm values are in k8s/storage/Garage/GarageValues.yaml.

It currently runs as a single replica using cc-local-ssd-fast for metadata and data, with the S3 API on NodePort 3900 and the web interface on 3902. For anything beyond a single node, raise replicaCount and set replicationFactor to match.

Applications request buckets through ObjectBucketClaim manifests alongside their other resources.