Networking

Cilium as the CNI and Gateway API implementation, load balancer addresses for the outside world, certificates, and meshes for nodes that can't reach each other directly.

Overview

Area	How it's done
CNI	Cilium, with its eBPF datapath replacing kube-proxy
Gateway	The Kubernetes Gateway API, served by Cilium's own Envoy
Load balancer addresses	Cilium IP address management, with L2 announcements where a router forwards to an address nothing holds
Certificates	cert-manager with Let's Encrypt, using HTTP-01 challenges through the Gateway API
Cross-site	A Nebula mesh through Defined Networking, or Tailscale
Ingress	Dedicated gateway nodes, selected by the `gateway` role
Kernel	BBR congestion control over fair queueing

Cilium

Cilium replaces a good deal of what K3s ships with. The installer starts K3s with --disable=traefik --disable=servicelb, installs the Gateway API resources itself at a pinned version, and then installs Cilium — in that order, because Cilium's operator builds its Gateway API controller once at startup from whatever it finds then.

Cilium's Helm values live in k8s/networking/cilium/Cilium.values.yaml. The CNI, the kube-proxy replacement, the Gateway API implementation, and load balancer IP management all come from there. Versions are set in network.cilium:

$ community-cloud run-bundle cilium all cc.config.json

The Gateway

The cluster's default Gateway, k8s/networking/gateway/Gateway.yaml, is served by Cilium's Envoy and carries the Let's Encrypt listeners. It uses allowedListeners, so only namespaces carrying the right label may attach listeners to it. Applications reach the outside through this shared Gateway rather than one of their own, each with an HTTPRoute.

Reaching the cluster from outside

How the outside world reaches the cluster is set in network.gateway. The gateway bundle turns it into a Cilium address pool, and on a home network an ARP announcement policy as well. There are two modes:

Mode	When to use it
`floating`	The address is already on an interface of a gateway node, routed there by a provider — typical of a VPS or colocated machine.
`port-forward`	A router maps ports to a LAN address that nothing holds, and Cilium answers ARP for it — typical of a home or office network.

"network": {
  "gateway": {
    "mode": "port-forward",
    "addresses": ["192.168.1.240-192.168.1.250"]
  }
}

Mark the nodes the outside world can reach with "gateway": true. They get the gateway role, and ingress is scheduled onto them.

Certificates

cert-manager issues certificates from Let's Encrypt, using HTTP-01 challenges answered through the Gateway API. Its Helm values are k8s/certs/CertManager.values.yaml, and k8s/certs/ClusterIssuer.yaml is the cluster-wide issuer. The cc-office namespace also has an issuer scoped to it.

Cross-site networking

Nodes in different places often can't reach each other directly. Community Cloud supports two meshes for that:

Nebula, through Defined Networking. The installer creates roles and firewall rules, and enrols each node. Configure it in network.nebula, with per-node settings — lighthouses, relays, static addresses — on each entry in nodes.
Tailscale, through the tailscale-operator package and a proxy in k8s/networking/tailscale/.

$ community-cloud run-pipeline nebula-network,nebula all cc.config.json

Mesh membership is deliberately left alone when a node is uninstalled or cleaned, since it's usually how you're reaching the machine.

Kernel networking

The network bundle sets TCP congestion control to BBR over the fq queueing discipline on every node. BBR holds throughput up over links that lose the odd packet, which the default CUBIC reads as congestion — exactly the sort of links a multi-site cluster runs over.

$ community-cloud run-bundle network all cc.config.json

Documentation