Documentation

Cirrus

The operator that runs web applications on Community Cloud — one WebApplication resource for an app's database, bucket, queue, hostnames and GitLab releases, and DnsProviders for publishing them.

What Cirrus does

Cirrus runs web applications on Community Cloud. A WebApplication says what an application needs, and Cirrus builds it out of the services the cluster already runs:

Asked for What Cirrus makes Through
always A Deployment and a Service Kubernetes
hostnames A ListenerSet on cc-letsencrypt-web-gateway, a route, an HTTP→HTTPS redirect, and a certificate The Gateway API, cert-manager
postgres: {} A role and a database in the namespace's own Postgres, cirrus-postgres CloudNativePG
bucket: {} A bucket, an S3 user limited to it, and its keys The SeaweedFS operator
rabbitmq: {} A vhost, and a user limited to it RabbitMQ's topology operator
migrations A Job, run with each new image before the rollout Kubernetes
source.gitlab A project webhook; each successful pipeline on ref rolls out GitLab's API

It also runs ExternalDNS for each DnsProvider, so a namespace's hostnames are put in DNS without anyone editing records.

Meteorologist is the web console for all of this: it creates and edits WebApplications through a form, and sets up namespaces, GitLab connections and DNS.

Installing it

Cirrus is the installer's cirrus package. It runs in the community-cloud namespace, and enabling it brings in the services it builds on — cert-manager, CloudNativePG, SeaweedFS and the RabbitMQ operator:

"packages": { "cirrus": true }

GitLab's webhooks reach Cirrus at values.domain.cirrus. If they can't, Cirrus still finds new pipelines by asking GitLab every ten minutes.

A WebApplication

apiVersion: community-cloud.technology/v1alpha1
kind: WebApplication
metadata:
  name: shop
  namespace: acme
spec:
  image:
    repository: registry.gitlab.com/acme/shop
    tag: main
  port: 3000
  healthCheck:
    path: /up
  hostnames: [shop.acme.example]
  postgres: {}
  bucket:
    quota: 5Gi
  rabbitmq: {}
  migrations:
    command: ["bin/rails", "db:migrate"]
  source:
    gitlab:
      project: acme/shop
      ref: main
      tokenSecret:
        name: gitlab-token     # key "token" by default
      imageTag: "{commit_sha}" # or "{short_sha}", "{ref}", combined freely

The least there is, an image on two hostnames with no source:

apiVersion: community-cloud.technology/v1alpha1
kind: WebApplication
metadata:
  name: brochure
  namespace: marketing
spec:
  image:
    repository: ghcr.io/acme/brochure
    tag: "2.4.1"
  port: 80
  hostnames:
    - acme.example
    - www.acme.example

spec also takes replicas, env, resources and imagePullSecrets, as a Deployment would.

Checking on it

kubectl get webapp shows the deployed image, readiness and address. kubectl describe webapp shop shows its conditions and what each is waiting on:

Condition What it means
Provisioned The database, bucket, queue and routes it asked for exist
RolledOut The newest release is running. False names the migration Job that failed
WebhookRegistered GitLab has a webhook for the project
ReleaseChecked Cirrus could ask GitLab for the newest release. It tries again within a couple of minutes, and meanwhile runs the last release it found
Ready Both Provisioned and RolledOut

What the application is given

Nobody copies passwords around. Each service the app asks for reaches it as environment variables:

  • Postgres: the Secret <app>-postgres, loaded whole, with DATABASE_URL, PGHOST, PGPORT, PGDATABASE, PGUSER and PGPASSWORD. The role and the database are both called <app>, with any - as _.
  • Bucket: S3_ENDPOINT, S3_BUCKET, S3_REGION, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, and the AWS_* equivalents that AWS's SDKs read on their own. The bucket is called <namespace>-<app>. Address it by path, not by virtual host.
  • RabbitMQ: the Secret <app>-rabbitmq, loaded whole, with AMQP_URL and RABBITMQ_HOST, _PORT, _USERNAME, _PASSWORD and _VHOST.

Anything in spec.env comes after these, so it wins.

Postgres

Each namespace has a Postgres of its own rather than a share of cc-postgres: a CloudNativePG Cluster, cirrus-postgres, made when the first application in the namespace asks for a database and shared by every later one. Keeping it in the namespace keeps it with the application. A namespace pinned to a region has every pod in it pinned there, Postgres's included, so an application never queries a database in another region.

Cirrus makes the Cluster once and then leaves it alone. Grow its volume, add instances or set up backups as on any CloudNativePG Cluster, and Cirrus won't put them back. It isn't owned by any application, so it outlives the last one; delete it by hand when the namespace is done with it.

The namespace

Two of Community Cloud's capabilities are labels on a namespace. Cirrus checks for them rather than adding them itself:

Label Needed for
whereWereGoing: "weDontNeedRoads" hostnames — the cluster Gateway refuses listeners from any other namespace
storage.community-cloud.technology/buckets: "true" bucket — SeaweedFS refuses buckets from any other namespace

Without the label, Provisioned is False and says which label is missing. Meteorologist shows these as switches on each namespace.

Releases from GitLab

GitLab has no webhook for an image being pushed, so Cirrus listens for successful pipelines on the tracked ref instead. When one arrives:

  1. Cirrus checks the hook's token, which is the application's own and is kept in <app>-cirrus-webhook.
  2. It works out the tag from imageTag and asks GitLab's registry API for the tag's digest. An image outside the project's GitLab registry is rolled out by tag.
  3. It runs the migration Job, <app>-migrate-<hash>, with the new image. The old version keeps serving while the Job runs.
  4. Once the Job completes, the Deployment gets the new image, by digest.

If the migration fails, RolledOut is False with the Job's name, and the old version stays. Each image's migrations run once; to rerun them, delete the Job.

Webhooks can get lost, so Cirrus also checks for the newest successful pipeline every ten minutes. Until a release arrives, the application runs spec.image.

The GitLab token in tokenSecret needs the api scope and the Maintainer role on the project, so Cirrus can manage the project's webhook and read its registry. A group access token is usually the best choice: it reaches every project in the group and keeps working whoever leaves.

Deleting an application

Everything in the application's namespace is owned by the WebApplication and is removed with it. What's in cc-office — the RabbitMQ vhost, user and permission — and the GitLab webhook are removed by Cirrus's finalizer.

Data is kept. CloudNativePG keeps the database, the namespace's Postgres stays, and SeaweedFS keeps the bucket. Removing a postgres, bucket or rabbitmq block from a live WebApplication stops Cirrus maintaining those resources but doesn't delete them.

Publishing DNS

A DnsProvider is one DNS provider account, the domains it holds, and the namespaces whose hostnames it publishes. It's cluster-scoped, since which namespace may write which domain is the cluster's decision, not a namespace's:

apiVersion: community-cloud.technology/v1alpha1
kind: DnsProvider
metadata:
  name: acme
spec:
  provider: cloudflare          # any ExternalDNS provider
  domains: [acme.example]
  excludeDomains: []            # parts given to another provider
  namespaces: [acme]            # one ExternalDNS each
  credentialsSecret: cirrus-dns-acme   # the default, in external-dns
  target: 203.0.113.7           # optional: what records point at
  policy: sync                  # or upsert-only, which never deletes

For each namespace Cirrus runs an ExternalDNS in external-dns, cirrus-dns-<provider>-<namespace>, that reads only that namespace's routes and writes only the provider's domains. Every key of the credentials Secret becomes one of its environment variables, named as the provider's documentation names them. The credentials live in external-dns, where no application runs, so access to a namespace doesn't mean being able to read the token:

credentials
$ kubectl -n external-dns create secret generic cirrus-dns-acme \
--from-literal=CF_API_TOKEN=...

A DnsProvider runs nothing, and says why in its Ready condition, when:

  • Invalid — it has no provider or domains, or an unknown policy.
  • DomainConflict — another DnsProvider publishing different namespaces holds the same domain, or one containing it without excluding it.
  • MissingCredentials — the credentials Secret isn't in external-dns.

DnsProviders sit beside the installer's own dns section without touching its records. Use the installer for the cluster's own domains, and DnsProviders for the domains of the people using it — which can then be added from Meteorologist without re-running a bundle. Don't give the same domain to both: neither sees the other's.

Deleting a DnsProvider removes its instances. With the default sync policy, the records they made are left behind, since nothing is left to delete them.

Not yet

  • Redis. The shared Redis is divided by database number, and handing out numbers safely needs a design of its own.
  • Forges other than GitLab.
  • Leader election. Cirrus runs as one replica.

The operator's own README, with its configuration and development notes, is operators/cirrus in the repository.