johnandersen777.bsky.social/socialweb-computer-kcp

Switch repository

Loading…

# 03 - Jobs

Audience: an operator running GitLab CI jobs on a tenant k3s cluster.

Outcome: a GitLab pipeline runs its jobs as Pods in the tenant cluster, one job per node, and the
cluster returns to zero nodes when the work stops.

Prerequisite: [02](02-gitlab-runner.md) done, so a runner points at the tenant cluster.

## The image a job runs in

A job Pod's image comes from the project's own `.gitlab-ci.yml`:

```yaml
  image: "$DOCKER_REPO:builder-$DOCKERFILE_HASH"
```

`DOCKER_REPO` is a variable in that file. `DOCKERFILE_HASH` comes from the included
`.ci/dockerfile-hash.yaml`, which is generated:

```bash
printf "variables:\n  DOCKERFILE_HASH: `git hash-object docker/build.Dockerfile`" > .ci/dockerfile-hash.yaml
```

The image is named by a hash of the Dockerfile. Change the Dockerfile, get a different image name,
and the old name stays valid.

### Check the image exists before running anything

Do not find a missing image as `ImagePullBackOff` twenty minutes into a node boot. The manifest
endpoint needs a token even for a public repository:

```bash
REPO=streamplace/streamplace
TAG=builder-3e14c389b80a359787d73741f221fdfaacb12f53

TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:${REPO}:pull&service=ghcr.io" \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')

curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer ${TOKEN}" \
  -H 'Accept: application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.v2+json' \
  "https://ghcr.io/v2/${REPO}/manifests/${TAG}"
```

`200` means the job will pull. `404 MANIFEST_UNKNOWN` means the job cannot start, whatever else is
right about the cluster.

### If the branch names an image nobody published

A branch can point `DOCKER_REPO` at a tag no job ever published, for example a `public.ecr.aws`
repository that answers `NAME_UNKNOWN` for the repository itself. Every job in that file is then
unrunnable as written. Read the upstream branch's `variables:` block before concluding an image is
missing: a stale branch and a missing image look identical from the failure.

Two ways to make the branch runnable, in order of cost:

1. **Move the branch.** Rebase onto the branch that publishes and use its own
   `.ci/dockerfile-hash.yaml`. Its `DOCKER_REPO` and hash already agree with each other.
2. **Build it and host it.** `docker build --target builder -f docker/build.Dockerfile .` at the
   hash the branch names, push to a registry the guest can reach, point `DOCKER_REPO` at it. The
   guest has outbound internet, so a public registry works and a private address does not without a
   route and a `registries.yaml`.

An image that resolves is not the same as a toolchain that builds. A builder image old enough to
predate a language feature the project uses fails inside the job, for example a Rust without
let-chains (`error[E0658]: let expressions in this position are unstable`) or a corepack that makes
no `pnpm` shims (`pnpm: No such file or directory`). Check the toolchain version in the image, not
only that the image pulls.

### What the guest does and does not pull

- The guest has outbound internet. Seed-free pulls work: `registry.gitlab.com/gitlab-org/gitlab-runner-helper`
  and `rancher/mirrored-library-busybox` both arrive over the network.
- `image_pull_policy = "if-not-present"` is right: the node image seeds some images into the guest's
  containerd, and always-pull throws that away.
- A big image cannot be seeded into the node image. `mkfs.ext4 -d` refuses a file of about 2 GiB or
  more, and a multi-gigabyte builder image is larger than that as a tarball. Pull it instead, and
  size the root filesystem for it. See
  [01](01-kcp-server.md#the-root-filesystem-fills-before-the-disk).

## The smallest pipeline that proves the integration

One stage, one job, and it tells you whether every hop works:

```yaml
stages: [prove]

prove-placement:
  stage: prove
  image: busybox:1.36
  script:
    - echo "running on $(hostname)"
    - echo "namespace $(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)"
```

It exercises, in order: the runner authenticates to the apiserver; a Pod is created in the
`gitlab-runner` namespace; the cluster observes intent and provisions a node; the Pod is placed;
the build container runs; the helper container streams the log back through the runner.

If this passes, the integration works. If it hangs at "Preparing the environment", the runner is
pointed at the kcp mount URL instead of loopback. See
[README.md](README.md#reachability-the-single-operational-fact).

To watch the placement from outside, query the tenant cluster as its admin:

```bash
kubectl --kubeconfig <tenant-admin> get pods -n gitlab-runner -o wide -w
kubectl --kubeconfig <tenant-admin> get nodes -w
```

## Jobs that need a service container

GitLab's `services:` become extra containers in the same Pod, reachable over the Pod's own localhost
and by DNS alias. They share the Pod's resource quota.

```yaml
test-with-postgres:
  stage: prove
  image: postgres:16-alpine
  services:
    - name: postgres:16-alpine
      alias: db
  variables:
    POSTGRES_PASSWORD: ci
    PGHOST: db
    PGUSER: postgres
  script:
    - psql -c 'select 1'
```

Two constraints come from the cluster, not from GitLab:

- Ports are per-Pod. Services cannot reuse a port with each other, and a service binding a privileged
  port needs the Pod's security context relaxed. Prefer non-privileged ports.
- The node image is small. Pull policies and image sizes matter more here than on a conventional
  cluster.

## A job that deploys to the same cluster

The Pod already has a service account token mounted. Use it rather than handing the job a long-lived
credential:

```yaml
deploy:
  stage: prove
  image: bitnami/kubectl:1.32
  script:
    - kubectl config set-cluster tenant --server=https://kubernetes.default.svc --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    - kubectl config set-credentials ci --token="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
    - kubectl config set-context tenant --cluster=tenant --user=ci
    - kubectl --context tenant -n gitlab-runner get pods
```

This works only if the runner's `service_account` may read what it reads. The ClusterRole in
[01](01-kcp-server.md#5-permit-that-group-in-the-tenant-cluster) is scoped to what the runner needs
to place jobs, which is not what a job needs to deploy. Grant the job's service account separately
and keep the two apart.

## The job Pod cannot reach your GitLab, by default

A job Pod runs inside a guest, on the tenant cluster's own network, not on the kcp host. The
repository URL the runner hands the helper container, `CI_REPOSITORY_URL`, is whatever host GitLab
is published on, and `http://127.0.0.1:8080/...` there is the guest's loopback. The clone fails:

```
fatal: unable to access 'https://.../repo.git/': ...
ERROR: Job failed: command terminated with exit code 128
```

GitLab reports it as `script_failure`, which reads like a broken runner.

Three ways out, in the order that costs least:

1. **Do not clone.** `variables: { GIT_STRATEGY: none }` skips the fetch entirely. Right for a job
   that only needs to run commands.
2. **Give the guest a route to GitLab.** Bind GitLab to an address the guest can dial, and set
   `external_url` accordingly, so `CI_REPOSITORY_URL` carries an address that resolves from inside
   the guest.
3. **Run the runner where the Pods are**, so the URL it hands out is one the Pods can reach.

Option 1 proves the integration. Options 2 and 3 are what a real pipeline needs, and which one is
right depends on whether GitLab is reachable from the tenant network at all.

## Artifacts and cache

- **Artifacts** travel back to GitLab over the runner's HTTPS connection to the GitLab instance, not
  through any cluster mechanism. They work unchanged.
- **Cache** does not. There is no object store in this design and no PersistentVolume: the cache
  volume GitLab would use is per-Pod. Configure an external S3-compatible endpoint in
  `[runners.cache]`, because a `hostPath` or PVC cache has no node to live on once the cluster
  returns to zero.

## Resource requests are the contract

`cpu_request` and `memory_request` are the runner's declaration of what one job needs. They size the
Pod today, and they are what `podsPerNode` divides by when the cluster scales out, so set them from a
real measurement rather than a guess.

A job that overrides its requests with `KUBERNETES_CPU_REQUEST` and friends asks for a placement
decision the runner made one layer up to be re-made, and it will disagree with the node that was
provisioned for it.

## One job per node

The runner creates one Pod per job. The cluster's own scale decision provisions the node that Pod
lands on, so N concurrent jobs is N Pods, and N Pods is what raises the node target.

Two knobs, on two sides.

Runner side: anti-affinity, in `config.toml` (see [02](02-gitlab-runner.md#the-file)):

```toml
    [runners.kubernetes.affinity]
      [runners.kubernetes.affinity.pod_anti_affinity]
        [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution]]
          topology_key = "kubernetes.io/hostname"
          [runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector]
            [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector.match_expressions]]
              key = "project.runner.gitlab.com/id"
              operator = "Exists"
```

Use `[runners.kubernetes.affinity]`. It is first-class config; `pod_spec` needs the beta
`FF_USE_ADVANCED_POD_SPEC_CONFIGURATION` flag and this does not. Use `required`, not `preferred`, or
a Pod co-schedules whenever the scheduler chooses to.

Cluster side, on the cluster provider:

```bash
SOCIALWEB_CLUSTERPROVIDER_SCALE_MAX_NODES=5
SOCIALWEB_CLUSTERPROVIDER_SCALE_PODS_PER_NODE=1
SOCIALWEB_CLUSTERPROVIDER_SCALE_HOLD=1m
```

| Variable | Meaning | Default |
| --- | --- | --- |
| `SOCIALWEB_CLUSTERPROVIDER_SCALE_MAX_NODES` | Ceiling. Below one is refused rather than clamped. Every node is a booted guest, so this is a cost knob as much as a behaviour one. | 10 |
| `SOCIALWEB_CLUSTERPROVIDER_SCALE_PODS_PER_NODE` | How much work one node is assumed to hold. Set it from a measured job. | 8 |
| `SOCIALWEB_CLUSTERPROVIDER_SCALE_HOLD` | Idle grace before scale-in. | 2m |

The decision counts Pods that are Pending, Running or Unknown, skipping k3s's own addons. At one Pod
per node, five live job Pods is a target of five. When the work stops the target is zero, and the
cluster goes to zero, not to an intermediate count.

### Prove it

Five jobs, one per node, then back to zero. The jobs echo and sleep. The image is one the node image
already carries, so a slow pull cannot hide the placement.

```yaml
.scale-check:
  stage: build
  interruptible: true
  image: rancher/mirrored-library-busybox:1.37.0@sha256:ad9fa4d07136a83e69a54ef00102f579d04eba431932de3b0f098cc5d5948f9f
  timeout: 30 minutes
  script:
    - echo "SCALE_CHECK job=${CI_JOB_NAME} pod=$(hostname)"
    - sleep 180

scale-check-1:
  extends: .scale-check
scale-check-2:
  extends: .scale-check
scale-check-3:
  extends: .scale-check
scale-check-4:
  extends: .scale-check
scale-check-5:
  extends: .scale-check
```

A working run reads:

```
5 job pods exist, all Pending, cluster at zero nodes
the scale decision reads Provision with nodes=5: intent and no node
5 nodes join on demand, all Ready
each job pod is on its own node
all 5 jobs finish success
the job pods are gone
the scale decision reads Teardown with nodes=0:
  nothing running for 1m0.9s, past the 1m0s hold
the cluster is back at zero nodes, control plane still Ready
```

Mid-flight, one node at a time: with one node up, several Pods sit `Init` on it and the rest stay
`Pending`, because the anti-affinity refuses to co-schedule, and those Pending Pods are what take the
count up. That is the loop working, not scheduling luck.

### Pods on different nodes, and DNS on every node

On every market cluster, DNS never leaves the node. Each node carries `169.254.20.10` on a dummy
interface, runs its own CoreDNS there, and its kubelet hands Pods `nameserver 169.254.20.10`. A job
Pod resolves cluster names and `$CI_SERVER_HOST` on the node it runs on, whichever node that is.

- The CoreDNS is a host-network DaemonSet, `kube-system/socialweb-node-dns`, selected by the node
  label `socialweb.computer/node-local-dns=true` and bound to `169.254.20.10`. The guest's
  user_data puts the address on the dummy interface and starts the k3s agent with
  `--kubelet-arg=cluster-dns=169.254.20.10`; the cluster provider applies the DaemonSet in every
  tenant.
- Pods reach Pods on other nodes through `socialweb-meshd`. k3s runs with
  `--flannel-backend=none` and `--cluster-cidr=10.42.0.0/16`, each node has a bridge CNI on its own
  pod CIDR, and `socialweb-meshd` routes every other node's pod CIDR over iroh QUIC datagrams to
  that node. Traffic leaving the cluster is masqueraded on the node it leaves from; traffic between
  Pods keeps its pod source address.

## Pause pods, and why they fight scale-to-zero

GitLab Runner 18.10 and later can keep low-priority pause pods on the cluster so a job Pod preempts
one and starts immediately instead of waiting for a node.

On a scale-to-zero cluster a pause pod interacts with the scale decision. A pause pod is a Pending
pod that is not one of k3s's own addons, so the scale decision counts it as intent and will
provision a node to place it, and the cluster will never return to zero while a pause pod exists.

The way out is the pause-pod policy's own cron. GitLab's `idle_count` is per-period and defaults to
0, so a policy asking for `idle_count = 0` outside working hours scales the pause-pod Deployment to
zero; with no pause pods there is no intent and the node is reaped as usual. Do not set a 24x7
`idle_count > 0` on a cluster you expect to reach zero.

The RBAC for pause pods is in
[01](01-kcp-server.md#5-permit-that-group-in-the-tenant-cluster); `priorityclasses` is
cluster-scoped, so it needs a ClusterRole.