03 - Jobs

Audience: an operator running GitLab CI jobs on a tenant k3s cluster.

Outcome: a GitLab pipeline runs its jobs as Pods in the tenant cluster, one job per node, and the cluster returns to zero nodes when the work stops.

Prerequisite: 02 done, so a runner points at the tenant cluster.

The image a job runs in

A job Pod's image comes from the project's own .gitlab-ci.yml:

  image: "$DOCKER_REPO:builder-$DOCKERFILE_HASH"

DOCKER_REPO is a variable in that file. DOCKERFILE_HASH comes from the included .ci/dockerfile-hash.yaml, which is generated:

printf "variables:\n  DOCKERFILE_HASH: `git hash-object docker/build.Dockerfile`" > .ci/dockerfile-hash.yaml

The image is named by a hash of the Dockerfile. Change the Dockerfile, get a different image name, and the old name stays valid.

Check the image exists before running anything

Do not find a missing image as ImagePullBackOff twenty minutes into a node boot. The manifest endpoint needs a token even for a public repository:

REPO=streamplace/streamplace
TAG=builder-3e14c389b80a359787d73741f221fdfaacb12f53

TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:${REPO}:pull&service=ghcr.io" \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')

curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer ${TOKEN}" \
  -H 'Accept: application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.v2+json' \
  "https://ghcr.io/v2/${REPO}/manifests/${TAG}"

200 means the job will pull. 404 MANIFEST_UNKNOWN means the job cannot start, whatever else is right about the cluster.

If the branch names an image nobody published

A branch can point DOCKER_REPO at a tag no job ever published, for example a public.ecr.aws repository that answers NAME_UNKNOWN for the repository itself. Every job in that file is then unrunnable as written. Read the upstream branch's variables: block before concluding an image is missing: a stale branch and a missing image look identical from the failure.

Two ways to make the branch runnable, in order of cost:

Move the branch. Rebase onto the branch that publishes and use its own .ci/dockerfile-hash.yaml. Its DOCKER_REPO and hash already agree with each other.
Build it and host it. docker build --target builder -f docker/build.Dockerfile . at the hash the branch names, push to a registry the guest can reach, point DOCKER_REPO at it. The guest has outbound internet, so a public registry works and a private address does not without a route and a registries.yaml.

An image that resolves is not the same as a toolchain that builds. A builder image old enough to predate a language feature the project uses fails inside the job, for example a Rust without let-chains (error[E0658]: let expressions in this position are unstable) or a corepack that makes no pnpm shims (pnpm: No such file or directory). Check the toolchain version in the image, not only that the image pulls.

What the guest does and does not pull

The guest has outbound internet. Seed-free pulls work: registry.gitlab.com/gitlab-org/gitlab-runner-helper and rancher/mirrored-library-busybox both arrive over the network.
image_pull_policy = "if-not-present" is right: the node image seeds some images into the guest's containerd, and always-pull throws that away.
A big image cannot be seeded into the node image. mkfs.ext4 -d refuses a file of about 2 GiB or more, and a multi-gigabyte builder image is larger than that as a tarball. Pull it instead, and size the root filesystem for it. See 01.

The smallest pipeline that proves the integration

One stage, one job, and it tells you whether every hop works:

stages: [prove]

prove-placement:
  stage: prove
  image: busybox:1.36
  script:
    - echo "running on $(hostname)"
    - echo "namespace $(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)"

It exercises, in order: the runner authenticates to the apiserver; a Pod is created in the gitlab-runner namespace; the cluster observes intent and provisions a node; the Pod is placed; the build container runs; the helper container streams the log back through the runner.

If this passes, the integration works. If it hangs at "Preparing the environment", the runner is pointed at the kcp mount URL instead of loopback. See README.md.

To watch the placement from outside, query the tenant cluster as its admin:

kubectl --kubeconfig <tenant-admin> get pods -n gitlab-runner -o wide -w
kubectl --kubeconfig <tenant-admin> get nodes -w

Jobs that need a service container

GitLab's services: become extra containers in the same Pod, reachable over the Pod's own localhost and by DNS alias. They share the Pod's resource quota.

test-with-postgres:
  stage: prove
  image: postgres:16-alpine
  services:
    - name: postgres:16-alpine
      alias: db
  variables:
    POSTGRES_PASSWORD: ci
    PGHOST: db
    PGUSER: postgres
  script:
    - psql -c 'select 1'

Two constraints come from the cluster, not from GitLab:

Ports are per-Pod. Services cannot reuse a port with each other, and a service binding a privileged port needs the Pod's security context relaxed. Prefer non-privileged ports.
The node image is small. Pull policies and image sizes matter more here than on a conventional cluster.

A job that deploys to the same cluster

The Pod already has a service account token mounted. Use it rather than handing the job a long-lived credential:

deploy:
  stage: prove
  image: bitnami/kubectl:1.32
  script:
    - kubectl config set-cluster tenant --server=https://kubernetes.default.svc --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    - kubectl config set-credentials ci --token="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
    - kubectl config set-context tenant --cluster=tenant --user=ci
    - kubectl --context tenant -n gitlab-runner get pods

This works only if the runner's service_account may read what it reads. The ClusterRole in 01 is scoped to what the runner needs to place jobs, which is not what a job needs to deploy. Grant the job's service account separately and keep the two apart.

The job Pod cannot reach your GitLab, by default

A job Pod runs inside a guest, on the tenant cluster's own network, not on the kcp host. The repository URL the runner hands the helper container, CI_REPOSITORY_URL, is whatever host GitLab is published on, and http://127.0.0.1:8080/... there is the guest's loopback. The clone fails:

fatal: unable to access 'https://.../repo.git/': ...
ERROR: Job failed: command terminated with exit code 128

GitLab reports it as script_failure, which reads like a broken runner.

Three ways out, in the order that costs least:

Do not clone. variables: { GIT_STRATEGY: none } skips the fetch entirely. Right for a job that only needs to run commands.
Give the guest a route to GitLab. Bind GitLab to an address the guest can dial, and set external_url accordingly, so CI_REPOSITORY_URL carries an address that resolves from inside the guest.
Run the runner where the Pods are, so the URL it hands out is one the Pods can reach.

Option 1 proves the integration. Options 2 and 3 are what a real pipeline needs, and which one is right depends on whether GitLab is reachable from the tenant network at all.

Artifacts and cache

Artifacts travel back to GitLab over the runner's HTTPS connection to the GitLab instance, not through any cluster mechanism. They work unchanged.
Cache does not. There is no object store in this design and no PersistentVolume: the cache volume GitLab would use is per-Pod. Configure an external S3-compatible endpoint in [runners.cache], because a hostPath or PVC cache has no node to live on once the cluster returns to zero.

Resource requests are the contract

cpu_request and memory_request are the runner's declaration of what one job needs. They size the Pod today, and they are what podsPerNode divides by when the cluster scales out, so set them from a real measurement rather than a guess.

A job that overrides its requests with KUBERNETES_CPU_REQUEST and friends asks for a placement decision the runner made one layer up to be re-made, and it will disagree with the node that was provisioned for it.

One job per node

The runner creates one Pod per job. The cluster's own scale decision provisions the node that Pod lands on, so N concurrent jobs is N Pods, and N Pods is what raises the node target.

Two knobs, on two sides.

Runner side: anti-affinity, in config.toml (see 02):

    [runners.kubernetes.affinity]
      [runners.kubernetes.affinity.pod_anti_affinity]
        [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution]]
          topology_key = "kubernetes.io/hostname"
          [runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector]
            [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector.match_expressions]]
              key = "project.runner.gitlab.com/id"
              operator = "Exists"

Use [runners.kubernetes.affinity]. It is first-class config; pod_spec needs the beta FF_USE_ADVANCED_POD_SPEC_CONFIGURATION flag and this does not. Use required, not preferred, or a Pod co-schedules whenever the scheduler chooses to.

Cluster side, on the cluster provider:

SOCIALWEB_CLUSTERPROVIDER_SCALE_MAX_NODES=5
SOCIALWEB_CLUSTERPROVIDER_SCALE_PODS_PER_NODE=1
SOCIALWEB_CLUSTERPROVIDER_SCALE_HOLD=1m

Variable	Meaning	Default
`SOCIALWEB_CLUSTERPROVIDER_SCALE_MAX_NODES`	Ceiling. Below one is refused rather than clamped. Every node is a booted guest, so this is a cost knob as much as a behaviour one.	10
`SOCIALWEB_CLUSTERPROVIDER_SCALE_PODS_PER_NODE`	How much work one node is assumed to hold. Set it from a measured job.	8
`SOCIALWEB_CLUSTERPROVIDER_SCALE_HOLD`	Idle grace before scale-in.	2m

The decision counts Pods that are Pending, Running or Unknown, skipping k3s's own addons. At one Pod per node, five live job Pods is a target of five. When the work stops the target is zero, and the cluster goes to zero, not to an intermediate count.

Prove it

Five jobs, one per node, then back to zero. The jobs echo and sleep. The image is one the node image already carries, so a slow pull cannot hide the placement.

.scale-check:
  stage: build
  interruptible: true
  image: rancher/mirrored-library-busybox:1.37.0@sha256:ad9fa4d07136a83e69a54ef00102f579d04eba431932de3b0f098cc5d5948f9f
  timeout: 30 minutes
  script:
    - echo "SCALE_CHECK job=${CI_JOB_NAME} pod=$(hostname)"
    - sleep 180

scale-check-1:
  extends: .scale-check
scale-check-2:
  extends: .scale-check
scale-check-3:
  extends: .scale-check
scale-check-4:
  extends: .scale-check
scale-check-5:
  extends: .scale-check

A working run reads:

5 job pods exist, all Pending, cluster at zero nodes
the scale decision reads Provision with nodes=5: intent and no node
5 nodes join on demand, all Ready
each job pod is on its own node
all 5 jobs finish success
the job pods are gone
the scale decision reads Teardown with nodes=0:
  nothing running for 1m0.9s, past the 1m0s hold
the cluster is back at zero nodes, control plane still Ready

Mid-flight, one node at a time: with one node up, several Pods sit Init on it and the rest stay Pending, because the anti-affinity refuses to co-schedule, and those Pending Pods are what take the count up. That is the loop working, not scheduling luck.

Pods on different nodes, and DNS on every node

On every market cluster, DNS never leaves the node. Each node carries 169.254.20.10 on a dummy interface, runs its own CoreDNS there, and its kubelet hands Pods nameserver 169.254.20.10. A job Pod resolves cluster names and $CI_SERVER_HOST on the node it runs on, whichever node that is.

The CoreDNS is a host-network DaemonSet, kube-system/socialweb-node-dns, selected by the node label socialweb.computer/node-local-dns=true and bound to 169.254.20.10. The guest's user_data puts the address on the dummy interface and starts the k3s agent with --kubelet-arg=cluster-dns=169.254.20.10; the cluster provider applies the DaemonSet in every tenant.
Pods reach Pods on other nodes through socialweb-meshd. k3s runs with --flannel-backend=none and --cluster-cidr=10.42.0.0/16, each node has a bridge CNI on its own pod CIDR, and socialweb-meshd routes every other node's pod CIDR over iroh QUIC datagrams to that node. Traffic leaving the cluster is masqueraded on the node it leaves from; traffic between Pods keeps its pod source address.

Pause pods, and why they fight scale-to-zero

GitLab Runner 18.10 and later can keep low-priority pause pods on the cluster so a job Pod preempts one and starts immediately instead of waiting for a node.

On a scale-to-zero cluster a pause pod interacts with the scale decision. A pause pod is a Pending pod that is not one of k3s's own addons, so the scale decision counts it as intent and will provision a node to place it, and the cluster will never return to zero while a pause pod exists.

The way out is the pause-pod policy's own cron. GitLab's idle_count is per-period and defaults to 0, so a policy asking for idle_count = 0 outside working hours scales the pause-pod Deployment to zero; with no pause pods there is no intent and the node is reaped as usual. Do not set a 24x7 idle_count > 0 on a cluster you expect to reach zero.

The RBAC for pause pods is in 01; priorityclasses is cluster-scoped, so it needs a ClusterRole.

johnandersen777.bsky.social/socialweb-computer-kcp Switch repository Loading…