02 - The GitLab runner

End state: a runner registered to a project, running on the kcp host, with host set to the tenant apiserver's loopback URL.

This needs 01 finished: a tenant cluster, its apiserver URL, the credential files /etc/gitlab-runner/tenant-ci/, and the group name gitlab-runner.

GitLab

Already have an instance? Skip to registering a runner. Otherwise, on the kcp host:

services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    container_name: gitlab
    restart: always
    hostname: 'gitlab'
    shm_size: '256m'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab:8080'
        gitlab_rails['gitlab_shell_ssh_port'] = 2222
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
    ports:
      - '8080:80'
      - '2222:22'
    volumes:
      - /srv/gitlab/config:/etc/gitlab
      - /srv/gitlab/logs:/var/log/gitlab
      - /srv/gitlab/data:/var/opt/gitlab

Two settings there are deliberate:

hostname is short, not an FQDN. A long container hostname breaks Docker's internal DNS, and the failure presents as an unreachable OIDC provider rather than as a DNS problem.
GITLAB_OMNIBUS_CONFIG only applies on first boot. Editing it later does nothing. Change /srv/gitlab/config/gitlab.rb and run docker exec gitlab gitlab-ctl reconfigure instead. This is the most common way a working GitLab is broken by a correct-looking edit.

Wait for readiness rather than sleeping:

until curl -fsS http://127.0.0.1:8080/-/health >/dev/null; do sleep 5; done

Install the runner on the same host:

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
sudo dnf install -y gitlab-runner

The runner does not run in the cluster. It runs here, because the apiserver binds loopback here.

Register

Create the runner in the project's Settings > CI/CD > Runners and copy the authentication token. The legacy registration-token flow is deprecated.

gitlab-runner register \
  --non-interactive \
  --config /etc/gitlab-runner/config.toml \
  --url https://gitlab.example.com/ \
  --token <runner authentication token> \
  --executor kubernetes

register writes a config with the token and nothing else useful. Replace the rest with the file below.

The file

Fill the <<...>> placeholders. The table after says why every other value is set.

concurrent = 8
check_interval = 3

[[runners]]
  name = "socialweb-tenant-ci"
  url = "https://gitlab.example.com/"
  token = "<<runner authentication token>>"
  executor = "kubernetes"

  # Runs AFTER the clone, before the job's script. A STRING, not an array.
  pre_build_script = "true"

  # Kilobytes. Default is 4096, which is 4 MiB and is NOT enough for a Go test log.
  output_limit = 51200

  [runners.kubernetes]
    host = "<<https://127.0.0.1:LEASED_PORT>>"
    cert_file = "/etc/gitlab-runner/tenant-ci/tls.crt"
    key_file = "/etc/gitlab-runner/tenant-ci/tls.key"
    ca_file = "/etc/gitlab-runner/tenant-ci/ca.crt"
    namespace = "gitlab-runner"
    image_pull_policy = "if-not-present"

    # Requests are what the scheduler places against. Set them EXPLICITLY.
    cpu_request = "100m"
    memory_request = "128Mi"
    cpu_limit = "10"
    memory_limit = "20Gi"

    # Helper and init containers too, or their limits become their requests.
    helper_cpu_request = "100m"
    helper_memory_request = "128Mi"
    helper_cpu_limit = "1"
    helper_memory_limit = "1Gi"

    poll_timeout = 1800
    cleanup_grace_period_seconds = 30

    [runners.kubernetes.affinity]
      [runners.kubernetes.affinity.pod_anti_affinity]
        [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution]]
          topology_key = "kubernetes.io/hostname"
          [runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector]
            [[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector.match_expressions]]
              key = "project.runner.gitlab.com/id"
              operator = "Exists"

Key	Why
`concurrent`	Jobs in flight at once. One per node with anti-affinity on. `1` wedges: a stale job from a cancelled pipeline holds the only slot.
`host`	The apiserver's own loopback URL, from 01. NOT the kcp mount URL, which is a reverse proxy and does not carry `pods/exec`.
`cert_file`/`key_file`/`ca_file`	The tenant credential. A client cert, NOT a bearer token.
`namespace`	Where job Pods go. Must exist.
`image_pull_policy`	`if-not-present`. Node images seed some images; always-pull throws that away.
`cpu_request`/`memory_request`	The scheduler uses these. Too big and the Pod never schedules.
`cpu_limit`/`memory_limit`	Enforcement only. The scheduler ignores them, so a limit over node size is fine.
`helper_*_request`	Set these. See the traps.
`poll_timeout`	Seconds to wait for the Pod to reach Running. Node boot plus a big image pull.
`output_limit`	Kilobytes of trace. Default 4096 = 4 MiB.
anti-affinity	One job per node, see 03.

Traps

pre_build_script is a string, not an array. An array is a startup failure, and the runner says so only in its own output:

FATAL: Service run failed  error=decoding configuration file: toml: line 9
(last key "runners.pre_build_script"): incompatible types: TOML value has type []any;
destination has type string

The runner exits, nothing picks work up, and GitLab still reports the runner online, so a watcher waits for a job that never runs. Fail on FATAL in the runner's own output, not on GitLab's view.

pre_build_script runs AFTER the clone. pre_get_sources_script runs BEFORE it. Anything that must guard the clone goes in pre_get_sources_script; anything that must run against the checked-out tree goes in pre_build_script.

A limit with no request IS the request. Kubernetes defaults a container's request to its limit. helper_cpu_limit = 1 alone makes every job Pod ask about 2.1 CPU: the build container's request plus the helper's and the init container's, each pulled up from its limit. A two-vCPU node answers:

Unschedulable: "0/1 nodes are available: 1 Insufficient cpu."

That reads as a slow cluster. It is a Pod that cannot fit.

The 4 MiB log cap is the runner's, not GitLab's.

Job's log exceeded limit of 4194304 bytes. Job execution will continue but no more output will be collected.

That string is in the gitlab-runner binary. GitLab's own ceiling is ci_jobs_trace_size_limit, a plan limit (project.actual_limits) that is not in /application/settings. Check both before assuming which one bit you.

Tools are per-image. A pre_build_script that assumes corepack, pnpm or npm fails the job with exit code 127 on an image that has none. Guard on the tool existing, and end with true so the guard never fails the job it protects:

pre_build_script = "if ! command -v pnpm >/dev/null 2>&1 && command -v npm >/dev/null 2>&1; then npm install -g pnpm@10.11.0; fi; true"

Start it

gitlab-runner run --config /etc/gitlab-runner/config.toml --working-directory /var/lib/gitlab-runner

Or as a service under a user that can read /etc/gitlab-runner/tenant-ci/.

Check the credential before blaming the runner

openssl x509 -in /etc/gitlab-runner/tenant-ci/tls.crt -noout -subject
kubectl --kubeconfig <tenant-admin> auth can-i create pods \
  --namespace gitlab-runner --as=gitlab-runner --as-group=gitlab-runner

Failure modes, in the order you will meet them

Symptom	Cause
Runner exits at startup	`config.toml` did not decode. Its own output says which key.
Job stays `pending`, no node appears	The runner reached a DIFFERENT apiserver, or the Pod went to a namespace the observer does not watch. Compare `kubectl get pods -A` with the cluster's own port.
`x509: certificate signed by unknown authority`	Wrong `ca_file`, or a `host` port belonging to another cluster. Ports are leased per cluster and are not shared.
`Unauthorized` at job start	The certificate is from another cluster's ClientCA, or the Organizations do not match the RBAC group.
`Forbidden` creating pods	The ClusterRoleBinding's subject does not match the certificate's Organizations.
Job hangs at "Preparing the environment"	`pods/exec` unreachable. This is what happens when the runner points at the kcp mount URL instead of loopback.
Job passes, then a retry fails	The credential expired. Check its `VALID_FOR`.
`Insufficient cpu`, no node appears	Pod requests do not fit a node. See the traps.

Next: 03 - Jobs.

johnandersen777.bsky.social/socialweb-computer-kcp Switch repository Loading…