0
0
# 02 - The GitLab runner
End state: a runner registered to a project, running on the kcp host, with `host` set to the tenant
apiserver's loopback URL.
This needs [01](01-kcp-server.md) finished: a tenant cluster, its apiserver URL, the credential files
`/etc/gitlab-runner/tenant-ci/`, and the group name `gitlab-runner`.
## GitLab
Already have an instance? Skip to registering a runner. Otherwise, on the kcp host:
```yaml
services:
gitlab:
image: gitlab/gitlab-ce:latest
container_name: gitlab
restart: always
hostname: 'gitlab'
shm_size: '256m'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://gitlab:8080'
gitlab_rails['gitlab_shell_ssh_port'] = 2222
nginx['listen_port'] = 80
nginx['listen_https'] = false
ports:
- '8080:80'
- '2222:22'
volumes:
- /srv/gitlab/config:/etc/gitlab
- /srv/gitlab/logs:/var/log/gitlab
- /srv/gitlab/data:/var/opt/gitlab
```
Two settings there are deliberate:
- **`hostname` is short, not an FQDN.** A long container hostname breaks Docker's internal DNS, and
the failure presents as an unreachable OIDC provider rather than as a DNS problem.
- **`GITLAB_OMNIBUS_CONFIG` only applies on first boot.** Editing it later does nothing. Change
`/srv/gitlab/config/gitlab.rb` and run `docker exec gitlab gitlab-ctl reconfigure` instead. This is
the most common way a working GitLab is broken by a correct-looking edit.
Wait for readiness rather than sleeping:
```bash
until curl -fsS http://127.0.0.1:8080/-/health >/dev/null; do sleep 5; done
```
Install the runner on the same host:
```bash
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
sudo dnf install -y gitlab-runner
```
The runner does not run in the cluster. It runs here, because the apiserver binds loopback here.
## Register
Create the runner in the project's **Settings > CI/CD > Runners** and copy the authentication token.
The legacy registration-token flow is deprecated.
```bash
gitlab-runner register \
--non-interactive \
--config /etc/gitlab-runner/config.toml \
--url https://gitlab.example.com/ \
--token <runner authentication token> \
--executor kubernetes
```
`register` writes a config with the token and nothing else useful. Replace the rest with the file
below.
## The file
Fill the `<<...>>` placeholders. The table after says why every other value is set.
```toml
concurrent = 8
check_interval = 3
[[runners]]
name = "socialweb-tenant-ci"
url = "https://gitlab.example.com/"
token = "<<runner authentication token>>"
executor = "kubernetes"
# Runs AFTER the clone, before the job's script. A STRING, not an array.
pre_build_script = "true"
# Kilobytes. Default is 4096, which is 4 MiB and is NOT enough for a Go test log.
output_limit = 51200
[runners.kubernetes]
host = "<<https://127.0.0.1:LEASED_PORT>>"
cert_file = "/etc/gitlab-runner/tenant-ci/tls.crt"
key_file = "/etc/gitlab-runner/tenant-ci/tls.key"
ca_file = "/etc/gitlab-runner/tenant-ci/ca.crt"
namespace = "gitlab-runner"
image_pull_policy = "if-not-present"
# Requests are what the scheduler places against. Set them EXPLICITLY.
cpu_request = "100m"
memory_request = "128Mi"
cpu_limit = "10"
memory_limit = "20Gi"
# Helper and init containers too, or their limits become their requests.
helper_cpu_request = "100m"
helper_memory_request = "128Mi"
helper_cpu_limit = "1"
helper_memory_limit = "1Gi"
poll_timeout = 1800
cleanup_grace_period_seconds = 30
[runners.kubernetes.affinity]
[runners.kubernetes.affinity.pod_anti_affinity]
[[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution]]
topology_key = "kubernetes.io/hostname"
[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector]
[[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector.match_expressions]]
key = "project.runner.gitlab.com/id"
operator = "Exists"
```
| Key | Why |
| --- | --- |
| `concurrent` | Jobs in flight at once. One per node with anti-affinity on. `1` wedges: a stale job from a cancelled pipeline holds the only slot. |
| `host` | The apiserver's own loopback URL, from [01](01-kcp-server.md#hand-back). NOT the kcp mount URL, which is a reverse proxy and does not carry `pods/exec`. |
| `cert_file`/`key_file`/`ca_file` | The tenant credential. A client cert, NOT a bearer token. |
| `namespace` | Where job Pods go. Must exist. |
| `image_pull_policy` | `if-not-present`. Node images seed some images; always-pull throws that away. |
| `cpu_request`/`memory_request` | The scheduler uses these. Too big and the Pod never schedules. |
| `cpu_limit`/`memory_limit` | Enforcement only. The scheduler ignores them, so a limit over node size is fine. |
| `helper_*_request` | Set these. See the traps. |
| `poll_timeout` | Seconds to wait for the Pod to reach Running. Node boot plus a big image pull. |
| `output_limit` | Kilobytes of trace. Default 4096 = 4 MiB. |
| anti-affinity | One job per node, see [03](03-jobs.md#one-job-per-node). |
## Traps
**`pre_build_script` is a string, not an array.** An array is a startup failure, and the runner says
so only in its own output:
```
FATAL: Service run failed error=decoding configuration file: toml: line 9
(last key "runners.pre_build_script"): incompatible types: TOML value has type []any;
destination has type string
```
The runner exits, nothing picks work up, and GitLab still reports the runner online, so a watcher
waits for a job that never runs. Fail on `FATAL` in the runner's own output, not on GitLab's view.
**`pre_build_script` runs AFTER the clone. `pre_get_sources_script` runs BEFORE it.** Anything that
must guard the clone goes in `pre_get_sources_script`; anything that must run against the checked-out
tree goes in `pre_build_script`.
**A limit with no request IS the request.** Kubernetes defaults a container's request to its limit.
`helper_cpu_limit = 1` alone makes every job Pod ask about 2.1 CPU: the build container's request plus
the helper's and the init container's, each pulled up from its limit. A two-vCPU node answers:
```
Unschedulable: "0/1 nodes are available: 1 Insufficient cpu."
```
That reads as a slow cluster. It is a Pod that cannot fit.
**The 4 MiB log cap is the runner's, not GitLab's.**
```
Job's log exceeded limit of 4194304 bytes. Job execution will continue but no more output will be collected.
```
That string is in the `gitlab-runner` binary. GitLab's own ceiling is `ci_jobs_trace_size_limit`, a
plan limit (`project.actual_limits`) that is not in `/application/settings`. Check both before
assuming which one bit you.
**Tools are per-image.** A `pre_build_script` that assumes `corepack`, `pnpm` or `npm` fails the job
with `exit code 127` on an image that has none. Guard on the tool existing, and end with `true` so the
guard never fails the job it protects:
```toml
pre_build_script = "if ! command -v pnpm >/dev/null 2>&1 && command -v npm >/dev/null 2>&1; then npm install -g pnpm@10.11.0; fi; true"
```
## Start it
```bash
gitlab-runner run --config /etc/gitlab-runner/config.toml --working-directory /var/lib/gitlab-runner
```
Or as a service under a user that can read `/etc/gitlab-runner/tenant-ci/`.
## Check the credential before blaming the runner
```bash
openssl x509 -in /etc/gitlab-runner/tenant-ci/tls.crt -noout -subject
kubectl --kubeconfig <tenant-admin> auth can-i create pods \
--namespace gitlab-runner --as=gitlab-runner --as-group=gitlab-runner
```
## Failure modes, in the order you will meet them
| Symptom | Cause |
| --- | --- |
| Runner exits at startup | `config.toml` did not decode. Its own output says which key. |
| Job stays `pending`, no node appears | The runner reached a DIFFERENT apiserver, or the Pod went to a namespace the observer does not watch. Compare `kubectl get pods -A` with the cluster's own port. |
| `x509: certificate signed by unknown authority` | Wrong `ca_file`, or a `host` port belonging to another cluster. Ports are leased per cluster and are not shared. |
| `Unauthorized` at job start | The certificate is from another cluster's ClientCA, or the Organizations do not match the RBAC group. |
| `Forbidden` creating pods | The ClusterRoleBinding's subject does not match the certificate's Organizations. |
| Job hangs at "Preparing the environment" | `pods/exec` unreachable. This is what happens when the runner points at the kcp mount URL instead of loopback. |
| Job passes, then a retry fails | The credential expired. Check its `VALID_FOR`. |
| `Insufficient cpu`, no node appears | Pod requests do not fit a node. See the traps. |
Next: [03 - Jobs](03-jobs.md).