1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
# 02 - The GitLab runner
End state: a runner registered to a project, running on the kcp host, with `host` set to the tenant
apiserver's loopback URL.
This needs [01](01-kcp-server.md) finished: a tenant cluster, its apiserver URL, the credential files
`/etc/gitlab-runner/tenant-ci/`, and the group name `gitlab-runner`.
## GitLab
Already have an instance? Skip to registering a runner. Otherwise, on the kcp host:
```yaml
services:
gitlab:
image: gitlab/gitlab-ce:latest
container_name: gitlab
restart: always
hostname: 'gitlab'
shm_size: '256m'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://gitlab:8080'
gitlab_rails['gitlab_shell_ssh_port'] = 2222
nginx['listen_port'] = 80
nginx['listen_https'] = false
ports:
- '8080:80'
- '2222:22'
volumes:
- /srv/gitlab/config:/etc/gitlab
- /srv/gitlab/logs:/var/log/gitlab
- /srv/gitlab/data:/var/opt/gitlab
```
Two settings there are deliberate:
- **`hostname` is short, not an FQDN.** A long container hostname breaks Docker's internal DNS, and
the failure presents as an unreachable OIDC provider rather than as a DNS problem.
- **`GITLAB_OMNIBUS_CONFIG` only applies on first boot.** Editing it later does nothing. Change
`/srv/gitlab/config/gitlab.rb` and run `docker exec gitlab gitlab-ctl reconfigure` instead. This is
the most common way a working GitLab is broken by a correct-looking edit.
Wait for readiness rather than sleeping:
```bash
until curl -fsS http://127.0.0.1:8080/-/health >/dev/null; do sleep 5; done
```
Install the runner on the same host:
```bash
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
sudo dnf install -y gitlab-runner
```
The runner does not run in the cluster. It runs here, because the apiserver binds loopback here.
## Register
Create the runner in the project's **Settings > CI/CD > Runners** and copy the authentication token.
The legacy registration-token flow is deprecated.
```bash
gitlab-runner register \
--non-interactive \
--config /etc/gitlab-runner/config.toml \
--url https://gitlab.example.com/ \
--token <runner authentication token> \
--executor kubernetes
```
`register` writes a config with the token and nothing else useful. Replace the rest with the file
below.
## The file
Fill the `<<...>>` placeholders. The table after says why every other value is set.
```toml
concurrent = 8
check_interval = 3
[[runners]]
name = "socialweb-tenant-ci"
url = "https://gitlab.example.com/"
token = "<<runner authentication token>>"
executor = "kubernetes"
# Runs AFTER the clone, before the job's script. A STRING, not an array.
pre_build_script = "true"
# Kilobytes. Default is 4096, which is 4 MiB and is NOT enough for a Go test log.
output_limit = 51200
[runners.kubernetes]
host = "<<https://127.0.0.1:LEASED_PORT>>"
cert_file = "/etc/gitlab-runner/tenant-ci/tls.crt"
key_file = "/etc/gitlab-runner/tenant-ci/tls.key"
ca_file = "/etc/gitlab-runner/tenant-ci/ca.crt"
namespace = "gitlab-runner"
image_pull_policy = "if-not-present"
# Requests are what the scheduler places against. Set them EXPLICITLY.
cpu_request = "100m"
memory_request = "128Mi"
cpu_limit = "10"
memory_limit = "20Gi"
# Helper and init containers too, or their limits become their requests.
helper_cpu_request = "100m"
helper_memory_request = "128Mi"
helper_cpu_limit = "1"
helper_memory_limit = "1Gi"
poll_timeout = 1800
cleanup_grace_period_seconds = 30
[runners.kubernetes.affinity]
[runners.kubernetes.affinity.pod_anti_affinity]
[[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution]]
topology_key = "kubernetes.io/hostname"
[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector]
[[runners.kubernetes.affinity.pod_anti_affinity.required_during_scheduling_ignored_during_execution.label_selector.match_expressions]]
key = "project.runner.gitlab.com/id"
operator = "Exists"
```
| Key | Why |
| --- | --- |
| `concurrent` | Jobs in flight at once. One per node with anti-affinity on. `1` wedges: a stale job from a cancelled pipeline holds the only slot. |
| `host` | The apiserver's own loopback URL, from [01](01-kcp-server.md#hand-back). NOT the kcp mount URL, which is a reverse proxy and does not carry `pods/exec`. |
| `cert_file`/`key_file`/`ca_file` | The tenant credential. A client cert, NOT a bearer token. |
| `namespace` | Where job Pods go. Must exist. |
| `image_pull_policy` | `if-not-present`. Node images seed some images; always-pull throws that away. |
| `cpu_request`/`memory_request` | The scheduler uses these. Too big and the Pod never schedules. |
| `cpu_limit`/`memory_limit` | Enforcement only. The scheduler ignores them, so a limit over node size is fine. |
| `helper_*_request` | Set these. See the traps. |
| `poll_timeout` | Seconds to wait for the Pod to reach Running. Node boot plus a big image pull. |
| `output_limit` | Kilobytes of trace. Default 4096 = 4 MiB. |
| anti-affinity | One job per node, see [03](03-jobs.md#one-job-per-node). |
## Traps
**`pre_build_script` is a string, not an array.** An array is a startup failure, and the runner says
so only in its own output:
```
FATAL: Service run failed error=decoding configuration file: toml: line 9
(last key "runners.pre_build_script"): incompatible types: TOML value has type []any;
destination has type string
```
The runner exits, nothing picks work up, and GitLab still reports the runner online, so a watcher
waits for a job that never runs. Fail on `FATAL` in the runner's own output, not on GitLab's view.
**`pre_build_script` runs AFTER the clone. `pre_get_sources_script` runs BEFORE it.** Anything that
must guard the clone goes in `pre_get_sources_script`; anything that must run against the checked-out
tree goes in `pre_build_script`.
**A limit with no request IS the request.** Kubernetes defaults a container's request to its limit.
`helper_cpu_limit = 1` alone makes every job Pod ask about 2.1 CPU: the build container's request plus
the helper's and the init container's, each pulled up from its limit. A two-vCPU node answers:
```
Unschedulable: "0/1 nodes are available: 1 Insufficient cpu."
```
That reads as a slow cluster. It is a Pod that cannot fit.
**The 4 MiB log cap is the runner's, not GitLab's.**
```
Job's log exceeded limit of 4194304 bytes. Job execution will continue but no more output will be collected.
```
That string is in the `gitlab-runner` binary. GitLab's own ceiling is `ci_jobs_trace_size_limit`, a
plan limit (`project.actual_limits`) that is not in `/application/settings`. Check both before
assuming which one bit you.
**Tools are per-image.** A `pre_build_script` that assumes `corepack`, `pnpm` or `npm` fails the job
with `exit code 127` on an image that has none. Guard on the tool existing, and end with `true` so the
guard never fails the job it protects:
```toml
pre_build_script = "if ! command -v pnpm >/dev/null 2>&1 && command -v npm >/dev/null 2>&1; then npm install -g pnpm@10.11.0; fi; true"
```
## Start it
```bash
gitlab-runner run --config /etc/gitlab-runner/config.toml --working-directory /var/lib/gitlab-runner
```
Or as a service under a user that can read `/etc/gitlab-runner/tenant-ci/`.
## Check the credential before blaming the runner
```bash
openssl x509 -in /etc/gitlab-runner/tenant-ci/tls.crt -noout -subject
kubectl --kubeconfig <tenant-admin> auth can-i create pods \
--namespace gitlab-runner --as=gitlab-runner --as-group=gitlab-runner
```
## Failure modes, in the order you will meet them
| Symptom | Cause |
| --- | --- |
| Runner exits at startup | `config.toml` did not decode. Its own output says which key. |
| Job stays `pending`, no node appears | The runner reached a DIFFERENT apiserver, or the Pod went to a namespace the observer does not watch. Compare `kubectl get pods -A` with the cluster's own port. |
| `x509: certificate signed by unknown authority` | Wrong `ca_file`, or a `host` port belonging to another cluster. Ports are leased per cluster and are not shared. |
| `Unauthorized` at job start | The certificate is from another cluster's ClientCA, or the Organizations do not match the RBAC group. |
| `Forbidden` creating pods | The ClusterRoleBinding's subject does not match the certificate's Organizations. |
| Job hangs at "Preparing the environment" | `pods/exec` unreachable. This is what happens when the runner points at the kcp mount URL instead of loopback. |
| Job passes, then a retry fails | The credential expired. Check its `VALID_FOR`. |
| `Insufficient cpu`, no node appears | Pod requests do not fit a node. See the traps. |
Next: [03 - Jobs](03-jobs.md).
