# 0004 - Run streamplace's `test` job on a tenant cluster Status: done. The `test` job of `streamplace/streamplace` runs green as a Pod on a tenant cluster this host provisions: job 183, upstream `next`, 2063 s, 1104 tests, 0 failures. See section 6. Opened 2026-10-06; closed 2026-10-06. Repo: socialweb-computer-kcp. Related: `docs/gitlab/` (the operator recipes), `cmd/socialweb-clusterprovider/live_gitlab_pipeline_test.go` (the only thing in this tree that drives a real GitLab pipeline against a real tenant cluster). ## 1. Goal Run the `test` job of `streamplace/streamplace`'s `.gitlab-ci.yml` as a Pod on a tenant k3s cluster this repo provisions on THIS host, with the real GitLab at `https://gitlab.dorper-matrix.ts.net:8443` as the source of the job. The branch it runs is `kcp-tenant-ci`, which is upstream `next` (`git.stream.place`, the project's canonical home; the GitLab instance is a mirror) with every job but `test` removed. `test` is byte-for-byte upstream's, including the `STREAMPLACE_TEST_POSTGRES_COMMAND` that branch added. The first attempts ran `eli/moqt-firehose`, the mirror's default branch and three months behind; `next` is the line upstream actually builds. kcp runs on this host and only this host. The GitLab host (`ssh gitlab`, 192.168.0.49) runs the GitLab instance and must not run kcp. The job, verbatim, is: ```yaml test: stage: build interruptible: true image: "$DOCKER_REPO:builder-$DOCKERFILE_HASH" timeout: 2 hours variables: STREAMPLACE_TEST_COUNT: "1" script: - git fetch --unshallow || echo 'already unshallow' - make dev-setup && make dev && make dev-test && make ci-lexicons artifacts: when: always paths: - test.xml reports: junit: test.xml ``` `DOCKER_REPO` is `public.ecr.aws/m4j3c0j7/streamplace` and `DOCKERFILE_HASH` comes from the included `.ci/dockerfile-hash.yaml`. ## 2. What exists, measured on 2026-10-06 - The four-step proof passes on this host, unmodified: control plane Ready at zero nodes, an empty cluster that stays at zero, a node provisioned by the scale decision when a tenant workload appears, and the workload running on it. See the run in section 6. - This host carries every live input the proofs want: `/dev/kvm`, Docker, postgres 18, `kine`, `k3s`, `kubectl`, and the operator's image store under `/home/johnandersen777/kcp-image/` (nodeimage.json, `artifacts/bin/firecracker`, `artifacts/img/ubuntu-24.04.squashfs`, `out/`). - GitLab is 19.2.7 at `gitlab.dorper-matrix.ts.net:8443`. Project `streamplace/streamplace` is id 1, its default branch is `eli/moqt-firehose`, and its five recorded pipelines all FAILED. - Exactly one runner is registered: id 1, `socialwebcomputer`, instance_type, online, tags `[socialwebcomputer]`, `run_untagged` true, version 19.2.1. It is NOT a kubernetes executor. It runs on the GitLab host as `executor = "instance"` with the Fleeting plugin `fleeting-plugin-socialwebcomputer`, which provisions VMs through the market (`request-vm-ssh`). It is a different mechanism from this plan's, and an untagged job is eligible for it. ## 3. The blockers, named - **B1 - the job's image does not exist.** `public.ecr.aws/m4j3c0j7/streamplace` answers `NAME_UNKNOWN` for the repository itself, so `builder-4dc3e0c...` cannot be pulled. As written, the job fails at image pull before its first script line. Nothing else in this plan can run until this is answered. - **B2 - the guest is smaller than the job.** A node boots at `DefaultVcpu = 2`, `DefaultMemMiB = 2048`, `DefaultDiskMiB = 16` (`internal/nodevm/spec.go:41-47`). The job builds Rust (cargo), meson/C, and Go, `-p 1`, in one Pod. The image alone is multi-gigabyte and the disk is 16 GiB. - **B3 - CLOSED 2026-10-06.** A guest is addressed behind pasta (`internal/nodevm/spec.go`) and the node agent writes the guest's only nameserver as the gateway (`cmd/socialweb-nodeagent/agent.go:228`), so the guest's resolution and its default route both go through this host. Measured, not reasoned: a job pod inside a firecracker guest cloned `johnandersen777/kcp-ci-probe` from the real GitLab. See section 6. - **B4 - CLOSED 2026-10-06.** The kubernetes executor's `host` is the apiserver's own loopback address on THIS host, and its port is leased per cluster. Solved by running the runner here, on the same host: nothing has to be tunnelled, and the port comes from the cluster's own lease. The existing instance runner is paused for a run, not deleted. - **B5 - CLOSED 2026-10-06.** `Vcpu`/`MemMiB` live on the image manifest (`internal/nodefirecracker/manifest.go:18-22`) and the builder wrote neither, so they always took the defaults; `DiskMiB` was a runtime `Options` field with no clusterprovider config key. Both are now config: `vcpu` and `memMiB` on the builder's configuration, folded into the image fingerprint, and `SOCIALWEB_CLUSTERPROVIDER_NODE_DISK_MIB` for the disk. `inputs_test.go`'s declaration table is what caught that they had to be recorded as well as read. - **B7 - the guest's root filesystem is where the job's image and its build outputs have to fit.** The node's root filesystem is `rootfs.ext4`, cloned per node (`internal/nodefirecracker/node.go:153`), and containerd's content store and every container layer live inside it. Its size is the staged tree plus `rootfs_headroom_mib`, and nothing else sizes it. The node disk is NOT an option: it is attached and mounted READ-ONLY as the identity disk (`cmd/socialweb-nodeagent/agent.go:173`), so `SOCIALWEB_CLUSTERPROVIDER_NODE_DISK_MIB` cannot hold state. Measured twice: at the default headroom the pull of the job's image dies with `no space left on device` writing `/var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content/ingest/...`, and at `rootfs_headroom_mib: 55000` the root filesystem is 58,401,488,896 bytes and the pull completes. A corollary that bit once: those files are written into the test's state directory, which is `os.MkdirTemp("", ...)` and therefore `/tmp`. On this host `/tmp` is a 24 GiB tmpfs, so the 58 GB root filesystem and the 150 GiB node disk have to land somewhere else -- the run is made with `TMPDIR` set to a directory on the host's own disk. A state directory that cannot hold a node's disks fails as a full filesystem partway through a boot, not as a configuration error. - **B10 - the job's own log exceeds the RUNNER's cap, not GitLab's.** `make dev-test` runs `go test -p 1 -timeout 30m ./pkg/... -v | tee /dev/stderr`, so the trace is the whole verbose test log, and it stops at 4,194,304 bytes: `Job's log exceeded limit of 4194304 bytes. Job execution will continue but no more output will be collected.` That string is in the `gitlab-runner` binary, and the key behind it is `output_limit`, "Maximum build trace size in kilobytes", default 4096 -- so the runner, not the server, was the binding constraint. Answering it the wrong way first is recorded because the wrong answer is plausible and cost a GitLab reconfigure: `ci_jobs_trace_size_limit` exists in `plan_limits` and `project.actual_limits` (`app/services/ci/append_build_trace_service.rb:87`) and this instance already had it at 100 MB, and `/application/settings` does not carry it at all. The runner config now sets `output_limit = 51200` (50 MB), between the 20 MB wanted and GitLab's own 100 MB ceiling so the server stays the outer bound. `gitlab.rb` is restored byte-for-byte; the key it briefly carried was never read by GitLab, so the generated `gitlab.yml` did not change. - **B9 - CLOSED 2026-10-06, by rebasing.** The published builder image for `eli/moqt-firehose`'s Dockerfile hash is a `builder-` tag built 2025-05-23, and its Rust predates let-chains (stable 1.88): `make dev-setup` died at `FAILED: rust/iroh-streamplace/libiroh_streamplace.so` with `error[E0658]: let expressions in this position are unstable` compiling `iroh-smol-kv`. The repo pins no `rust-toolchain.toml`, so rustup used the image's default. Upstream's `next` names a `DOCKERFILE_HASH` whose image was built 2026-10-04, so the branch is rebased onto `next` and the stale image is not used at all. - **B8 - the `eli/moqt-firehose` builder image has no `pnpm`.** The job's first make target is `dev-setup` -> `app-cached` -> `make install` -> `pnpm install`, and in `ghcr.io/streamplace/streamplace:builder-` that fails with `make[2]: pnpm: No such file or directory`. A fresh build of the SAME `docker/build.Dockerfile` on this host has `/usr/local/bin/pnpm`, a symlink to corepack's `pnpm.js`, so the published tag predates the corepack that writes those shims -- the builder image is named by the Dockerfile's hash, and no published tag matches this branch's hash. The project's own macOS job runs `npm install -g corepack@latest --force && corepack enable` explicitly, which is the same step. Answered runner-side, see D7. - **B6 - the node image cannot carry a large image.** `layDownRootfs` writes the root filesystem with `mkfs.ext4 -d`, which refuses a file of about 2 GiB or more (`__populate_fs: Ext2 file too big while writing file ...`). A 3.3 GB tarball of the builder image is refused at 6 GB and at 12 GB, with the file alone and with the whole stage; a 3.5 GB file of zeros is accepted at 4.9 GB and 6.5 GB, and 1 GB of the same tarball is accepted. Measured, not assumed; see section 7. So the node image cannot seed this job's image, and the guest pulls it instead. ## 4. Decisions - **D1 - kcp runs here; the runner is the part that moves.** The apiserver binds loopback on this host, so a kubernetes-executor runner is run here. An `ssh -R` tunnel from here to the GitLab host is the fallback if the runner has to live beside GitLab; it is a raw TCP forward, so it preserves the SPDY `pods/exec` paths the kcp mount URL does not, and `docs/gitlab/README.md`'s warning is about the mount, not about a tunnel. - **D2 - prove the integration with a small job before the real one.** A separate GitLab project carries the probe job, so the real project's `.gitlab-ci.yml` is never overwritten by an experiment and the real job changes nothing until it is the thing being measured. - **D3 - pause, do not delete, the existing runner.** Runner 1 is someone's working setup for a different mechanism. An untagged job is eligible for it, so it is paused for the duration of a run and restored after. Deleting it is not this plan's to do. - **D4 - the job image comes from the registry upstream already publishes to, and upstream already says so.** B1 has no fix to make: `public.ecr.aws/m4j3c0j7/streamplace` answers `NAME_UNKNOWN`, but upstream's own `.gitlab-ci.yml` sets `DOCKER_REPO: "ghcr.io/streamplace/streamplace"` and publishes `builder-` there from `.github/workflows/docker.yaml`. On `eli/moqt-firehose` -- the June branch this plan started from -- `DOCKER_REPO` still named the ECR repo, which is why the first attempts had to work out the ghcr path at all. Rebasing onto `upstream/next` removed that problem rather than solving it: the branch now carries upstream's own `variables:` block, and the tag its own `.ci/dockerfile-hash.yaml` names (`builder-3e14c389b80a359787d73741f221fdfaacb12f53`) exists, built 2026-10-04. **Seeding the image into the node image was tried first and does not work**: the node image is written with `mkfs.ext4 -d`, and it refuses a file larger than about 2 GiB (`Ext2 file too big while writing file ...`), while the builder tarball is 3.3 GB. That is measured, not assumed -- see section 7. - **D5 - the job runs from a branch that carries only the job.** The runner is `concurrent = 1`, this host runs one node, and every other job in `streamplace/.gitlab-ci.yml` is untagged, so the other build jobs would each take the only slot for up to their own 2h timeout and `test` would never be reached -- it sorts last in its stage. A branch `kcp-tenant-ci` therefore carries the `test` job verbatim and nothing else, and names the registry this project publishes its builder image to. The job's own keys -- image, script, timeout, artifacts -- are byte-for-byte what the branch this is measured against has. - **D8 - the runner is not pinned to one job, and the harness stopped dropping the scale knobs.** `concurrent = 1` was taken from `docs/gitlab/README.md`, which says to set it until the cluster can place more than one job. That limit is gone -- the scale decision carries a target total -- so a run with the old value has a single slot that a job from any earlier, cancelled pipeline can take, and the reading then comes from a pipeline nobody was watching. Measured: a cancelled run's job held the slot and the current pipeline sat `pending`. The harness also passed `SCALE_MAX_NODES` and `SCALE_PODS_PER_NODE` through but not `NODE_DISK_MIB`, so the disk knob this plan added was dropped by the code written to exercise it. - **D7 - pnpm is provided by the runner, not by the job.** B8 is a property of the image, and the image is named by a Dockerfile hash nothing published matches. Editing the job to add a setup step would change the thing being measured, so the step goes where the project already puts it for macOS -- a `pre_build_script` in the runner's `config.toml`, which runs in the build container before the job's script. It prefers `corepack enable`, which honours the `packageManager` field the project pins, and falls back to installing the version that field names. Against upstream `next`'s image it is a no-op -- the step prints nothing, because `command -v pnpm` succeeds -- which is the reading that says the `next` image carries the shims the `eli/moqt-firehose` one did not. It is left in as the guard it turned out to be. - **D6 - the runner gets a scoped credential, not the superuser.** `docs/gitlab/02` is explicit that handing a runner `system:admin`/`system:masters` makes every RBAC rule decorative. The existing live test mints the superuser credential; the new one mints `gitlab-runner` / `gitlab-runner` from the ClientCA on disk and applies the ClusterRole and ClusterRoleBinding the document lists, so the identity under test is the one a real deployment would use. ## 5. Steps 1. Four-step proof on this host. DONE (section 6). 2. Egress: a guest reaching GitLab. DONE (section 6), by running a cloning job rather than by dialing -- the clone is the thing the job actually does. 3. A small pipeline from a probe project, run by a kubernetes-executor runner registered for it, against a tenant cluster this host provisioned. DONE (section 6), with the scoped identity. 4. Sizing: `vcpu`, `memMiB` and `diskMiB`, plumbed to the manifest and the node provider, so a node can be sized for a real job. DONE, committed. Closes B5 and B2. 5. The job image. DONE, by pointing the branch at the registry this project already publishes the builder stage to. Closes B1; B6 is the reason seeding was abandoned. 6. The real `test` job, run through the real GitLab, with the job's own trace and `test.xml` as the reading. IN FLIGHT. ## 6. Readings **2026-10-06, a real GitLab pipeline runs a job on a tenant cluster.** `TestAGitLabPipelineRunsOnATenantCluster` against the real GitLab and the probe project, PASS, 37.58 s. Pipeline 46 succeeded; GitLab's runner created `runner-8t2uishn2-project-5-concurrent-0-ny1rvwue` and this cluster placed it on `liveweb-node1`, the node the scale decision provisioned. The existing instance runner (`socialwebcomputer`, id 1) was paused for the run so the reading could not have come from it, and is restored after. **2026-10-06, a job pod clones its repository from inside the guest.** `TestALiveGitLabJobRunsOnATenantCluster` against the probe project, PASS, 42.51 s. The job's own trace: ``` Getting source from Git repository Fetching changes with git depth set to 20... Initialized empty Git repository in /builds/johnandersen777/kcp-ci-probe/.git/ Checking out c6d139c1 as detached HEAD (ref is main) ... GIT_DIR_PRESENT Job succeeded ``` This is the reading that closes B3. It also shows the runner's helper image being pulled from the internet by the guest, because the helper image is not one the node image seeds. **2026-10-06, four-step proof, this host, firecracker substrate.** PASS. The control plane reached `Ready` at `https://127.0.0.1:62202` with zero nodes; `kubectl get nodes` was empty and stayed empty across 10 passes 2 s apart over 20 s; the scale decision read `Provision: intent and no node`; the node joined and the cloud controller manager stamped it `providerID=socialweb://liveweb/liveweb-node1`; `INSTANCE_START_TO_READY_S=5.504` measured from the firecracker VMM's instance record to the Node's `Ready` condition at 250 ms resolution. **2026-10-06, the job's own script, as far as it has run.** The `test` job of project 1 on `kcp-tenant-ci`, pipeline 57, job 181, running as a Pod on `liveweb-node1`: ``` $ make dev-setup && make dev && make dev-test && make ci-lexicons pnpm install ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-10.11.0.tgz Content-addressable store is at: /builds/.pnpm-store/v10 FFmpeg| Has header "pthread_np.h" : NO HEAD is now at 20cbb63 meson: fix x264 version check without stdint.h header proxy-libintl| ... meson.build:14: WARNING: ... Progress: resolved 0, reused 0, downloaded 2917, added 3875 ``` That is the target itself: the job's first `make` target, the pinned pnpm, the meson configure of FFmpeg from source, and the subproject fetches, all inside a firecracker guest this host booted because a Pod could not be placed. **2026-10-06, the streamplace `test` job PASSES on a tenant cluster.** Pipeline 57, job 183, branch `kcp-tenant-ci` (upstream `next` with only `test` kept), project 1 on the real GitLab at `gitlab.dorper-matrix.ts.net:8443`. Job status `success`, duration **2063 s**, `failure_reason` `None`. `TestALiveGitLabJobRunsOnATenantCluster` PASS, 2095.10 s, and it records the placement: ``` STEP 10 job test ran in pod(s) map[runner-hhvwl7zg6-project-1-concurrent-0-tifm253t:liveweb-node1], placed by this cluster ``` `liveweb-node1` is a firecracker guest this host booted because the job's Pod was Pending and `internal/scale` read that as intent. The Job's own junit artifact, `test.xml`, downloaded from `/projects/1/jobs/183/artifacts` and 19,820,634 bytes: ``` testsuites 82 tests 1104 failures 0 errors 0 skipped 15 stream.place/streamplace/pkg/api 67 tests 0 failures stream.place/streamplace/pkg/aqtime 38 tests 0 failures stream.place/streamplace/pkg/viewlog 38 tests 0 failures stream.place/streamplace/pkg/vod 26 tests 0 failures ``` That is `make dev-setup && make dev && make dev-test && make ci-lexicons` run to exit 0: FFmpeg, GStreamer, x264 and libav built from source in the guest, the Rust `iroh-streamplace` cdylib built, the Go tree built, and 1104 tests green -- on a cluster whose control plane was at zero nodes when the job arrived. The console trace is truncated at 4 MiB by the runner's `output_limit` (B10); the junit artifact is the complete record and is the better instrument anyway. ## 6a. Autoscaling to five nodes, and back to zero The same instrument, pointed at the scale decision instead of at one job. The branch carries five `scale-check-N` jobs in place of `test` -- `test` is commented out rather than deleted, because the two need different cluster shapes: a toolchain build wants one large node, an autoscale check wants five small ones at once. Each job sleeps, so all five Pods are alive together; a job that exits immediately is a Pod that is gone before the second one is scheduled. **One job per node is configured, not hoped for.** The runner's `podAntiAffinity` is first-class config, unlike `pod_spec` which needs the beta `FF_USE_ADVANCED_POD_SPEC_CONFIGURATION` flag: ```toml [runners.kubernetes.affinity] [runners.kubernetes.affinity.pod_anti_affinity] [[...required_during_scheduling_ignored_during_execution]] topology_key = "kubernetes.io/hostname" [...label_selector.match_expressions] key = "project.runner.gitlab.com/id" operator = "Exists" ``` `project.runner.gitlab.com/id` is a label the runner puts on every job Pod and that nothing else carries, so the Pods repel each other and nothing else. `Demand` counts Pods that are Pending, Running or Unknown, so five live job Pods are a target of five at one Pod per node, and zero once they are gone. One thing about the harness that costs a run to learn: **the scale decision is DRIVEN, not running.** `fourstepStack.pass` calls `prov.Reconcile` and `driver.RunPass` explicitly, so a test that stops asking never sees a decision and waits for a teardown no one requested -- which is why the other live tests call `stack.choose(t, ctx, scale.Teardown)`. The first version of this test waited for the cluster to fall to zero on its own and sat at five nodes until the deadline. `TestFiveGitLabJobPodsScaleToFiveNodesAndBackToZero` **PASS, 346.11 s**, on 2026-10-06 against the real GitLab, pipeline 69, nodes of 2 vCPU and 3072 MiB, five `scale-check-N` jobs: ``` STEP 3 the cluster is at zero nodes before anything is applied: "" STEP 5 5 job pods exist in gitlab-runner STEP 6 the scale decision read Provision with nodes=5: intent and no node STEP 7 5 nodes joined on demand; `kubectl get nodes`: liveweb-node1 .. liveweb-node5, all Ready STEP 8 each job pod is on its own node: liveweb-node1 <- ...-concurrent-0 liveweb-node4 <- ...-concurrent-3 liveweb-node2 <- ...-concurrent-1 liveweb-node5 <- ...-concurrent-4 liveweb-node3 <- ...-concurrent-2 STEP 9 all 5 scale-check jobs finished success STEP 10 the job pods are gone (0 left), so nothing is asking for a node STEP 11 the scale decision read Teardown with nodes=0: nothing running for 1m0.917972328s, past the 1m0s hold STEP 12 the cluster is back at zero nodes; `kubectl get nodes` said "" STEP 13 the cluster is at zero nodes and its control plane is still Ready at https://127.0.0.1:62302 ``` That is the whole loop: zero, five nodes because five Pods could not be placed, one job per node, the work finishing, and the cluster falling back to zero on its own hold while the apiserver keeps answering. Nine GitLab jobs ran on this cluster across the three passes it took to get here, and the fifth pass is the one that is a result. Observed mid-flight, one node at a time: with one node up, three Pods sat `Init` on it and two stayed `Pending` -- the anti-affinity refusing to co-schedule -- and the two Pending Pods are what took the count to three, four and five. That is the loop working, not a coincidence of scheduling. The nodes of a scale check are small on purpose: five of the nodes the `test` job needs would be 60 vCPU and 120 GiB, which this host does not have. The Pod requests drop to 100m/128Mi for the same reason -- requests are what the scheduler places against, and limits stay generous because limits are not placement. **Two things the first attempt at this got wrong, both worth keeping.** - **A limit is a request when no request is given.** `helper_cpu_limit = 1` with no helper request made every job Pod ask for about 2.1 CPU -- the build container's own request plus the helper's and the init container's, each pulled up from its limit -- and a two-vCPU node answers that with `Unschedulable: "0/1 nodes are available: 1 Insufficient cpu"`. The reading is a Pod that sits `Pending` while nodes join around it, which looks like a slow cluster and is actually a Pod that cannot fit on any of it. `helper_cpu_request` and `helper_memory_request` exist; setting them says where the Pod is placed without capping what it may burst to. - **A job Pod resolves through CoreDNS, and CoreDNS is an addon that comes up after the node it runs on.** The scale-check jobs cloned like every other job, and the first attempt lost three of five to `fatal: unable to access '...': Could not resolve host: gitlab.dorper-matrix.ts.net (Timeout while contacting DNS servers)`, reported as `script_failure` -- which reads as a broken job. A Pod's `/etc/resolv.conf` is k3s's `kube-dns`, so the query goes Pod -> CoreDNS -> the node's own resolver -> pasta -> the host, and CoreDNS is created when the first node joins and is not serving for the first seconds after. A job that starts the moment its node joins is a job that queries a resolver that is not there yet. **pasta is not the fault, and it was the first thing suspected.** Querying its forwarder at the gateway address from inside its own namespace answers `gitlab.dorper-matrix.ts.net`, `github.com` and `ghcr.io` in about ten milliseconds each. The host's `nameserver 127.0.0.53` is systemd-resolved's stub and pasta reaches it fine, tailscale MagicDNS included. The fault is one hop further in. The fix is `pre_get_sources_script`, which runs BEFORE the clone; `pre_build_script` runs after it, which is why the pnpm step could never have covered this. The gate waits for the job's own nameserver to answer for `CI_SERVER_HOST` -- the name the clone is about to need -- and gives up quietly rather than failing the job it exists to protect. **It is not a startup delay, and it does not go away.** Measured on a live five-node cluster, from a Pod on each node, TCP to the CoreDNS Pod's own IP: ``` liveweb-node1 <- CoreDNS runs here REACHABLE liveweb-node2 UNREACHABLE liveweb-node3 UNREACHABLE liveweb-node4 UNREACHABLE liveweb-node5 UNREACHABLE ``` and `nslookup` inside those four answers `connection timed out; no servers could be reached`. The reason is not CoreDNS: **a guest cannot reach another guest.** Each sits behind its own pasta instance, pasta is user-mode networking, and it carries a guest's traffic out to the host and nothing between two guests, so the nodes have no path to each other and k3s's flannel overlay has nothing to run over. Cluster DNS is one hop that happens to need that path. So this is a limit of the node network and not a race in the job: **anything that resolves a name works on a one-node cluster and on whichever node CoreDNS is on, and fails on the others.** It is recorded in `NEXT.yaml` as a limit, with the pod-to-pod reading above as its instrument. The scale-check jobs therefore carry `GIT_STRATEGY: none` -- not to route around the failure but because a check of the scale decision must not depend on a network path the cluster does not have -- and the `pre_get_sources_script` wait stays for the jobs that do clone. Routing around this silently was the first response, by giving those jobs `GIT_STRATEGY: none` before the cause was known; that is recorded because it is the tempting answer, it hides the failure in the one check whose whole job is to find it, and it leaves every other job exposed. ## 7. The mkfs measurement, in full B6 is the reason the job's image is pulled rather than seeded, so it is written down rather than summarized. Every line is `mkfs.ext4 -q -F -d -L