~ knotless Search/
↑↓ move↵ open⇧↵ all results

johnandersen777.bsky.social/socialweb-computer-kcp
Switch repository
Loading…

0
0
main
Switch branches or tags
435 lines · 27.9 KB Raw History

0004 - Run streamplace's test job on a tenant cluster

Status: done. The test job of streamplace/streamplace runs green as a Pod on a tenant cluster this host provisions: job 183, upstream next, 2063 s, 1104 tests, 0 failures. See section 6. Opened 2026-10-06; closed 2026-10-06. Repo: socialweb-computer-kcp. Related: docs/gitlab/ (the operator recipes), cmd/socialweb-clusterprovider/live_gitlab_pipeline_test.go (the only thing in this tree that drives a real GitLab pipeline against a real tenant cluster).

1. Goal

Run the test job of streamplace/streamplace's .gitlab-ci.yml as a Pod on a tenant k3s cluster this repo provisions on THIS host, with the real GitLab at https://gitlab.dorper-matrix.ts.net:8443 as the source of the job.

The branch it runs is kcp-tenant-ci, which is upstream next (git.stream.place, the project's canonical home; the GitLab instance is a mirror) with every job but test removed. test is byte-for-byte upstream's, including the STREAMPLACE_TEST_POSTGRES_COMMAND that branch added. The first attempts ran eli/moqt-firehose, the mirror's default branch and three months behind; next is the line upstream actually builds.

kcp runs on this host and only this host. The GitLab host (ssh gitlab, 192.168.0.49) runs the GitLab instance and must not run kcp.

The job, verbatim, is:

test:
  stage: build
  interruptible: true
  image: "$DOCKER_REPO:builder-$DOCKERFILE_HASH"
  timeout: 2 hours
  variables:
    STREAMPLACE_TEST_COUNT: "1"
  script:
    - git fetch --unshallow || echo 'already unshallow'
    - make dev-setup && make dev && make dev-test && make ci-lexicons
  artifacts:
    when: always
    paths:
      - test.xml
    reports:
      junit: test.xml

DOCKER_REPO is public.ecr.aws/m4j3c0j7/streamplace and DOCKERFILE_HASH comes from the included .ci/dockerfile-hash.yaml.

2. What exists, measured on 2026-10-06

  • The four-step proof passes on this host, unmodified: control plane Ready at zero nodes, an empty cluster that stays at zero, a node provisioned by the scale decision when a tenant workload appears, and the workload running on it. See the run in section 6.
  • This host carries every live input the proofs want: /dev/kvm, Docker, postgres 18, kine, k3s, kubectl, and the operator's image store under /home/johnandersen777/kcp-image/ (nodeimage.json, artifacts/bin/firecracker, artifacts/img/ubuntu-24.04.squashfs, out/).
  • GitLab is 19.2.7 at gitlab.dorper-matrix.ts.net:8443. Project streamplace/streamplace is id 1, its default branch is eli/moqt-firehose, and its five recorded pipelines all FAILED.
  • Exactly one runner is registered: id 1, socialwebcomputer, instance_type, online, tags [socialwebcomputer], run_untagged true, version 19.2.1. It is NOT a kubernetes executor. It runs on the GitLab host as executor = "instance" with the Fleeting plugin fleeting-plugin-socialwebcomputer, which provisions VMs through the market (request-vm-ssh). It is a different mechanism from this plan's, and an untagged job is eligible for it.

3. The blockers, named

  • B1 - the job's image does not exist. public.ecr.aws/m4j3c0j7/streamplace answers NAME_UNKNOWN for the repository itself, so builder-4dc3e0c... cannot be pulled. As written, the job fails at image pull before its first script line. Nothing else in this plan can run until this is answered.
  • B2 - the guest is smaller than the job. A node boots at DefaultVcpu = 2, DefaultMemMiB = 2048, DefaultDiskMiB = 16 (internal/nodevm/spec.go:41-47). The job builds Rust (cargo), meson/C, and Go, -p 1, in one Pod. The image alone is multi-gigabyte and the disk is 16 GiB.
  • B3 - CLOSED 2026-10-06. A guest is addressed behind pasta (internal/nodevm/spec.go) and the node agent writes the guest's only nameserver as the gateway (cmd/socialweb-nodeagent/agent.go:228), so the guest's resolution and its default route both go through this host. Measured, not reasoned: a job pod inside a firecracker guest cloned johnandersen777/kcp-ci-probe from the real GitLab. See section 6.
  • B4 - CLOSED 2026-10-06. The kubernetes executor's host is the apiserver's own loopback address on THIS host, and its port is leased per cluster. Solved by running the runner here, on the same host: nothing has to be tunnelled, and the port comes from the cluster's own lease. The existing instance runner is paused for a run, not deleted.
  • B5 - CLOSED 2026-10-06. Vcpu/MemMiB live on the image manifest (internal/nodefirecracker/manifest.go:18-22) and the builder wrote neither, so they always took the defaults; DiskMiB was a runtime Options field with no clusterprovider config key. Both are now config: vcpu and memMiB on the builder's configuration, folded into the image fingerprint, and SOCIALWEB_CLUSTERPROVIDER_NODE_DISK_MIB for the disk. inputs_test.go's declaration table is what caught that they had to be recorded as well as read.
  • B7 - the guest's root filesystem is where the job's image and its build outputs have to fit. The node's root filesystem is rootfs.ext4, cloned per node (internal/nodefirecracker/node.go:153), and containerd's content store and every container layer live inside it. Its size is the staged tree plus rootfs_headroom_mib, and nothing else sizes it. The node disk is NOT an option: it is attached and mounted READ-ONLY as the identity disk (cmd/socialweb-nodeagent/agent.go:173), so SOCIALWEB_CLUSTERPROVIDER_NODE_DISK_MIB cannot hold state. Measured twice: at the default headroom the pull of the job's image dies with no space left on device writing /var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content/ingest/..., and at rootfs_headroom_mib: 55000 the root filesystem is 58,401,488,896 bytes and the pull completes. A corollary that bit once: those files are written into the test's state directory, which is os.MkdirTemp("", ...) and therefore /tmp. On this host /tmp is a 24 GiB tmpfs, so the 58 GB root filesystem and the 150 GiB node disk have to land somewhere else -- the run is made with TMPDIR set to a directory on the host's own disk. A state directory that cannot hold a node's disks fails as a full filesystem partway through a boot, not as a configuration error.
  • B10 - the job's own log exceeds the RUNNER's cap, not GitLab's. make dev-test runs go test -p 1 -timeout 30m ./pkg/... -v | tee /dev/stderr, so the trace is the whole verbose test log, and it stops at 4,194,304 bytes: Job's log exceeded limit of 4194304 bytes. Job execution will continue but no more output will be collected. That string is in the gitlab-runner binary, and the key behind it is output_limit, "Maximum build trace size in kilobytes", default 4096 -- so the runner, not the server, was the binding constraint. Answering it the wrong way first is recorded because the wrong answer is plausible and cost a GitLab reconfigure: ci_jobs_trace_size_limit exists in plan_limits and project.actual_limits (app/services/ci/append_build_trace_service.rb:87) and this instance already had it at 100 MB, and /application/settings does not carry it at all. The runner config now sets output_limit = 51200 (50 MB), between the 20 MB wanted and GitLab's own 100 MB ceiling so the server stays the outer bound. gitlab.rb is restored byte-for-byte; the key it briefly carried was never read by GitLab, so the generated gitlab.yml did not change.
  • B9 - CLOSED 2026-10-06, by rebasing. The published builder image for eli/moqt-firehose's Dockerfile hash is a builder- tag built 2025-05-23, and its Rust predates let-chains (stable 1.88): make dev-setup died at FAILED: rust/iroh-streamplace/libiroh_streamplace.so with error[E0658]: let expressions in this position are unstable compiling iroh-smol-kv. The repo pins no rust-toolchain.toml, so rustup used the image's default. Upstream's next names a DOCKERFILE_HASH whose image was built 2026-10-04, so the branch is rebased onto next and the stale image is not used at all.
  • B8 - the eli/moqt-firehose builder image has no pnpm. The job's first make target is dev-setup -> app-cached -> make install -> pnpm install, and in ghcr.io/streamplace/streamplace:builder- that fails with make[2]: pnpm: No such file or directory. A fresh build of the SAME docker/build.Dockerfile on this host has /usr/local/bin/pnpm, a symlink to corepack's pnpm.js, so the published tag predates the corepack that writes those shims -- the builder image is named by the Dockerfile's hash, and no published tag matches this branch's hash. The project's own macOS job runs npm install -g corepack@latest --force && corepack enable explicitly, which is the same step. Answered runner-side, see D7.
  • B6 - the node image cannot carry a large image. layDownRootfs writes the root filesystem with mkfs.ext4 -d, which refuses a file of about 2 GiB or more (__populate_fs: Ext2 file too big while writing file ...). A 3.3 GB tarball of the builder image is refused at 6 GB and at 12 GB, with the file alone and with the whole stage; a 3.5 GB file of zeros is accepted at 4.9 GB and 6.5 GB, and 1 GB of the same tarball is accepted. Measured, not assumed; see section 7. So the node image cannot seed this job's image, and the guest pulls it instead.

4. Decisions

  • D1 - kcp runs here; the runner is the part that moves. The apiserver binds loopback on this host, so a kubernetes-executor runner is run here. An ssh -R tunnel from here to the GitLab host is the fallback if the runner has to live beside GitLab; it is a raw TCP forward, so it preserves the SPDY pods/exec paths the kcp mount URL does not, and docs/gitlab/README.md's warning is about the mount, not about a tunnel.
  • D2 - prove the integration with a small job before the real one. A separate GitLab project carries the probe job, so the real project's .gitlab-ci.yml is never overwritten by an experiment and the real job changes nothing until it is the thing being measured.
  • D3 - pause, do not delete, the existing runner. Runner 1 is someone's working setup for a different mechanism. An untagged job is eligible for it, so it is paused for the duration of a run and restored after. Deleting it is not this plan's to do.
  • D4 - the job image comes from the registry upstream already publishes to, and upstream already says so. B1 has no fix to make: public.ecr.aws/m4j3c0j7/streamplace answers NAME_UNKNOWN, but upstream's own .gitlab-ci.yml sets DOCKER_REPO: "ghcr.io/streamplace/streamplace" and publishes builder-<DOCKERFILE_HASH> there from .github/workflows/docker.yaml. On eli/moqt-firehose -- the June branch this plan started from -- DOCKER_REPO still named the ECR repo, which is why the first attempts had to work out the ghcr path at all. Rebasing onto upstream/next removed that problem rather than solving it: the branch now carries upstream's own variables: block, and the tag its own .ci/dockerfile-hash.yaml names (builder-3e14c389b80a359787d73741f221fdfaacb12f53) exists, built 2026-10-04. Seeding the image into the node image was tried first and does not work: the node image is written with mkfs.ext4 -d, and it refuses a file larger than about 2 GiB (Ext2 file too big while writing file ...), while the builder tarball is 3.3 GB. That is measured, not assumed -- see section 7.
  • D5 - the job runs from a branch that carries only the job. The runner is concurrent = 1, this host runs one node, and every other job in streamplace/.gitlab-ci.yml is untagged, so the other build jobs would each take the only slot for up to their own 2h timeout and test would never be reached -- it sorts last in its stage. A branch kcp-tenant-ci therefore carries the test job verbatim and nothing else, and names the registry this project publishes its builder image to. The job's own keys -- image, script, timeout, artifacts -- are byte-for-byte what the branch this is measured against has.
  • D8 - the runner is not pinned to one job, and the harness stopped dropping the scale knobs. concurrent = 1 was taken from docs/gitlab/README.md, which says to set it until the cluster can place more than one job. That limit is gone -- the scale decision carries a target total -- so a run with the old value has a single slot that a job from any earlier, cancelled pipeline can take, and the reading then comes from a pipeline nobody was watching. Measured: a cancelled run's job held the slot and the current pipeline sat pending. The harness also passed SCALE_MAX_NODES and SCALE_PODS_PER_NODE through but not NODE_DISK_MIB, so the disk knob this plan added was dropped by the code written to exercise it.
  • D7 - pnpm is provided by the runner, not by the job. B8 is a property of the image, and the image is named by a Dockerfile hash nothing published matches. Editing the job to add a setup step would change the thing being measured, so the step goes where the project already puts it for macOS -- a pre_build_script in the runner's config.toml, which runs in the build container before the job's script. It prefers corepack enable, which honours the packageManager field the project pins, and falls back to installing the version that field names. Against upstream next's image it is a no-op -- the step prints nothing, because command -v pnpm succeeds -- which is the reading that says the next image carries the shims the eli/moqt-firehose one did not. It is left in as the guard it turned out to be.
  • D6 - the runner gets a scoped credential, not the superuser. docs/gitlab/02 is explicit that handing a runner system:admin/system:masters makes every RBAC rule decorative. The existing live test mints the superuser credential; the new one mints gitlab-runner / gitlab-runner from the ClientCA on disk and applies the ClusterRole and ClusterRoleBinding the document lists, so the identity under test is the one a real deployment would use.

5. Steps

  1. Four-step proof on this host. DONE (section 6).
  2. Egress: a guest reaching GitLab. DONE (section 6), by running a cloning job rather than by dialing -- the clone is the thing the job actually does.
  3. A small pipeline from a probe project, run by a kubernetes-executor runner registered for it, against a tenant cluster this host provisioned. DONE (section 6), with the scoped identity.
  4. Sizing: vcpu, memMiB and diskMiB, plumbed to the manifest and the node provider, so a node can be sized for a real job. DONE, committed. Closes B5 and B2.
  5. The job image. DONE, by pointing the branch at the registry this project already publishes the builder stage to. Closes B1; B6 is the reason seeding was abandoned.
  6. The real test job, run through the real GitLab, with the job's own trace and test.xml as the reading. IN FLIGHT.

6. Readings

2026-10-06, a real GitLab pipeline runs a job on a tenant cluster. TestAGitLabPipelineRunsOnATenantCluster against the real GitLab and the probe project, PASS, 37.58 s. Pipeline 46 succeeded; GitLab's runner created runner-8t2uishn2-project-5-concurrent-0-ny1rvwue and this cluster placed it on liveweb-node1, the node the scale decision provisioned. The existing instance runner (socialwebcomputer, id 1) was paused for the run so the reading could not have come from it, and is restored after.

2026-10-06, a job pod clones its repository from inside the guest. TestALiveGitLabJobRunsOnATenantCluster against the probe project, PASS, 42.51 s. The job's own trace:

Getting source from Git repository
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/johnandersen777/kcp-ci-probe/.git/
Checking out c6d139c1 as detached HEAD (ref is main)
...
GIT_DIR_PRESENT
Job succeeded

This is the reading that closes B3. It also shows the runner's helper image being pulled from the internet by the guest, because the helper image is not one the node image seeds.

2026-10-06, four-step proof, this host, firecracker substrate. PASS. The control plane reached Ready at https://127.0.0.1:62202 with zero nodes; kubectl get nodes was empty and stayed empty across 10 passes 2 s apart over 20 s; the scale decision read Provision: intent and no node; the node joined and the cloud controller manager stamped it providerID=socialweb://liveweb/liveweb-node1; INSTANCE_START_TO_READY_S=5.504 measured from the firecracker VMM's instance record to the Node's Ready condition at 250 ms resolution.

2026-10-06, the job's own script, as far as it has run. The test job of project 1 on kcp-tenant-ci, pipeline 57, job 181, running as a Pod on liveweb-node1:

$ make dev-setup && make dev && make dev-test && make ci-lexicons
pnpm install
! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-10.11.0.tgz
  Content-addressable store is at: /builds/.pnpm-store/v10
FFmpeg| Has header "pthread_np.h" : NO
HEAD is now at 20cbb63 meson: fix x264 version check without stdint.h header
proxy-libintl| ... meson.build:14: WARNING: ...
Progress: resolved 0, reused 0, downloaded 2917, added 3875

That is the target itself: the job's first make target, the pinned pnpm, the meson configure of FFmpeg from source, and the subproject fetches, all inside a firecracker guest this host booted because a Pod could not be placed.

2026-10-06, the streamplace test job PASSES on a tenant cluster. Pipeline 57, job 183, branch kcp-tenant-ci (upstream next with only test kept), project 1 on the real GitLab at gitlab.dorper-matrix.ts.net:8443. Job status success, duration 2063 s, failure_reason None. TestALiveGitLabJobRunsOnATenantCluster PASS, 2095.10 s, and it records the placement:

STEP 10 job test ran in pod(s)
  map[runner-hhvwl7zg6-project-1-concurrent-0-tifm253t:liveweb-node1],
  placed by this cluster

liveweb-node1 is a firecracker guest this host booted because the job's Pod was Pending and internal/scale read that as intent. The Job's own junit artifact, test.xml, downloaded from /projects/1/jobs/183/artifacts and 19,820,634 bytes:

testsuites 82   tests 1104   failures 0   errors 0   skipped 15
  stream.place/streamplace/pkg/api   67 tests  0 failures
  stream.place/streamplace/pkg/aqtime 38 tests  0 failures
  stream.place/streamplace/pkg/viewlog 38 tests 0 failures
  stream.place/streamplace/pkg/vod    26 tests  0 failures

That is make dev-setup && make dev && make dev-test && make ci-lexicons run to exit 0: FFmpeg, GStreamer, x264 and libav built from source in the guest, the Rust iroh-streamplace cdylib built, the Go tree built, and 1104 tests green -- on a cluster whose control plane was at zero nodes when the job arrived. The console trace is truncated at 4 MiB by the runner's output_limit (B10); the junit artifact is the complete record and is the better instrument anyway.

6a. Autoscaling to five nodes, and back to zero

The same instrument, pointed at the scale decision instead of at one job. The branch carries five scale-check-N jobs in place of test -- test is commented out rather than deleted, because the two need different cluster shapes: a toolchain build wants one large node, an autoscale check wants five small ones at once. Each job sleeps, so all five Pods are alive together; a job that exits immediately is a Pod that is gone before the second one is scheduled.

One job per node is configured, not hoped for. The runner's podAntiAffinity is first-class config, unlike pod_spec which needs the beta FF_USE_ADVANCED_POD_SPEC_CONFIGURATION flag:

    [runners.kubernetes.affinity]
      [runners.kubernetes.affinity.pod_anti_affinity]
        [[...required_during_scheduling_ignored_during_execution]]
          topology_key = "kubernetes.io/hostname"
          [...label_selector.match_expressions]
            key = "project.runner.gitlab.com/id"
            operator = "Exists"

project.runner.gitlab.com/id is a label the runner puts on every job Pod and that nothing else carries, so the Pods repel each other and nothing else. Demand counts Pods that are Pending, Running or Unknown, so five live job Pods are a target of five at one Pod per node, and zero once they are gone.

One thing about the harness that costs a run to learn: the scale decision is DRIVEN, not running. fourstepStack.pass calls prov.Reconcile and driver.RunPass explicitly, so a test that stops asking never sees a decision and waits for a teardown no one requested -- which is why the other live tests call stack.choose(t, ctx, scale.Teardown). The first version of this test waited for the cluster to fall to zero on its own and sat at five nodes until the deadline.

TestFiveGitLabJobPodsScaleToFiveNodesAndBackToZero PASS, 346.11 s, on 2026-10-06 against the real GitLab, pipeline 69, nodes of 2 vCPU and 3072 MiB, five scale-check-N jobs:

STEP 3  the cluster is at zero nodes before anything is applied: ""
STEP 5  5 job pods exist in gitlab-runner
STEP 6  the scale decision read Provision with nodes=5: intent and no node
STEP 7  5 nodes joined on demand; `kubectl get nodes`: liveweb-node1 .. liveweb-node5, all Ready
STEP 8  each job pod is on its own node:
        liveweb-node1 <- ...-concurrent-0    liveweb-node4 <- ...-concurrent-3
        liveweb-node2 <- ...-concurrent-1    liveweb-node5 <- ...-concurrent-4
        liveweb-node3 <- ...-concurrent-2
STEP 9  all 5 scale-check jobs finished success
STEP 10 the job pods are gone (0 left), so nothing is asking for a node
STEP 11 the scale decision read Teardown with nodes=0:
        nothing running for 1m0.917972328s, past the 1m0s hold
STEP 12 the cluster is back at zero nodes; `kubectl get nodes` said ""
STEP 13 the cluster is at zero nodes and its control plane is still Ready at https://127.0.0.1:62302

That is the whole loop: zero, five nodes because five Pods could not be placed, one job per node, the work finishing, and the cluster falling back to zero on its own hold while the apiserver keeps answering. Nine GitLab jobs ran on this cluster across the three passes it took to get here, and the fifth pass is the one that is a result.

Observed mid-flight, one node at a time: with one node up, three Pods sat Init on it and two stayed Pending -- the anti-affinity refusing to co-schedule -- and the two Pending Pods are what took the count to three, four and five. That is the loop working, not a coincidence of scheduling.

The nodes of a scale check are small on purpose: five of the nodes the test job needs would be 60 vCPU and 120 GiB, which this host does not have. The Pod requests drop to 100m/128Mi for the same reason -- requests are what the scheduler places against, and limits stay generous because limits are not placement.

Two things the first attempt at this got wrong, both worth keeping.

  • A limit is a request when no request is given. helper_cpu_limit = 1 with no helper request made every job Pod ask for about 2.1 CPU -- the build container's own request plus the helper's and the init container's, each pulled up from its limit -- and a two-vCPU node answers that with Unschedulable: "0/1 nodes are available: 1 Insufficient cpu". The reading is a Pod that sits Pending while nodes join around it, which looks like a slow cluster and is actually a Pod that cannot fit on any of it. helper_cpu_request and helper_memory_request exist; setting them says where the Pod is placed without capping what it may burst to.

  • A job Pod resolves through CoreDNS, and CoreDNS is an addon that comes up after the node it runs on. The scale-check jobs cloned like every other job, and the first attempt lost three of five to fatal: unable to access '...': Could not resolve host: gitlab.dorper-matrix.ts.net (Timeout while contacting DNS servers), reported as script_failure -- which reads as a broken job. A Pod's /etc/resolv.conf is k3s's kube-dns, so the query goes Pod -> CoreDNS -> the node's own resolver -> pasta -> the host, and CoreDNS is created when the first node joins and is not serving for the first seconds after. A job that starts the moment its node joins is a job that queries a resolver that is not there yet.

    pasta is not the fault, and it was the first thing suspected. Querying its forwarder at the gateway address from inside its own namespace answers gitlab.dorper-matrix.ts.net, github.com and ghcr.io in about ten milliseconds each. The host's nameserver 127.0.0.53 is systemd-resolved's stub and pasta reaches it fine, tailscale MagicDNS included. The fault is one hop further in.

    The fix is pre_get_sources_script, which runs BEFORE the clone; pre_build_script runs after it, which is why the pnpm step could never have covered this. The gate waits for the job's own nameserver to answer for CI_SERVER_HOST -- the name the clone is about to need -- and gives up quietly rather than failing the job it exists to protect.

    It is not a startup delay, and it does not go away. Measured on a live five-node cluster, from a Pod on each node, TCP to the CoreDNS Pod's own IP:

    liveweb-node1  <- CoreDNS runs here   REACHABLE
liveweb-node2                         UNREACHABLE
liveweb-node3                         UNREACHABLE
liveweb-node4                         UNREACHABLE
liveweb-node5                         UNREACHABLE

    and nslookup inside those four answers connection timed out; no servers could be reached. The reason is not CoreDNS: a guest cannot reach another guest. Each sits behind its own pasta instance, pasta is user-mode networking, and it carries a guest's traffic out to the host and nothing between two guests, so the nodes have no path to each other and k3s's flannel overlay has nothing to run over. Cluster DNS is one hop that happens to need that path.

    So this is a limit of the node network and not a race in the job: anything that resolves a name works on a one-node cluster and on whichever node CoreDNS is on, and fails on the others. It is recorded in NEXT.yaml as a limit, with the pod-to-pod reading above as its instrument. The scale-check jobs therefore carry GIT_STRATEGY: none -- not to route around the failure but because a check of the scale decision must not depend on a network path the cluster does not have -- and the pre_get_sources_script wait stays for the jobs that do clone. Routing around this silently was the first response, by giving those jobs GIT_STRATEGY: none before the cause was known; that is recorded because it is the tempting answer, it hides the failure in the one check whose whole job is to find it, and it leaves every other job exposed.

7. The mkfs measurement, in full

B6 is the reason the job's image is pulled rather than seeded, so it is written down rather than summarized. Every line is mkfs.ext4 -q -F -d <dir> -L <label> <image on an existing sparse file>, on this host, mke2fs 1.47.3, and the reading is the exit status and the message.

Directory populatedFilesystemResult
one 3,500,000,000-byte file of zeros4,900 MBaccepted, file present at that size
the same file, 4 KiB blocks, 256-byte inodes6,553,600,000 bytesaccepted, file present at that size
the 3,336,075,264-byte builder tarball alone6,000 MBExt2 file too big while writing file "127.0.0.1-5000-...tar"
the same tarball alone, 4 KiB blocks, 256-byte inodes6,000 MBsame refusal
the same tarball, renamed to short.tar6,000 MBsame refusal
the whole 3,877 MiB stage directory, as the builder lays it down5,377 MiB and 12,000 MBsame refusal
the first 1,000,000,000 bytes of the same tarball3,000 MBaccepted

So the refusal tracks the tarball and not the filesystem size, the block size, the inode size or the file's name, and it is not a hard 2 GiB limit on files in general -- a larger file of zeros is accepted in a smaller filesystem. What was not pursued is the exact property of that tarball that triggers it; the job does not need seeding, and saying so is cheaper than a wrong explanation.