0002: Kubernetes executor for knotless-run

Status: implemented and dogfooded

Depends on: 0001

Working rules

• Do not open pull requests, merge requests, or send patches to any upstream repository, for any project.
• Work only in this repository and in the local forks:
  • ~/src/knotless-fork for knotless-run
  • ~/src/socialweb-computer-kcp-fork for any scoped-credential or tenant-cluster change
• Each fork has upstream for the original project and fork for our own knotless remote. Treat upstream as read-only: fetch from it if needed, never push to it.
• Push only to our own repositories under johnandersen777.bsky.social, over knotless:// remotes, as work lands. Never push to upstream.

Goal

Make a Kubernetes Job a first-class knotless-run executor, so a repository can name a socialweb-computer runner and have each workflow run on a hosted tenant k3s cluster.

The runner daemon stays on a stable host. It keeps state.json, secrets-key.json, caches, mirrors and login sessions. The Job pod is only the run's environment, and its Pending state is the tenant's scale signal.

Non-goal: run the knotless-run daemon inside the tenant cluster. That would keep a node alive, and it would lose runner state when the node is torn down.

Selection model

No scheduler changes are needed.

• The repository record's runners array names the runner DIDs.
• A runner with --discover watches repositories that name it.
• Every named, watching runner runs every workflow unless the workflow's runners: field pins one.
• To avoid duplicate checks, name one runner or pin workflows.
• The merge-queue lander is the first named runner that may push, else the first named runner. It must run --queue.
• --default-runner only sets the runner for new repositories.

M1: executor seam

Status: done

• Add a Kubernetes backend to crates/knotless-run/src/executor.rs.
• Prefer a small trait (start, command, stop, cleanup) over a growing closed enum, but a variant is acceptable for the first change.
• Add options: --executor kubernetes, --kubeconfig, --context, --namespace, --image, --cpu-request, --memory-request, --service-account, --node-selector.
• Keep docker as the default.
• Reject the Kubernetes backend for --pulls and --queue until it has an isolation story for untrusted code.
• Update docs/reference/cli.md and docs/design/runner.md.

Acceptance:

• knotless-run --executor kubernetes starts, runs one step, writes a check, and stops.
• cargo fmt --check, clippy and the existing test suite pass.
• The docker path is unchanged.

M2: checkout, steps and outputs

Status: done

• Move the PoC's tar-over-kubectl exec -i into the executor.
• Preserve file modes, symlinks, executable bits and empty directories.
• Stream shared steps into /steps, each at /steps/<n>, as docker does.
• Stream /outputs back before the Job is deleted.
• Use an emptyDir for /work, /steps and /outputs; do not use a hostPath.
• Bound upload and download by size and by the run timeout.
• Reuse the existing check output code so images become check outputs.

Acceptance:

• The existing test workflow's checkout works unchanged.
• A 100 MiB checkout and a 10 MiB output each complete.
• A missing or unreadable path fails the run with an error check, not a stuck Job.

M3: lifecycle, logs and cancellation

Status: done

• Job: backoffLimit: 0, activeDeadlineSeconds from the run timeout, ttlSecondsAfterFinished when not keeping, labels dev.knotless.runner and dev.knotless.run, annotations with the raw runner DID and check key.
• Pod: restartPolicy: Never, no service account token, no service links.
• Wait for the pod Ready, not just the Job.
• Stream pods/log --follow or the step's exec output into the run log.
• On step timeout or cancellation, delete the Job, not only kill the local kubectl.
• On runner start, delete Jobs labeled with this runner that are not in going, mirroring the docker executor's remove_left.
• Prune old Jobs with the tenant's TTL, and always delete on success, failure, timeout and cancellation unless --keep.

Acceptance:

• A hanging step is killed at its timeout and its check is failure with no exit code.
• A cancelled run leaves no Job or pod.
• A runner restart removes its orphaned Jobs and starts clean.
• Logs reach the check's log blob up to the 1 MB cap.

M4: variables and secrets

Status: done

Today docker passes variable values through the client's environment, never argv, and per-step secrets through docker exec -e. Kubernetes exec has no -e, and a Secret object would put the values in the tenant API server and let every step read them.

Design: the step payload goes over the exec stream.

• For each step, build a small POSIX shell payload: export NAME='value' for the run's variables and the step's granted secrets, then exec sh -ec '<step script>'.
• Shell-quote every value with single quotes. A value containing a single quote becomes '\''; newlines are valid inside single quotes. Shell variables cannot contain NUL.
• Run it as kubectl exec -i <pod> -- sh -s and write the payload to stdin. Values are never in argv, the Job spec, the pod spec, a Kubernetes Secret, the environment of the runner's kubectl process, or a file in the pod.
• No Secret object is created. The runner therefore needs no secrets RBAC on the tenant cluster.
• Keep the no-secrets rule for pull and batch runs exactly as today.
• Mask granted values in logs with the existing longest-first masking.
• After a step that was given secrets, list /proc before and after and kill the processes the step left, as the docker executor does. A background process must not carry a secret into the next step.
• The same stream carries non-secret run variables and rerun.env values.

Rejected alternatives:

• A per-run Secret with envFrom: simple, but values live in the API server, every step sees every secret, and the runner gains secrets permission.
• Values in kubectl exec -- env NAME=value ...: visible in the process list and easy to log.
• Secret files on the emptyDir: recoverable by later steps and by anything that can read the pod filesystem.

Acceptance:

• A push run with a granted secret sees it; a pull or batch run does not.
• No Kubernetes Secret is created by a run.
• Secret values do not appear in the Job spec, pod spec, argv, a pod file, the check log or the check record.
• A workflow that asks for an ungranted secret logs the name only.
• A secret-bearing step that leaves a background process has it killed before the next step.
• A step's environment contains only its own granted values.

M5: caches

Status: done

A Kubernetes cache is a tar stream between the runner's cache store and the pod. Before the steps, the run's base is streamed into the pod at the cache's path. After the steps, and before the Job is deleted, that path is streamed back into the run's upper layer. The existing promote and keep rules then run unchanged: a passing push promotes, a trusted pull's pass may be kept, and an untrusted run's layer is dropped.

• No PV, object store or cache service is needed.
• The first run of a cache is cold.
• uses_caches is true for docker and kubernetes; other executors lease no cache and create no cache directories.

Acceptance:

• A live test ran a step that wrote /cache/warm, then a second run saw it and reported warm.
• An untrusted run gets an apart base, exactly as under docker.

M6: trust and isolation

Status: done

• Pull and batch runs get no secrets, no cache keep and an apart cache base.
• One Job per node by required hostname anti-affinity, so each run is alone in its Firecracker guest.
• No hostPath, no privileged containers, no host network, no service account token, no privilege escalation.
• The runner applies a NetworkPolicy selecting its runner label and denying all ingress, so one run's pod cannot reach another. Egress is not restricted.
• Tenant node shape: 2 vCPU, 4 GiB memory, 50 GiB rootfs. Resource requests above it leave the pod Pending, which is an error check.

Acceptance:

• A pull run cannot read another run's secrets; the live test with the scoped credential applied the policy and ran Jobs in knotless-runs.
• A malicious workflow cannot reach the runner host or another tenant.
• Requests fit the node, or the run is visibly unschedulable.

M7: credentials and RBAC

Status: done

Two paths exist:

• Runner on the kcp/gateway host: internal tenantcredential, scoped client cert, loopback https://status.url.
• Runner off host: the gateway's getKubeconfig is cluster-admin. Run scripts/scoped-kubeconfig.sh once with that admin kubeconfig. It makes a ServiceAccount, a namespaced Role and a 24-hour token kubeconfig, and checks that the token can create Jobs but cannot read nodes.

The Role grants Jobs, pods, pods/log, pods/exec, events and NetworkPolicies, in one namespace. It grants no secrets. The runner needs no other cluster permission. Refresh the token before 24 hours by running the script again.

Acceptance:

• The live test ran with the scoped kubeconfig in knotless-runs.
• auth can-i get nodes is no; auth can-i create jobs is yes.
• A leaked pod has no service account token to mint another credential.

M8: queue and runner selection

Status: done

• Keep queue landing on a stable runner with persistent state.
• Allow --queue on the Kubernetes backend only after M6.
• Keep exactly one lander for a repository; document the current lander() behavior where a named runner that may push is chosen even if it is not running.
• Pin heavy workflows with runners: while dogfooding, and name the socialweb runner in repo.runners.
• Do not run two runners for the same repository during rollout unless duplicate checks are acceptable.

Acceptance:

• --pulls and --queue are accepted with the Kubernetes executor.
• Pull and batch runs get no secrets and an apart cache base, as under docker; one Job per node is the isolation boundary.
• The lander rules are unchanged from the docker executor.

M9: observability

Status: done

• Check logs stream live through dev.knotless.putRunOutput as they do under docker.
• Check outputs are images from /outputs.
• Add structured log lines for Job start, pod Ready, step start, step end, Job delete and orphan cleanup.
• Add the runner's existing telemetry fields for the Job name, namespace and image.
• Add a startup sweep count and a cleanup failure count.

Acceptance:

• A failed run's log names the failing step and exit code.
• A missing image or unschedulable pod produces a clear error check.
• Orphan cleanup logs what it deleted.

M10: tests and proof

Status: done

• Unit tests with a fake kubectl, as in 0001.
• Manifest tests for labels, resources, TTL, anti-affinity and activeDeadlineSeconds.
• A live test against knotless-poc: one Job, two steps, outputs, delete, node scale to zero.
• A live knotless-run --executor kubernetes --pulls --queue against a disposable repository once M6 lands.
• A failure matrix: bad image, image pull denied, pod unschedulable, step nonzero, step timeout, runner killed mid-run, node lost mid-run.
• Keep these out of the normal knotless CI until they have a fake API server; do not make CI depend on a live market.

Acceptance:

• The failure matrix leaves no Job, pod, Secret or cache volume behind.
• The live proof records the node count before and after.

M11: rollout

Status: done

1. Land M1-M3 behind --executor kubernetes; docker remains default.
2. Dogfood on this repository, pinned by runners:.
3. Run the repository's full workflow set on the tenant cluster.
4. Add docs: docs/guide/kubernetes-runner.md, updates to docs/design/runner.md, docs/reference/cli.md.
5. Offer the executor to other repositories; keep docker as fallback.
6. Only then consider --queue for untrusted work.

Acceptance:

• Every workflow this repository defines runs on the socialweb runner: check, test, docs, package and poc.
• The tenant returns to zero nodes after the runs.
• Rolling back to docker needs only a config change.

Open questions

• Whether a tenant administrator can inspect pods/exec streams. Assume they can; use a runner-owned cluster, and keep secrets out of every durable object.
• Whether one Job per node is mandatory for all untrusted work or only when the workflow runs untrusted code.
• Cost and latency of a market VM per job versus a warmed node pool.
• Multi-tenant runners: one cluster per runner account, or one cluster per trust domain.
• Interaction with superseded: finish and runner handover.
• Image pull egress from tenant nodes and registry authentication.

Implementation evidence

• crates/knotless-run/src/executor.rs: Executor::Kubernetes, Session::Kubernetes, Job manifest, kubectl client, tar transfer, stdin step payload, /proc cleanup, orphaned-Job cleanup.
• crates/knotless-run/src/lib.rs: --executor kubernetes, kubeconfig, context, namespace, kubectl, service account and node selector flags; pull and queue validation.
• crates/knotless-run/tests/kubernetes_live.rs: ignored live test that runs a Job on a tenant cluster, checks the output file, deletes the Job and checks orphan cleanup.
• crates/knotless-run/src/executor.rs tests: manifest, label and payload shape, a full fake-kubectl run, secret-bearing step cleanup, orphan deletion.
• Docs: docs/guide/kubernetes-runner.md, docs/design/runner.md, docs/reference/cli.md; change file .changeset/kubernetes-executor.md.
• Fork commits: b1e831a, 85abaa3, afaf890, 84e1733 and 5762212 in knotless://johnandersen777.bsky.social/knotless.
• scripts/scoped-kubeconfig.sh in this repository: a ServiceAccount, namespaced Role and 24-hour token kubeconfig, checked to allow Jobs and deny nodes.

Verification run

cargo test -p knotless-run --lib: 56 passed.

cargo clippy -p knotless-run --all-targets -- -D warnings: clean.

Live executor test, on the scoped kubeconfig in knotless-runs:

KNOTLESS_K8S_LIVE=1 KNOTLESS_K8S_NAMESPACE=knotless-runs \
  KUBECONFIG=$HOME/.tmp/knotless-runner-k8s/tenant/knotless-runner.kubeconfig.yaml \
  cargo test -p knotless-run --test kubernetes_live -- --ignored --nocapture

Passed in 81.65 s. It ran two Jobs: the first wrote /cache/warm and reported cold; the second saw the cache and reported warm. It applied the NetworkPolicy, deleted both Jobs, and the orphan sweep ran.

Dogfood proof

The repository knotless://johnandersen777.bsky.social/knotless-runner-k8s names the alice runner. Its full workflow set ran at commit 184f1b1c90b67720a51708e1fa61cff45c54f4da through the new executor, with the scoped kubeconfig and the default-deny NetworkPolicy:

Workflow	Status	Step	Exit	Time
check	success	shell	0	366 ms
test	success	cargo	0	1382 ms
docs	success	files	0	419 ms
package	success	archive	0	465 ms
poc	success	dogfood	0	436 ms

The Jobs were deleted, and the tenant returned to zero nodes. The test workflow used cache: [target], so its second run started from the cache streamed back by the first.

Earlier, the one-step poc workflow proved the same path at c10c79921c3f3f670706310575d24da2a3727916. The first attempt failed because KNOTLESS_RUN_OUTPUTS was empty; 85abaa3 carries the run variables into the Kubernetes step payload. A later concurrent start hit AlreadyExists on the NetworkPolicy; 84e1733 treats an existing policy as success.

The tenant cluster knotless-poc was deleted after the proof.