# 0002: Kubernetes executor for knotless-run

Status: implemented and dogfooded

Depends on: 0001

## Working rules

- Do not open pull requests, merge requests, or send patches to any
  upstream repository, for any project.
- Work only in this repository and in the local forks:
  - `~/src/knotless-fork` for `knotless-run`
  - `~/src/socialweb-computer-kcp-fork` for any scoped-credential or
    tenant-cluster change
- Each fork has `upstream` for the original project and `fork` for our
  own knotless remote. Treat `upstream` as read-only: fetch from it if
  needed, never push to it.
- Push only to our own repositories under
  `johnandersen777.bsky.social`, over `knotless://` remotes, as work
  lands. Never push to `upstream`.

## Goal

Make a Kubernetes Job a first-class `knotless-run` executor, so a
repository can name a socialweb-computer runner and have each workflow
run on a hosted tenant k3s cluster.

The runner daemon stays on a stable host. It keeps `state.json`,
`secrets-key.json`, caches, mirrors and login sessions. The Job pod is
only the run's environment, and its Pending state is the tenant's scale
signal.

Non-goal: run the `knotless-run` daemon inside the tenant cluster. That
would keep a node alive, and it would lose runner state when the node is
torn down.

## Selection model

No scheduler changes are needed.

- The repository record's `runners` array names the runner DIDs.
- A runner with `--discover` watches repositories that name it.
- Every named, watching runner runs every workflow unless the workflow's
  `runners:` field pins one.
- To avoid duplicate checks, name one runner or pin workflows.
- The merge-queue lander is the first named runner that may push, else the
  first named runner. It must run `--queue`.
- `--default-runner` only sets the runner for new repositories.

## M1: executor seam

Status: done

- Add a Kubernetes backend to `crates/knotless-run/src/executor.rs`.
- Prefer a small trait (`start`, `command`, `stop`, `cleanup`) over a
  growing closed enum, but a variant is acceptable for the first change.
- Add options: `--executor kubernetes`, `--kubeconfig`, `--context`,
  `--namespace`, `--image`, `--cpu-request`, `--memory-request`,
  `--service-account`, `--node-selector`.
- Keep docker as the default.
- Reject the Kubernetes backend for `--pulls` and `--queue` until it has
  an isolation story for untrusted code.
- Update `docs/reference/cli.md` and `docs/design/runner.md`.

Acceptance:

- `knotless-run --executor kubernetes` starts, runs one step, writes a
  check, and stops.
- `cargo fmt --check`, clippy and the existing test suite pass.
- The docker path is unchanged.

## M2: checkout, steps and outputs

Status: done

- Move the PoC's tar-over-`kubectl exec -i` into the executor.
- Preserve file modes, symlinks, executable bits and empty directories.
- Stream shared steps into `/steps`, each at `/steps/<n>`, as docker does.
- Stream `/outputs` back before the Job is deleted.
- Use an `emptyDir` for `/work`, `/steps` and `/outputs`; do not use a
  hostPath.
- Bound upload and download by size and by the run timeout.
- Reuse the existing `check` output code so images become check outputs.

Acceptance:

- The existing `test` workflow's checkout works unchanged.
- A 100 MiB checkout and a 10 MiB output each complete.
- A missing or unreadable path fails the run with an `error` check, not a
  stuck Job.

## M3: lifecycle, logs and cancellation

Status: done

- Job: `backoffLimit: 0`, `activeDeadlineSeconds` from the run timeout,
  `ttlSecondsAfterFinished` when not keeping, labels
  `dev.knotless.runner` and `dev.knotless.run`, annotations with the raw
  runner DID and check key.
- Pod: `restartPolicy: Never`, no service account token, no service links.
- Wait for the pod Ready, not just the Job.
- Stream `pods/log --follow` or the step's exec output into the run log.
- On step timeout or cancellation, delete the Job, not only kill the local
  `kubectl`.
- On runner start, delete Jobs labeled with this runner that are not in
  `going`, mirroring the docker executor's `remove_left`.
- Prune old Jobs with the tenant's TTL, and always delete on success,
  failure, timeout and cancellation unless `--keep`.

Acceptance:

- A hanging step is killed at its timeout and its check is `failure` with
  no exit code.
- A cancelled run leaves no Job or pod.
- A runner restart removes its orphaned Jobs and starts clean.
- Logs reach the check's `log` blob up to the 1 MB cap.

## M4: variables and secrets

Status: done

Today docker passes variable values through the client's environment,
never argv, and per-step secrets through `docker exec -e`. Kubernetes
`exec` has no `-e`, and a `Secret` object would put the values in the
tenant API server and let every step read them.

Design: the step payload goes over the exec stream.

- For each step, build a small POSIX shell payload:
  `export NAME='value'` for the run's variables and the step's granted
  secrets, then `exec sh -ec '<step script>'`.
- Shell-quote every value with single quotes. A value containing a single
  quote becomes `'\''`; newlines are valid inside single quotes. Shell
  variables cannot contain NUL.
- Run it as `kubectl exec -i <pod> -- sh -s` and write the payload to
  stdin. Values are never in argv, the Job spec, the pod spec, a
  Kubernetes `Secret`, the environment of the runner's `kubectl` process,
  or a file in the pod.
- No `Secret` object is created. The runner therefore needs no
  `secrets` RBAC on the tenant cluster.
- Keep the no-secrets rule for pull and batch runs exactly as today.
- Mask granted values in logs with the existing longest-first masking.
- After a step that was given secrets, list `/proc` before and after and
  kill the processes the step left, as the docker executor does. A
  background process must not carry a secret into the next step.
- The same stream carries non-secret run variables and `rerun.env` values.

Rejected alternatives:

- A per-run `Secret` with `envFrom`: simple, but values live in the API
  server, every step sees every secret, and the runner gains
  `secrets` permission.
- Values in `kubectl exec -- env NAME=value ...`: visible in the process
  list and easy to log.
- Secret files on the emptyDir: recoverable by later steps and by
  anything that can read the pod filesystem.

Acceptance:

- A push run with a granted secret sees it; a pull or batch run does not.
- No Kubernetes `Secret` is created by a run.
- Secret values do not appear in the Job spec, pod spec, argv, a pod
  file, the check log or the check record.
- A workflow that asks for an ungranted secret logs the name only.
- A secret-bearing step that leaves a background process has it killed
  before the next step.
- A step's environment contains only its own granted values.

## M5: caches

Status: done

A Kubernetes cache is a tar stream between the runner's cache store and
the pod. Before the steps, the run's base is streamed into the pod at the
cache's path. After the steps, and before the Job is deleted, that path is
streamed back into the run's upper layer. The existing promote and keep
rules then run unchanged: a passing push promotes, a trusted pull's pass
may be kept, and an untrusted run's layer is dropped.

- No PV, object store or cache service is needed.
- The first run of a cache is cold.
- `uses_caches` is true for docker and kubernetes; other executors lease
  no cache and create no cache directories.

Acceptance:

- A live test ran a step that wrote `/cache/warm`, then a second run saw
  it and reported `warm`.
- An untrusted run gets an apart base, exactly as under docker.

## M6: trust and isolation

Status: done

- Pull and batch runs get no secrets, no cache keep and an apart cache
  base.
- One Job per node by required hostname anti-affinity, so each run is
  alone in its Firecracker guest.
- No hostPath, no privileged containers, no host network, no service
  account token, no privilege escalation.
- The runner applies a `NetworkPolicy` selecting its runner label and
  denying all ingress, so one run's pod cannot reach another. Egress is
  not restricted.
- Tenant node shape: 2 vCPU, 4 GiB memory, 50 GiB rootfs. Resource
  requests above it leave the pod Pending, which is an `error` check.

Acceptance:

- A pull run cannot read another run's secrets; the live test with the
  scoped credential applied the policy and ran Jobs in `knotless-runs`.
- A malicious workflow cannot reach the runner host or another tenant.
- Requests fit the node, or the run is visibly unschedulable.

## M7: credentials and RBAC

Status: done

Two paths exist:

- Runner on the kcp/gateway host: `internal tenantcredential`, scoped
  client cert, loopback `https://status.url`.
- Runner off host: the gateway's `getKubeconfig` is cluster-admin. Run
  `scripts/scoped-kubeconfig.sh` once with that admin kubeconfig. It makes
  a ServiceAccount, a namespaced Role and a 24-hour token kubeconfig, and
  checks that the token can create Jobs but cannot read nodes.

The Role grants Jobs, pods, pods/log, pods/exec, events and
NetworkPolicies, in one namespace. It grants no secrets. The runner needs
no other cluster permission. Refresh the token before 24 hours by running
the script again.

Acceptance:

- The live test ran with the scoped kubeconfig in `knotless-runs`.
- `auth can-i get nodes` is `no`; `auth can-i create jobs` is `yes`.
- A leaked pod has no service account token to mint another credential.

## M8: queue and runner selection

Status: done

- Keep queue landing on a stable runner with persistent state.
- Allow `--queue` on the Kubernetes backend only after M6.
- Keep exactly one lander for a repository; document the current
  `lander()` behavior where a named runner that may push is chosen even if
  it is not running.
- Pin heavy workflows with `runners:` while dogfooding, and name the
  socialweb runner in `repo.runners`.
- Do not run two runners for the same repository during rollout unless
  duplicate checks are acceptable.

Acceptance:

- `--pulls` and `--queue` are accepted with the Kubernetes executor.
- Pull and batch runs get no secrets and an apart cache base, as under
  docker; one Job per node is the isolation boundary.
- The lander rules are unchanged from the docker executor.

## M9: observability

Status: done

- Check logs stream live through `dev.knotless.putRunOutput` as they do
  under docker.
- Check outputs are images from `/outputs`.
- Add structured log lines for Job start, pod Ready, step start, step end,
  Job delete and orphan cleanup.
- Add the runner's existing telemetry fields for the Job name, namespace
  and image.
- Add a startup sweep count and a cleanup failure count.

Acceptance:

- A failed run's log names the failing step and exit code.
- A missing image or unschedulable pod produces a clear `error` check.
- Orphan cleanup logs what it deleted.

## M10: tests and proof

Status: done

- Unit tests with a fake `kubectl`, as in 0001.
- Manifest tests for labels, resources, TTL, anti-affinity and
  `activeDeadlineSeconds`.
- A live test against `knotless-poc`: one Job, two steps, outputs, delete,
  node scale to zero.
- A live `knotless-run --executor kubernetes --pulls --queue` against a
  disposable repository once M6 lands.
- A failure matrix: bad image, image pull denied, pod unschedulable, step
  nonzero, step timeout, runner killed mid-run, node lost mid-run.
- Keep these out of the normal knotless CI until they have a fake API
  server; do not make CI depend on a live market.

Acceptance:

- The failure matrix leaves no Job, pod, Secret or cache volume behind.
- The live proof records the node count before and after.

## M11: rollout

Status: done

1. Land M1-M3 behind `--executor kubernetes`; docker remains default.
2. Dogfood on this repository, pinned by `runners:`.
3. Run the repository's full workflow set on the tenant cluster.
4. Add docs: `docs/guide/kubernetes-runner.md`, updates to
   `docs/design/runner.md`, `docs/reference/cli.md`.
5. Offer the executor to other repositories; keep docker as fallback.
6. Only then consider `--queue` for untrusted work.

Acceptance:

- Every workflow this repository defines runs on the socialweb runner:
  `check`, `test`, `docs`, `package` and `poc`.
- The tenant returns to zero nodes after the runs.
- Rolling back to docker needs only a config change.

## Open questions

- Whether a tenant administrator can inspect `pods/exec` streams. Assume
  they can; use a runner-owned cluster, and keep secrets out of every
  durable object.
- Whether one Job per node is mandatory for all untrusted work or only
  when the workflow runs untrusted code.
- Cost and latency of a market VM per job versus a warmed node pool.
- Multi-tenant runners: one cluster per runner account, or one cluster
  per trust domain.
- Interaction with `superseded: finish` and runner handover.
- Image pull egress from tenant nodes and registry authentication.

## Implementation evidence

- `crates/knotless-run/src/executor.rs`: `Executor::Kubernetes`,
  `Session::Kubernetes`, Job manifest, kubectl client, tar transfer,
  stdin step payload, `/proc` cleanup, orphaned-Job cleanup.
- `crates/knotless-run/src/lib.rs`: `--executor kubernetes`, kubeconfig,
  context, namespace, kubectl, service account and node selector flags;
  pull and queue validation.
- `crates/knotless-run/tests/kubernetes_live.rs`: ignored live test that
  runs a Job on a tenant cluster, checks the output file, deletes the Job
  and checks orphan cleanup.
- `crates/knotless-run/src/executor.rs` tests: manifest, label and payload
  shape, a full fake-kubectl run, secret-bearing step cleanup, orphan
  deletion.
- Docs: `docs/guide/kubernetes-runner.md`, `docs/design/runner.md`,
  `docs/reference/cli.md`; change file `.changeset/kubernetes-executor.md`.
- Fork commits: `b1e831a`, `85abaa3`, `afaf890`, `84e1733` and
  `5762212` in `knotless://johnandersen777.bsky.social/knotless`.
- `scripts/scoped-kubeconfig.sh` in this repository: a ServiceAccount,
  namespaced Role and 24-hour token kubeconfig, checked to allow Jobs and
  deny nodes.

## Verification run

`cargo test -p knotless-run --lib`: 56 passed.

`cargo clippy -p knotless-run --all-targets -- -D warnings`: clean.

Live executor test, on the scoped kubeconfig in `knotless-runs`:

```bash
KNOTLESS_K8S_LIVE=1 KNOTLESS_K8S_NAMESPACE=knotless-runs \
  KUBECONFIG=$HOME/.tmp/knotless-runner-k8s/tenant/knotless-runner.kubeconfig.yaml \
  cargo test -p knotless-run --test kubernetes_live -- --ignored --nocapture
```

Passed in 81.65 s. It ran two Jobs: the first wrote `/cache/warm` and
reported `cold`; the second saw the cache and reported `warm`. It applied
the NetworkPolicy, deleted both Jobs, and the orphan sweep ran.

## Dogfood proof

The repository `knotless://johnandersen777.bsky.social/knotless-runner-k8s`
names the alice runner. Its full workflow set ran at commit
`184f1b1c90b67720a51708e1fa61cff45c54f4da` through the new executor, with
the scoped kubeconfig and the default-deny NetworkPolicy:

| Workflow | Status | Step | Exit | Time |
| --- | --- | --- | --- | --- |
| `check` | success | shell | 0 | 366 ms |
| `test` | success | cargo | 0 | 1382 ms |
| `docs` | success | files | 0 | 419 ms |
| `package` | success | archive | 0 | 465 ms |
| `poc` | success | dogfood | 0 | 436 ms |

The Jobs were deleted, and the tenant returned to zero nodes. The `test`
workflow used `cache: [target]`, so its second run started from the cache
streamed back by the first.

Earlier, the one-step `poc` workflow proved the same path at
`c10c79921c3f3f670706310575d24da2a3727916`. The first attempt failed
because `KNOTLESS_RUN_OUTPUTS` was empty; `85abaa3` carries the run
variables into the Kubernetes step payload. A later concurrent start hit
`AlreadyExists` on the NetworkPolicy; `84e1733` treats an existing policy
as success.

The tenant cluster `knotless-poc` was deleted after the proof.