johnandersen777.bsky.social/knotless-runner-k8s

Switch repository

Loading…

# 0001: Hosted tenant k3s Job executor live proof

Status: done

Date: 2026-11-04

## Goal

Prove that a `knotless-run`-shaped workflow can execute as one Kubernetes
Job on a socialweb-computer hosted tenant k3s cluster, with the runner
daemon outside the cluster and the Job pod driving the tenant node
autoscaler.

## What was built

A standalone Rust crate and binary in this repository:

- `knotless-runner-k8s run`: one `batch/v1 Job` per run.
- One long-lived pod, one `kubectl exec` per step, so tools installed by
  an earlier step are present for later steps.
- Checkout streamed into `/work` with tar over `kubectl exec -i`.
- Shared steps streamed into `/steps`.
- `/outputs` streamed back to the host.
- One run log with step output, exit codes, durations and timeout state.
- Result JSON whose `steps` entries mirror `dev.knotless.check`.
- `--dry-run` prints the Job manifest and every `kubectl` call.
- Unit tests and a fake-`kubectl` end-to-end test; no cluster needed for
  `cargo test`.

Helper scripts:

- `scripts/tenant-kubeconfig.sh`: drives
  `socialweb-computer-kcp/scripts/oauth-signin.ts` with the alice
  credential file, creates a hosted tenant cluster, waits for `Ready`,
  and writes its kubeconfig mode 0600 under `~/.tmp`.
- `scripts/live-proof.sh`: obtains the kubeconfig and runs the proof.

`PLAN.md` maps the proof to `crates/knotless-run/src/executor.rs`.

## Live proof

- Gateway: `https://k8s.socialweb.computer`
- Cluster: `knotless-poc`
- Namespace: `default`
- Image: `busybox:1.36`
- Steps: `date`, `hostname`, `files`, `outputs`
- Result: all four steps success, exit code 0
- `/work/steps.tsv` was present after the upload
- `/outputs/ran.txt` was copied back
- Job deleted after the run
- A node appeared for the Job, then the tenant returned to zero nodes

The result, log and output paths are recorded in `LIVE.md`; no credential
value, session cookie or kubeconfig is committed.

## What this establishes

- A `getKubeconfig` client certificate from the hosted gateway supports
  `kubectl apply`, `wait`, `get`, `exec` and `delete` against a tenant
  apiserver.
- A Pending Job pod raises tenant demand and causes a NodeClaim, so the
  market rents a VM and joins it to k3s.
- Tar over `kubectl exec -i` is enough to move a checkout in and outputs
  out; Kubernetes has no file copy API of its own.
- A single pod per run keeps step state, matching the docker executor's
  one-container-per-run behavior.
- The result shape has the fields a check needs.

## Not established

- No change to `knotless-run`; this is a standalone proof.
- No workflow YAML parsing; steps are `name<TAB>script`.
- No secrets or run variables.
- No caches, PVCs or artifact store.
- No cancellation over XRPC; timeout kills the local `kubectl` client and
  Job deletion stops the remote process.
- No scoped off-host credential; `getKubeconfig` is cluster-admin by
  construction.
- The tenant cluster from this proof still exists with zero nodes.

## Files

- `src/executor.rs` equivalent: this crate's `src/runner.rs`,
  `src/kubectl.rs`, `src/manifest.rs`.
- `src/steps.rs`: proof-only steps format.
- `tests/fake_kubectl.rs`: command sequence without a cluster.
- `scripts/tenant-kubeconfig.sh`, `scripts/live-proof.sh`.
- `LIVE.md`, `PLAN.md`, `README.md`.

## Next

Continue in `0002`.