0001: Hosted tenant k3s Job executor live proof

Status: done

Date: 2026-11-04

Goal

Prove that a knotless-run-shaped workflow can execute as one Kubernetes Job on a socialweb-computer hosted tenant k3s cluster, with the runner daemon outside the cluster and the Job pod driving the tenant node autoscaler.

What was built

A standalone Rust crate and binary in this repository:

• knotless-runner-k8s run: one batch/v1 Job per run.
• One long-lived pod, one kubectl exec per step, so tools installed by an earlier step are present for later steps.
• Checkout streamed into /work with tar over kubectl exec -i.
• Shared steps streamed into /steps.
• /outputs streamed back to the host.
• One run log with step output, exit codes, durations and timeout state.
• Result JSON whose steps entries mirror dev.knotless.check.
• --dry-run prints the Job manifest and every kubectl call.
• Unit tests and a fake-kubectl end-to-end test; no cluster needed for cargo test.

Helper scripts:

• scripts/tenant-kubeconfig.sh: drives socialweb-computer-kcp/scripts/oauth-signin.ts with the alice credential file, creates a hosted tenant cluster, waits for Ready, and writes its kubeconfig mode 0600 under ~/.tmp.
• scripts/live-proof.sh: obtains the kubeconfig and runs the proof.

PLAN.md maps the proof to crates/knotless-run/src/executor.rs.

Live proof

• Gateway: https://k8s.socialweb.computer
• Cluster: knotless-poc
• Namespace: default
• Image: busybox:1.36
• Steps: date, hostname, files, outputs
• Result: all four steps success, exit code 0
• /work/steps.tsv was present after the upload
• /outputs/ran.txt was copied back
• Job deleted after the run
• A node appeared for the Job, then the tenant returned to zero nodes

The result, log and output paths are recorded in LIVE.md; no credential value, session cookie or kubeconfig is committed.

What this establishes

• A getKubeconfig client certificate from the hosted gateway supports kubectl apply, wait, get, exec and delete against a tenant apiserver.
• A Pending Job pod raises tenant demand and causes a NodeClaim, so the market rents a VM and joins it to k3s.
• Tar over kubectl exec -i is enough to move a checkout in and outputs out; Kubernetes has no file copy API of its own.
• A single pod per run keeps step state, matching the docker executor's one-container-per-run behavior.
• The result shape has the fields a check needs.

Not established

• No change to knotless-run; this is a standalone proof.
• No workflow YAML parsing; steps are name<TAB>script.
• No secrets or run variables.
• No caches, PVCs or artifact store.
• No cancellation over XRPC; timeout kills the local kubectl client and Job deletion stops the remote process.
• No scoped off-host credential; getKubeconfig is cluster-admin by construction.
• The tenant cluster from this proof still exists with zero nodes.

Files

• src/executor.rs equivalent: this crate's src/runner.rs, src/kubectl.rs, src/manifest.rs.
• src/steps.rs: proof-only steps format.
• tests/fake_kubectl.rs: command sequence without a cluster.
• scripts/tenant-kubeconfig.sh, scripts/live-proof.sh.
• LIVE.md, PLAN.md, README.md.

Next

Continue in 0002.