Plan: tenant k3s Job executor for knotless-run

Goal

Prove the execution mechanics for a knotless-run executor that runs one workflow run as one Kubernetes Job on a socialweb-computer tenant k3s cluster.

The runner daemon stays outside the cluster. The Job pod is the unit that raises the tenant's scale demand, so a Pending Job causes a NodeClaim, the market RFP rents a VM, the pod is scheduled, and the cluster returns to zero after the run.

Scope of this proof of concept

One Job per run, one long-lived pod, one kubectl exec per step.
The checkout is streamed into /work with tar over kubectl exec -i.
Shared steps are streamed into /steps.
Outputs are streamed back from /outputs and become images in the check.
A run writes a JSON result whose step fields mirror dev.knotless.check.
--dry-run prints the Job and every call.
Tests use a fake kubectl; no cluster is needed for cargo test.
A live script obtains a tenant kubeconfig through the hosted gateway with the alice credential file and oauth-signin.ts.

Out of scope for this proof:

Patching knotless-run itself.
Real workflow YAML parsing.
Caches, PVCs, and artifact stores.
Secret injection.
Cancellation over XRPC.
Queue landing, which stays on a stable runner.

Mapping to knotless's executor seam

crates/knotless-run/src/executor.rs has:

Executor::{Shell, Docker} and Session::{Shell, Docker}.
Executor::start(setup, log, timeout) starts the run's container.
Session::command(script, own) returns the step command.
Session::stop() removes the container.
run(cmd, log, timeout, cancelled) enforces timeout and cancellation.

This proof is the same shape:

knotless	This proof
`Setup.dir`	`--workdir`, streamed to `/work`
`Setup.steps`	`--steps-dir`, streamed to `/steps`
`Setup.outputs`	`--outputs`, streamed from `/outputs`
`Setup.image`	`--image`
`Setup.vars`	not yet; pod env or ephemeral Secret later
`Executor::start`	`kubectl apply`, `kubectl wait`
`Session::command`	`kubectl exec <pod> -- sh -ec <script>`
`executor::run`	spawn, poll, on timeout kill and delete the Job
`Session::stop`	`kubectl delete job`
`CheckStep`	`StepResult`
`CheckRecord.log`	`--log`
check outputs	`--outputs`

The real change is a closed-enum variant and a new Session::Kubernetes, or a small trait with start, command, stop, cleanup.

What the proof must show

A Job's pod becomes Ready and accepts exec through the tenant client-certificate front.
A checkout and shared steps stream into the pod.
Steps share one environment, so a tool installed by one step is there for the next.
Logs stream into one run log.
Outputs stream back.
The Job is deleted, so the node scales to zero.
A result JSON has the step statuses and exit codes a check needs.

Live proof

scripts/tenant-kubeconfig.sh
export KUBECONFIG=$HOME/.tmp/knotless-runner-k8s/tenant/knotless-poc.kubeconfig.yaml
cargo run --quiet -- run \
  --name knotless-poc \
  --runner did:plc:runner \
  --image busybox:1.36 \
  --workdir examples \
  --steps-file examples/steps.tsv \
  --outputs "$HOME/.tmp/knotless-runner-k8s/outputs" \
  --result "$HOME/.tmp/knotless-runner-k8s/result.json" \
  --log "$HOME/.tmp/knotless-runner-k8s/run.log"

Then check the tenant returned to zero nodes:

kubectl get nodes

Next changes after this proof

Add Executor::Kubernetes to knotless-run, behind the same CLI validation that rejects the shell executor for pulls and queue.
Inject per-step variables through an ephemeral Secret and pod env.
Add a cold-cache strategy; no PV or object store exists in the design.
Decide one job per node for untrusted runs and set required hostname anti-affinity.
Add startup cleanup by dev.knotless.runner and dev.knotless.run labels, mirroring the docker executor's remove_left.

johnandersen777.bsky.social/knotless-runner-k8s Switch repository Loading…