# Plan: tenant k3s Job executor for knotless-run

## Goal

Prove the execution mechanics for a `knotless-run` executor that runs one
workflow run as one Kubernetes Job on a socialweb-computer tenant k3s
cluster.

The runner daemon stays outside the cluster. The Job pod is the unit that
raises the tenant's scale demand, so a Pending Job causes a NodeClaim,
the market RFP rents a VM, the pod is scheduled, and the cluster returns
to zero after the run.

## Scope of this proof of concept

- One Job per run, one long-lived pod, one `kubectl exec` per step.
- The checkout is streamed into `/work` with tar over `kubectl exec -i`.
- Shared steps are streamed into `/steps`.
- Outputs are streamed back from `/outputs` and become images in the
  check.
- A run writes a JSON result whose step fields mirror `dev.knotless.check`.
- `--dry-run` prints the Job and every call.
- Tests use a fake `kubectl`; no cluster is needed for `cargo test`.
- A live script obtains a tenant kubeconfig through the hosted gateway
  with the alice credential file and `oauth-signin.ts`.

Out of scope for this proof:

- Patching `knotless-run` itself.
- Real workflow YAML parsing.
- Caches, PVCs, and artifact stores.
- Secret injection.
- Cancellation over XRPC.
- Queue landing, which stays on a stable runner.

## Mapping to knotless's executor seam

`crates/knotless-run/src/executor.rs` has:

- `Executor::{Shell, Docker}` and `Session::{Shell, Docker}`.
- `Executor::start(setup, log, timeout)` starts the run's container.
- `Session::command(script, own)` returns the step command.
- `Session::stop()` removes the container.
- `run(cmd, log, timeout, cancelled)` enforces timeout and cancellation.

This proof is the same shape:

| knotless | This proof |
|---|---|
| `Setup.dir` | `--workdir`, streamed to `/work` |
| `Setup.steps` | `--steps-dir`, streamed to `/steps` |
| `Setup.outputs` | `--outputs`, streamed from `/outputs` |
| `Setup.image` | `--image` |
| `Setup.vars` | not yet; pod env or ephemeral Secret later |
| `Executor::start` | `kubectl apply`, `kubectl wait` |
| `Session::command` | `kubectl exec <pod> -- sh -ec <script>` |
| `executor::run` | spawn, poll, on timeout kill and delete the Job |
| `Session::stop` | `kubectl delete job` |
| `CheckStep` | `StepResult` |
| `CheckRecord.log` | `--log` |
| check outputs | `--outputs` |

The real change is a closed-enum variant and a new `Session::Kubernetes`,
or a small trait with `start`, `command`, `stop`, `cleanup`.

## What the proof must show

1. A Job's pod becomes Ready and accepts exec through the tenant
   client-certificate front.
2. A checkout and shared steps stream into the pod.
3. Steps share one environment, so a tool installed by one step is there
   for the next.
4. Logs stream into one run log.
5. Outputs stream back.
6. The Job is deleted, so the node scales to zero.
7. A result JSON has the step statuses and exit codes a check needs.

## Live proof

```bash
scripts/tenant-kubeconfig.sh
export KUBECONFIG=$HOME/.tmp/knotless-runner-k8s/tenant/knotless-poc.kubeconfig.yaml
cargo run --quiet -- run \
  --name knotless-poc \
  --runner did:plc:runner \
  --image busybox:1.36 \
  --workdir examples \
  --steps-file examples/steps.tsv \
  --outputs "$HOME/.tmp/knotless-runner-k8s/outputs" \
  --result "$HOME/.tmp/knotless-runner-k8s/result.json" \
  --log "$HOME/.tmp/knotless-runner-k8s/run.log"
```

Then check the tenant returned to zero nodes:

```bash
kubectl get nodes
```

## Next changes after this proof

1. Add `Executor::Kubernetes` to `knotless-run`, behind the same CLI
   validation that rejects the shell executor for pulls and queue.
2. Inject per-step variables through an ephemeral Secret and pod env.
3. Add a cold-cache strategy; no PV or object store exists in the design.
4. Decide one job per node for untrusted runs and set required hostname
   anti-affinity.
5. Add startup cleanup by `dev.knotless.runner` and `dev.knotless.run`
   labels, mirroring the docker executor's `remove_left`.