# knotless-runner-k8s

Proof of concept for a Kubernetes Job executor for `knotless-run`.

`knotless-run` normally runs one docker container per workflow run. This
crate proves the same shape on a socialweb-computer tenant k3s cluster:
one Job per run, one pod that stays alive, one `kubectl exec` per step.
The runner daemon stays outside the cluster; the Job pod is what makes
the tenant scale a node from zero.

Status: proof of concept. It does not patch `knotless-run` yet. `PLAN.md`
maps this code to `crates/knotless-run/src/executor.rs`.

## Working rules

- Do not open pull requests, merge requests, or send patches to any
  upstream repository. All work stays here and in our own forks.
- Target repos are local forks, not the original checkouts:
  `~/src/knotless-fork` and `~/src/socialweb-computer-kcp-fork`.
  Their `upstream` remotes are read-only; push to `fork` as work lands,
  over `knotless://`.
- Never print or commit credential values. `scripts/oauth-signin.ts`
  reads the credential file itself.

See `AGENTS.md` for the full rules.

## Build and test

```bash
cargo test
cargo run -- --help
```

Tests use a fake `kubectl`, so no cluster is needed.

## Dry run

```bash
cargo run --quiet -- run \
  --name demo \
  --image busybox:1.36 \
  --workdir examples \
  --steps-file examples/steps.tsv \
  --outputs "$HOME/.tmp/knotless-runner-k8s/outputs" \
  --dry-run
```

This prints the Job manifest and every `kubectl` call without touching a
cluster.

## Live run on a hosted tenant cluster

The gateway is API-only, so sign-in is driven by
`socialweb-computer-kcp/scripts/oauth-signin.ts`. The script below reads
`~/.aliceoa_bsky_social_source` through that script, never itself, and
writes the session cookie and kubeconfig 0600 under
`~/.tmp/knotless-runner-k8s/tenant`.

```bash
scripts/tenant-kubeconfig.sh
export KUBECONFIG=$HOME/.tmp/knotless-runner-k8s/tenant/knotless-poc.kubeconfig.yaml
kubectl get nodes
```

Then run a workflow:

```bash
cargo run --quiet -- run \
  --name knotless-poc \
  --runner did:plc:runner \
  --image busybox:1.36 \
  --workdir examples \
  --steps-file examples/steps.tsv \
  --outputs "$HOME/.tmp/knotless-runner-k8s/outputs" \
  --result "$HOME/.tmp/knotless-runner-k8s/result.json" \
  --log "$HOME/.tmp/knotless-runner-k8s/run.log"
```

The Job is deleted unless `--keep` is given. A deleted Job lets the
tenant return to zero nodes after its scale hold.

### Scoped kubeconfig

`scripts/tenant-kubeconfig.sh` returns the gateway's admin kubeconfig.
Make a runner-scoped one from it:

```bash
ADMIN_KUBECONFIG=$HOME/.tmp/knotless-runner-k8s/tenant/knotless-poc.kubeconfig.yaml \
  scripts/scoped-kubeconfig.sh
```

That creates a ServiceAccount, a namespaced Role for Jobs, pods, exec,
logs, events and NetworkPolicies, and a 24-hour token kubeconfig at
`~/.tmp/knotless-runner-k8s/tenant/knotless-runner.kubeconfig.yaml`, mode
0600. The script checks that the token can create Jobs but cannot read
nodes.

To delete the tenant cluster:

```bash
DELETE=1 scripts/tenant-kubeconfig.sh
```

## How a run works

1. `kubectl apply` a `batch/v1 Job` whose pod runs `sleep infinity`.
2. Wait for the pod to be Ready.
3. Stream the checkout to `/work` with tar over `kubectl exec -i`.
4. Stream shared steps to `/steps`.
5. Run each step with `kubectl exec <pod> -- sh -ec <script>`, appending
   stdout and stderr to one log.
6. Stream `/outputs` back, if asked.
7. Delete the Job.

A step timeout kills the local `kubectl` client and then the Job cleanup
deletes the pod, which stops the remote process. The proof does not yet
use a Kubernetes client crate; it shells out to `kubectl`, as the docker
executor shells out to `docker`.

## Result

```json
{
  "name": "knotless-poc",
  "job": "knotless-poc",
  "namespace": "default",
  "pod": "knotless-poc-abcde",
  "status": "success",
  "steps": [
    {"name": "date", "status": "success", "exitCode": 0, "durationMs": 42, "timedOut": false}
  ],
  "log": "/home/.../run.log",
  "outputs": "/home/.../outputs",
  "jobDeleted": true,
  "dryRun": false,
  "error": null
}
```

The step fields mirror `dev.knotless.check`'s `steps` array.

## Not in this proof

- workflow YAML parsing; steps are `name<TAB>script`
- caches and persistent volumes
- secret and variable injection
- XRPC cancellation
- queue landing
- a scoped runner identity for the public tenant route; `getKubeconfig`
  is cluster-admin by construction, and `internal tenantcredential` is
  for the loopback apiserver

See `PLAN.md` for the integration plan and the remaining risks.