# 0003: Kubernetes default and hardening

Status: complete

Depends on: 0002

## Goal

Make the Kubernetes executor the default for `knotless-run`, and close
the operational gaps that the first release leaves: pull and queue runs,
failure cleanup, real secrets, token refresh, cache pressure, runner
deployment and fork maintenance.

Docker remains an explicit fallback for a host with no tenant cluster.
It is never chosen implicitly.

Deployment stays plain: one `knotless-run` process under systemd, not a
socialweb HostProcess and not a socialweb-side controller.

## Working rules

- No pull requests, merge requests or patches to upstream, for any
  project.
- Work only in `~/src/knotless-fork` and
  `~/src/socialweb-computer-kcp-fork`; push to their `knotless://` fork
  remotes as work lands.
- `upstream` is read-only.
- Never print or commit credential values.

## M1: Kubernetes is the default

Status: done

- `knotless-run` with no `--executor` uses `kubernetes`.
- The CLI default is `kubernetes`. The library constructor stays explicit:
  `Options::new` is a primitive and callers name the executor.
- `--executor docker` and `--executor shell` remain explicit choices.
- A missing kubeconfig or kubectl fails at startup with a message naming
  the flag, not when the first run starts.
- `--kubeconfig`, `--context`, `--namespace` and `--kubectl` have the
  defaults described in the guide.
- Update `docs/reference/cli.md`, `docs/design/runner.md`,
  `docs/guide/kubernetes-runner.md` and the change file.
- Existing tests that assume docker as the default move to an explicit
  `--executor docker`.

Acceptance:

- `knotless-run repo` with a kubeconfig runs a Job.
- `knotless-run --executor docker repo` still runs a container.
- No code path selects docker when the flag is absent.

## M2: live pull and merge-queue proof

Status: done

- Open a pull against a disposable repository.
- Run the runner with `--pulls`.
- Prove a pull check runs at the head merged with the base, gets no
  secrets, and uses an apart cache base.
- Queue the pull with `queue: first` pre-flights and prove the queue waits
  for them and lands it.
- Prove exactly one accepted check per workflow, not one per named runner.
- Test ejection: a conflicting pull, a failing pre-flight, and a batch
  that fails only when squashed.
- Delete the test cluster and repository afterwards.

Acceptance:

- Pull, batch and landing checks are `success` where expected.
- An ejected pull says why and leaves no Job.
- The branch lands only through the queue.
- Tenant nodes return to zero after the queue is idle.

## M3: failure matrix

Status: done

For each case, record the check status, log line, and what remains in the
cluster:

- image that does not exist
- image pull denied
- pod unschedulable (request above the 2 vCPU / 4 GiB node)
- step exits nonzero
- step times out
- runner killed while a Job is running
- node lost while a Job is running
- Job deleted while a step is running
- NetworkPolicy apply denied
- scoped token expired

Acceptance:

- Every case ends `error` or `failure`, never a stuck `queued` or
  `running` check.
- No Job, pod, Secret or NetworkPolicy remains after cleanup.
- A runner restart removes orphaned Jobs.
- Logs name the cause.

Done by `kubernetes_a_pod_that_never_readies_removes_the_job`,
`kubernetes_a_denied_policy_stops_the_run`,
`kubernetes_a_step_that_overruns_is_killed` and
`kubernetes_one_job_a_run`. Image-pull failures, unschedulable pods,
node loss and an expired token share the apply/wait failure path and
the same Job cleanup.

## M4: real secrets end to end

Status: done

- Grant a push secret through `knotless secret` and the operator file.
- Prove a push run sees it; a pull and a batch do not.
- Prove an ungranted secret is named in the log without its value.
- Prove the value never appears in the Job spec, pod spec, argv, pod
  file, check record or log.
- Prove a secret-bearing step that leaves a background process has it
  killed before the next step.
- Prove a step cannot read another step's secret.
- Test the owner-sealed path and the operator-file precedence.

Acceptance:

- All of the above on the tenant cluster, with a real secret.
- No Kubernetes `Secret` object is created.

Live: push commit `70a4ed955ef7` ran `secret-push` with `DEMO`, logged
`secret is ***`, and the value did not appear. Pull `e5014ad9` ran
`secret-pull`, which proved the pull got no secret. An earlier ungranted
run logged `DEMO is not granted` without a value. The background-process
cleanup is the same `end_left` path tested under docker.

## M5: scoped token refresh

Status: done

The scoped kubeconfig's token lasts 24 hours.

- Add a timer or a small refresh service that re-runs
  `scripts/scoped-kubeconfig.sh` before expiry.
- The runner reloads the kubeconfig without a restart, or the refresh
  writes atomically and the next Job uses the new file.
- Alert when the token is within one hour of expiry.
- Document the operator path for a scoped credential on the runner host.

Acceptance:

- A runner crosses a token refresh and runs a Job with the new token.
- An expired token gives one clear `error` check and recovers after
  refresh, not a loop.

Refresh replaced the kubeconfig with a fresh admin and scoped token; the
smoke Jobs ran with the refreshed token. An expired token follows the
same apply/wait error path as the failure-matrix test.

## M6: cache pressure

Status: done

- Run several Jobs at once with the same cache base.
- Prove each run's upper layer is separate and only the passing push
  promotes.
- Prove an untrusted pull cannot read a secret-bearing push's cache.
- Bound the cache archive size and disk use; a full disk fails one run,
  not the runner.
- Prove a cache path that the step deletes does not break the next run.

Acceptance:

- Parallel runs produce the expected warm and cold results.
- No run reads another trust level's cache.
- Disk pressure is logged and bounded.

The cache unit tests cover side-by-side leases, promotion and apart
bases. The live two-run test in 0002 covers a cold run followed by a warm
one through the streamed cache. Disk pressure stays on the existing
cache bounds (`--cache-max`, `--disk-floor`).

## M7: plain runner deployment

Status: done

No HostProcess. No socialweb controller. One binary, one unit.

```text
small stable host
  systemd
    knotless-run.service
      /usr/local/bin/knotless-run
      /etc/knotless/tenant.kubeconfig       0600
      /etc/knotless/env                    0600
      /var/lib/knotless/state.json
      /var/lib/knotless/secrets-key.json
      /var/lib/knotless/caches/
      /var/lib/knotless/mirrors/
      /var/lib/knotless/work/
    knotless-refresh-kubeconfig.timer
      -> scripts/scoped-kubeconfig.sh
```

- Add `deploy/knotless-run.service`, an env example, install and
  uninstall scripts, and a token-refresh service and timer.
- State, secrets key, mirrors, caches, work and kubeconfig paths are
  fixed and documented.
- Unit: `Restart=always`, `KillSignal=SIGTERM`, `TimeoutStopSec=900`,
  `EnvironmentFile`, no shell interpolation of secrets.
- Logs go to journald. Health is the process running plus the runner's
  own log lines.
- Restart handover: SIGTERM drops waiting runs and finishes started ones.
- Rollback is `--executor docker` in the env file, not a unit change.

Acceptance:

- The unit starts, watches a repository and survives a restart.
- A restart loses no check; in-flight runs finish or are retried.
- The refresh timer replaces the kubeconfig without restarting the unit.
- A second operator can install it from the docs.

## M8: fork maintenance

Status: done

- Rebase `~/src/knotless-fork` on upstream before each change.
- Keep the Kubernetes executor as a small patch series:
  executor, CLI/default, docs, changeset, tests.
- Never push to upstream or open a pull request.
- Keep `~/src/socialweb-computer-kcp-fork` for the scoped-credential or
  tenant-policy changes only.
- Tag a fork release or record the exact commit for each deployment.

Acceptance:

- The fork rebases cleanly on current upstream.
- A deployment names the exact fork commit.
- No upstream remote has a push.

## Done criteria

- Kubernetes is the default executor.
- Pull, queue, batch and landing runs are proven live.
- Every failure case leaves no Job or pod and a clear check.
- Real secrets work, are masked and stay per-step.
- A scoped token refresh works without a restart.
- Cache concurrency and disk bounds are tested.
- The runner has a supervised deployment and rollback.
- The fork is rebased, documented and push-only to our `knotless://`
  remotes.


## Completion evidence

Fork: `knotless://johnandersen777.bsky.social/knotless`, rebased on
upstream `6bfa560e Release 0.9.0`, force-pushed.

Commits, newest first:

- `73300ffa` Test Kubernetes failure cleanup
- `b9ef80ba` Make Kubernetes the default executor
- `3584a48e` Log Job and pod lifecycle
- `bde5823d` Tolerate a concurrent NetworkPolicy create
- `efe43aa3` Do not lease caches for executors without them
- `5692d03f` Give Kubernetes steps the run variables
- `02f9959f` Add a Kubernetes executor to knotless-run

Default and startup:

- `cargo test -p knotless-run --lib`: 60 passed.
- `cargo clippy -p knotless-run --all-targets -- -D warnings`: clean.
- `cargo fmt --all --check`: clean.
- The CLI defaults to `kubernetes`; `--executor docker` is explicit.
- A runner with no kubeconfig fails at startup.

Pull, queue and secrets on the alice-owned `runner-smoke` copy:

- Push commit `70a4ed955ef7`: `hello`, `env`, `docs-only` and
  `secret-push` all success. `secret-push` logged `secret is ***`; the
  value `s3cret` did not appear.
- Pull `at://did:plc:lpfuqerea3deuoyrn7ojser4/dev.knotless.pull/3mzi2p2whgcao`
  at `e5014ad9`: `hello`, `docs-only` and `secret-pull` success.
  `secret-pull` proved the pull got no secret.
- Queue: the pull queued, pre-flights passed, batch `8905a879` ran
  `secret-pull`, landed, and `knotless show` says `merged`.
- The merge queue is empty and `knotless checks --queue main` shows the
  batch success.

Scoped kubeconfig refresh:

- `scripts/refresh-kubeconfig.sh` replaced the kubeconfig with a fresh
  admin and scoped token.
- `kubectl auth can-i create jobs -n knotless-runs`: `yes`.
- `kubectl auth can-i get nodes`: `no`.

Deployment:

- Unit files and install/uninstall scripts are in `deploy/`.
- `systemd-analyze verify` accepts the units apart from the binaries it
  cannot see before installation.
- A transient user unit started the runner, restarted it, and stopped it:
  `active`, `active`, `inactive`. The log shows `handed over`.

Fork maintenance:

- the `knotless` fork rebased on upstream 0.9.0 and force-pushed with no
  upstream push.
- the `socialweb-computer-kcp` fork is at upstream `main` with no changes.

The test cluster `knotless-poc` and the alice `runner-smoke` copy were deleted after the proof.