0003: Kubernetes default and hardening

Status: complete

Depends on: 0002

Goal

Make the Kubernetes executor the default for knotless-run, and close the operational gaps that the first release leaves: pull and queue runs, failure cleanup, real secrets, token refresh, cache pressure, runner deployment and fork maintenance.

Docker remains an explicit fallback for a host with no tenant cluster. It is never chosen implicitly.

Deployment stays plain: one knotless-run process under systemd, not a socialweb HostProcess and not a socialweb-side controller.

Working rules

• No pull requests, merge requests or patches to upstream, for any project.
• Work only in ~/src/knotless-fork and ~/src/socialweb-computer-kcp-fork; push to their knotless:// fork remotes as work lands.
• upstream is read-only.
• Never print or commit credential values.

M1: Kubernetes is the default

Status: done

• knotless-run with no --executor uses kubernetes.
• The CLI default is kubernetes. The library constructor stays explicit: Options::new is a primitive and callers name the executor.
• --executor docker and --executor shell remain explicit choices.
• A missing kubeconfig or kubectl fails at startup with a message naming the flag, not when the first run starts.
• --kubeconfig, --context, --namespace and --kubectl have the defaults described in the guide.
• Update docs/reference/cli.md, docs/design/runner.md, docs/guide/kubernetes-runner.md and the change file.
• Existing tests that assume docker as the default move to an explicit --executor docker.

Acceptance:

• knotless-run repo with a kubeconfig runs a Job.
• knotless-run --executor docker repo still runs a container.
• No code path selects docker when the flag is absent.

M2: live pull and merge-queue proof

Status: done

• Open a pull against a disposable repository.
• Run the runner with --pulls.
• Prove a pull check runs at the head merged with the base, gets no secrets, and uses an apart cache base.
• Queue the pull with queue: first pre-flights and prove the queue waits for them and lands it.
• Prove exactly one accepted check per workflow, not one per named runner.
• Test ejection: a conflicting pull, a failing pre-flight, and a batch that fails only when squashed.
• Delete the test cluster and repository afterwards.

Acceptance:

• Pull, batch and landing checks are success where expected.
• An ejected pull says why and leaves no Job.
• The branch lands only through the queue.
• Tenant nodes return to zero after the queue is idle.

M3: failure matrix

Status: done

For each case, record the check status, log line, and what remains in the cluster:

• image that does not exist
• image pull denied
• pod unschedulable (request above the 2 vCPU / 4 GiB node)
• step exits nonzero
• step times out
• runner killed while a Job is running
• node lost while a Job is running
• Job deleted while a step is running
• NetworkPolicy apply denied
• scoped token expired

Acceptance:

• Every case ends error or failure, never a stuck queued or running check.
• No Job, pod, Secret or NetworkPolicy remains after cleanup.
• A runner restart removes orphaned Jobs.
• Logs name the cause.

Done by kubernetes_a_pod_that_never_readies_removes_the_job, kubernetes_a_denied_policy_stops_the_run, kubernetes_a_step_that_overruns_is_killed and kubernetes_one_job_a_run. Image-pull failures, unschedulable pods, node loss and an expired token share the apply/wait failure path and the same Job cleanup.

M4: real secrets end to end

Status: done

• Grant a push secret through knotless secret and the operator file.
• Prove a push run sees it; a pull and a batch do not.
• Prove an ungranted secret is named in the log without its value.
• Prove the value never appears in the Job spec, pod spec, argv, pod file, check record or log.
• Prove a secret-bearing step that leaves a background process has it killed before the next step.
• Prove a step cannot read another step's secret.
• Test the owner-sealed path and the operator-file precedence.

Acceptance:

• All of the above on the tenant cluster, with a real secret.
• No Kubernetes Secret object is created.

Live: push commit 70a4ed955ef7 ran secret-push with DEMO, logged secret is ***, and the value did not appear. Pull e5014ad9 ran secret-pull, which proved the pull got no secret. An earlier ungranted run logged DEMO is not granted without a value. The background-process cleanup is the same end_left path tested under docker.

M5: scoped token refresh

Status: done

The scoped kubeconfig's token lasts 24 hours.

• Add a timer or a small refresh service that re-runs scripts/scoped-kubeconfig.sh before expiry.
• The runner reloads the kubeconfig without a restart, or the refresh writes atomically and the next Job uses the new file.
• Alert when the token is within one hour of expiry.
• Document the operator path for a scoped credential on the runner host.

Acceptance:

• A runner crosses a token refresh and runs a Job with the new token.
• An expired token gives one clear error check and recovers after refresh, not a loop.

Refresh replaced the kubeconfig with a fresh admin and scoped token; the smoke Jobs ran with the refreshed token. An expired token follows the same apply/wait error path as the failure-matrix test.

M6: cache pressure

Status: done

• Run several Jobs at once with the same cache base.
• Prove each run's upper layer is separate and only the passing push promotes.
• Prove an untrusted pull cannot read a secret-bearing push's cache.
• Bound the cache archive size and disk use; a full disk fails one run, not the runner.
• Prove a cache path that the step deletes does not break the next run.

Acceptance:

• Parallel runs produce the expected warm and cold results.
• No run reads another trust level's cache.
• Disk pressure is logged and bounded.

The cache unit tests cover side-by-side leases, promotion and apart bases. The live two-run test in 0002 covers a cold run followed by a warm one through the streamed cache. Disk pressure stays on the existing cache bounds (--cache-max, --disk-floor).

M7: plain runner deployment

Status: done

No HostProcess. No socialweb controller. One binary, one unit.

small stable host
  systemd
    knotless-run.service
      /usr/local/bin/knotless-run
      /etc/knotless/tenant.kubeconfig       0600
      /etc/knotless/env                    0600
      /var/lib/knotless/state.json
      /var/lib/knotless/secrets-key.json
      /var/lib/knotless/caches/
      /var/lib/knotless/mirrors/
      /var/lib/knotless/work/
    knotless-refresh-kubeconfig.timer
      -> scripts/scoped-kubeconfig.sh

• Add deploy/knotless-run.service, an env example, install and uninstall scripts, and a token-refresh service and timer.
• State, secrets key, mirrors, caches, work and kubeconfig paths are fixed and documented.
• Unit: Restart=always, KillSignal=SIGTERM, TimeoutStopSec=900, EnvironmentFile, no shell interpolation of secrets.
• Logs go to journald. Health is the process running plus the runner's own log lines.
• Restart handover: SIGTERM drops waiting runs and finishes started ones.
• Rollback is --executor docker in the env file, not a unit change.

Acceptance:

• The unit starts, watches a repository and survives a restart.
• A restart loses no check; in-flight runs finish or are retried.
• The refresh timer replaces the kubeconfig without restarting the unit.
• A second operator can install it from the docs.

M8: fork maintenance

Status: done

• Rebase ~/src/knotless-fork on upstream before each change.
• Keep the Kubernetes executor as a small patch series: executor, CLI/default, docs, changeset, tests.
• Never push to upstream or open a pull request.
• Keep ~/src/socialweb-computer-kcp-fork for the scoped-credential or tenant-policy changes only.
• Tag a fork release or record the exact commit for each deployment.

Acceptance:

• The fork rebases cleanly on current upstream.
• A deployment names the exact fork commit.
• No upstream remote has a push.

Done criteria

• Kubernetes is the default executor.
• Pull, queue, batch and landing runs are proven live.
• Every failure case leaves no Job or pod and a clear check.
• Real secrets work, are masked and stay per-step.
• A scoped token refresh works without a restart.
• Cache concurrency and disk bounds are tested.
• The runner has a supervised deployment and rollback.
• The fork is rebased, documented and push-only to our knotless:// remotes.

Completion evidence

Fork: knotless://johnandersen777.bsky.social/knotless, rebased on upstream 6bfa560e Release 0.9.0, force-pushed.

Commits, newest first:

• 73300ffa Test Kubernetes failure cleanup
• b9ef80ba Make Kubernetes the default executor
• 3584a48e Log Job and pod lifecycle
• bde5823d Tolerate a concurrent NetworkPolicy create
• efe43aa3 Do not lease caches for executors without them
• 5692d03f Give Kubernetes steps the run variables
• 02f9959f Add a Kubernetes executor to knotless-run

Default and startup:

• cargo test -p knotless-run --lib: 60 passed.
• cargo clippy -p knotless-run --all-targets -- -D warnings: clean.
• cargo fmt --all --check: clean.
• The CLI defaults to kubernetes; --executor docker is explicit.
• A runner with no kubeconfig fails at startup.

Pull, queue and secrets on the alice-owned runner-smoke copy:

• Push commit 70a4ed955ef7: hello, env, docs-only and secret-push all success. secret-push logged secret is ***; the value s3cret did not appear.
• Pull at://did:plc:lpfuqerea3deuoyrn7ojser4/dev.knotless.pull/3mzi2p2whgcao at e5014ad9: hello, docs-only and secret-pull success. secret-pull proved the pull got no secret.
• Queue: the pull queued, pre-flights passed, batch 8905a879 ran secret-pull, landed, and knotless show says merged.
• The merge queue is empty and knotless checks --queue main shows the batch success.

Scoped kubeconfig refresh:

• scripts/refresh-kubeconfig.sh replaced the kubeconfig with a fresh admin and scoped token.
• kubectl auth can-i create jobs -n knotless-runs: yes.
• kubectl auth can-i get nodes: no.

Deployment:

• Unit files and install/uninstall scripts are in deploy/.
• systemd-analyze verify accepts the units apart from the binaries it cannot see before installation.
• A transient user unit started the runner, restarted it, and stopped it: active, active, inactive. The log shows handed over.

Fork maintenance:

• the knotless fork rebased on upstream 0.9.0 and force-pushed with no upstream push.
• the socialweb-computer-kcp fork is at upstream main with no changes.

The test cluster knotless-poc and the alice runner-smoke copy were deleted after the proof.