Someone trying knotless-run on Kubernetes asked how it could run jobs on a cluster. --executor takes only docker or shell, so a runner in a cluster needs a docker daemon beside it. That means a privileged docker-in-docker sidecar, which is not fit for code nobody vouches for. I have read the code for this but tried none of it on a cluster.
That sidecar route needs no new code for trusted repositories: install.sh brings knotless-run, and the published knotless-server image holds it and the docker client. No guide covers running it in a cluster.
The proposal is --executor kube, for a runner that runs inside the cluster it runs jobs on, with no new XRPC, record or stored kubeconfig. It takes its credentials from standard kube client config: the pod's ServiceAccount, or KUBECONFIG. An exec credential plugin should then work without knotless knowing of it, so short-lived credentials stay the operator's choice.
An owner can already run their own runner and send only some workflows to it, with runners: in the workflow and knotless secret set --runner. It needs only outbound connections, so any provider's cluster works with nothing inbound. The cost is one stateful pod and an atproto account for the runner.
Not part of this: a hosted runner running jobs on an owner's cluster. That needs a credential to travel, onto a host that runs strangers' code. A service token proves which runner is asking, but not which repository or ref the run is for, and knotless has no identity per run. That would be its own feature, and should come first if this is ever wanted. A step that deploys with kubectl already works with a secret.
What the executor has to solve
executor.rs is the only place that runs docker, behind the Executor and Session enums. Labels, cleanup and exit codes should translate easily. These will not:
Caches. They are overlay volumes, with renames and hard links on a disk shared with the runner (caches.rs). The first version can run without them, cold, saying so in the log. Node-pinned storage can follow.
Checkout, outputs and shared steps. These are bind mounts from the runner's disk. Pods on other nodes need them copied in and out, or a shared volume.
Secrets for each step.docker exec -e NAME takes the value from the client's environment, and kubectl exec has no -e. executor::run sets stdin to null, so sending them over stdin needs a new path. They cannot go in the pod spec, as the container must never hold them.
Long steps. A step is one attached stream, so an idle timeout on a load balancer could look like a failed step. This needs testing on a real cluster, and steps may have to run detached and be polled.
Limits.
--cpus, --memory and --runtime map to resources and RuntimeClass.
--network needs a NetworkPolicy and a CNI that enforces it.
--pids has no per-pod setting that I know of, only a kubelet-wide one.
--disk-floor does not apply.
What run pods can reach. They need automountServiceAccountToken: false, and the runner's Role should cover only pods and pods/exec in one namespace. Otherwise a stranger's pull gets the runner's access to the cluster.
The runner itself. Its state file and mirror are local, so it needs a persistent volume. A runner handing over runs beside the next one on the same state directory (docs/design/runner.md, Handing over), which a one-replica StatefulSet cannot do: it starts the new pod only once the old one is gone. So a restart either waits out the old runner's runs, with termination grace as long as --stop-wait, or needs another way to start the new one beside it.
Before building
Confirm that the person asking wants their own runner in their own cluster, not a hosted runner reaching into it.
chadtmiller.com/knotless
What happened
Someone trying knotless-run on Kubernetes asked how it could run jobs on a cluster.
--executortakes onlydockerorshell, so a runner in a cluster needs a docker daemon beside it. That means a privileged docker-in-docker sidecar, which is not fit for code nobody vouches for. I have read the code for this but tried none of it on a cluster.That sidecar route needs no new code for trusted repositories:
install.shbringsknotless-run, and the publishedknotless-serverimage holds it and the docker client. No guide covers running it in a cluster.The proposal is
--executor kube, for a runner that runs inside the cluster it runs jobs on, with no new XRPC, record or stored kubeconfig. It takes its credentials from standard kube client config: the pod's ServiceAccount, orKUBECONFIG. An exec credential plugin should then work without knotless knowing of it, so short-lived credentials stay the operator's choice.An owner can already run their own runner and send only some workflows to it, with
runners:in the workflow andknotless secret set --runner. It needs only outbound connections, so any provider's cluster works with nothing inbound. The cost is one stateful pod and an atproto account for the runner.Not part of this: a hosted runner running jobs on an owner's cluster. That needs a credential to travel, onto a host that runs strangers' code. A service token proves which runner is asking, but not which repository or ref the run is for, and knotless has no identity per run. That would be its own feature, and should come first if this is ever wanted. A step that deploys with
kubectlalready works with a secret.What the executor has to solve
executor.rsis the only place that runs docker, behind theExecutorandSessionenums. Labels, cleanup and exit codes should translate easily. These will not:caches.rs). The first version can run without them, cold, saying so in the log. Node-pinned storage can follow.docker exec -e NAMEtakes the value from the client's environment, andkubectl exechas no-e.executor::runsets stdin to null, so sending them over stdin needs a new path. They cannot go in the pod spec, as the container must never hold them.--cpus,--memoryand--runtimemap to resources and RuntimeClass.--networkneeds a NetworkPolicy and a CNI that enforces it.--pidshas no per-pod setting that I know of, only a kubelet-wide one.--disk-floordoes not apply.automountServiceAccountToken: false, and the runner's Role should cover only pods andpods/execin one namespace. Otherwise a stranger's pull gets the runner's access to the cluster.docs/design/runner.md, Handing over), which a one-replica StatefulSet cannot do: it starts the new pod only once the old one is gone. So a restart either waits out the old runner's runs, with termination grace as long as--stop-wait, or needs another way to start the new one beside it.Before building
Confirm that the person asking wants their own runner in their own cluster, not a hosted runner reaching into it.
Related: Self-hosted runners: ship knotless-run and document running your own