chadtmiller.com/knotless

Switch repository

Loading…

What changes

knotless.dev's runner waits an hour for its runs as it stops. deploy/docker-compose.yml gives run --stop-wait 3600, and stop_grace_period 65 minutes, up from the default 15 and 20. A deploy that changes the runner hands over to a new one, and the old one now finishes the runs it has going, a release's build among them, rather than cancelling those still going after 15 minutes. A step that hangs still ends at --step-timeout (30 minutes).

The runner host keeps 15 minutes. Its deploy recreates the runner rather than handing over, so a longer wait there would hold its deploy, and every pull's checks, for as long. Its compose file says why.

deploy/README.md says the same.

Why

Three of the last four releases were cancelled, 0.5.0, 0.6.0 and 0.8.0, each "as the runner stopped, past what a stop waits for", and published only when run again by hand. A version bump changes every binary, so the deploy at the release's commit always hands the runner over, and the release's build of four targets takes 14 to 17 minutes, past the old runner's 15.

A runner draining what it runs is how CI runners stop, and keeps a release and a deploy independent of each other. Waiting costs no capacity: the new runner lends the old one the jobs its runs hold, so the two use no more than one, and the deploy itself does not wait (handover.sh starts the new runner at once). A release made to wait after: [deploy] was the other way considered, A release waits after the deploy and runs on the runner handed over to, now closed: it needed the runner to drop such runs on a stop, and tied every client release to a deploy passing.

How it was checked

Read against handover.sh and deploy.sh: the main host's deploy recreates every service but run, and hands run over in the background, so nothing in a deploy waits on the longer stop. A runner that writes no handover note is recreated by that background script, which then waits up to the grace period, as it waited up to 20 minutes before. The setting takes effect with the deploy of this pull, which restarts the runner, its settings having changed.

Changeset: none

Shortcomings

An old runner can stay up for an hour after a deploy when one of its runs hangs, holding a lent job meanwhile. A run asked for by name, as a release run again by hand, is still cancelled at once on a stop, and run again by the runner started next.

This sets --stop-wait 3600 and stop_grace_period: 65m for the main host's runner, so a handed-over runner can finish a release build. The runner-host stays at the default 15 minutes, with a comment saying why.

I read handover.sh against the change. The hand-over sends SIGTERM with docker kill and sets --restart=no. The new runner starts right after the handover note appears, and the old one is removed in the background. A deploy therefore never waits on the longer stop. The 65m grace period only matters for a docker stop by hand or for the no-note recreate path, and it is longer than --stop-wait there as intended.

--stop-wait is already a parsed flag, and the README and compose comments match the new values.

One thing to fix later, not a blocker. docs/reference/cli.md and docs/design/runner.md still give the default as 900. That stays correct, because the default is unchanged. The hour is deployment-specific.