apps(forgejo): clear stale dind pid on start so runner restarts self-heal #14

Merged
sirius merged 1 commit from fix/dind-stale-pid into master 2026-11-17 20:10:52 +00:00
Owner

Problem

siriussec was rebooted and the amd64 Forgejo runner came back in CrashLoopBackOff:

failed to start daemon, ensure docker is not running or delete
/var/run/docker.pid: process with PID 7 is still running

dind-run is an emptyDir mounted at /var/run. A reboot restarts the dind
container inside the same pod, so the emptyDir survives and still holds the
dead daemon's docker.pid. In the fresh container PID 7 is some other process,
so dockerd's liveness check fails and it refuses to start — permanently, until
someone deletes the pod.

This affects runner-arm64.yaml identically; it just hadn't rebooted into the
bad state yet.

Change

Clear the stale runtime state immediately before starting dockerd, in both
runner manifests:

rm -f /var/run/docker.pid /var/run/docker.sock

The container is fresh at that point, so there is no live daemon to disturb —
this only removes leftovers from the previous container instance.

Verification

  • Both files parse (yaml.safe_load_all): amd64 -> PVC + Deployment, arm64 -> Deployment.
  • The rendered dind command passes sh -n.
  • Immediate recovery on the live cluster was confirmed by deleting the wedged
    pod (state lives on the forgejo-runner-amd64 PVC, nothing lost):
forgejo-runner-amd64-f7bb99fb-7v95c   2/2   Running   0   7s
msg="runner: siriussec-amd64, with version: v5.0.1, with labels: [docker-amd64 amd64], declared successfully"
msg="[poller 0] launched"

No cluster-side manifest change was made outside this PR; the live fix was the
pod delete above.

## Problem siriussec was rebooted and the amd64 Forgejo runner came back in `CrashLoopBackOff`: ``` failed to start daemon, ensure docker is not running or delete /var/run/docker.pid: process with PID 7 is still running ``` `dind-run` is an `emptyDir` mounted at `/var/run`. A reboot restarts the dind container *inside the same pod*, so the emptyDir survives and still holds the dead daemon's `docker.pid`. In the fresh container PID 7 is some other process, so dockerd's liveness check fails and it refuses to start — permanently, until someone deletes the pod. This affects `runner-arm64.yaml` identically; it just hadn't rebooted into the bad state yet. ## Change Clear the stale runtime state immediately before starting dockerd, in both runner manifests: ```sh rm -f /var/run/docker.pid /var/run/docker.sock ``` The container is fresh at that point, so there is no live daemon to disturb — this only removes leftovers from the previous container instance. ## Verification - Both files parse (`yaml.safe_load_all`): amd64 -> PVC + Deployment, arm64 -> Deployment. - The rendered `dind` command passes `sh -n`. - Immediate recovery on the live cluster was confirmed by deleting the wedged pod (state lives on the `forgejo-runner-amd64` PVC, nothing lost): ``` forgejo-runner-amd64-f7bb99fb-7v95c 2/2 Running 0 7s msg="runner: siriussec-amd64, with version: v5.0.1, with labels: [docker-amd64 amd64], declared successfully" msg="[poller 0] launched" ``` No cluster-side manifest change was made outside this PR; the live fix was the pod delete above.
The dind container mounts the dind-run emptyDir at /var/run. A reboot
restarts the container inside the same pod, so that emptyDir keeps the
dead daemon's docker.pid; dockerd then aborts with "process with PID N
is still running" and the runner sits in CrashLoopBackOff until someone
deletes the pod. Removing the stale pid and socket before starting
dockerd makes both the amd64 and arm64 runners recover on their own.
sirius merged commit 5f5a0752ff into master 2026-11-17 20:10:52 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
sirius/homek8s!14
No description provided.