Keep Docker job networks inside the range that can reach the cache #6

Merged
matchu merged 1 commit from armin-bot/openneo-code:claude/docker-address-pool into main 2026-11-17 12:39:04 -08:00
Contributor

CI has been flipping between fast runs (about 2 minutes, cache hits) and slow ones (8–12 minutes, where actions/cache times out and every native gem compiles from scratch). Moving the runner to its own box didn't fix it, because the box was never the problem.

Cause

  • The runner creates a fresh Docker network for every job.
  • Docker 26 and older don't reuse the lowest free subnet: they rotate through their whole default list. That's predefinedStartIndex in libnetwork/ipam/allocator.go, and the list is in ipamutils/utils.go: 172.17–172.31 as /16s, then 192.168.0.0/16 as /20s.
  • ufw only lets 172.16.0.0/12 reach the cache proxy. It leaves out 192.168.0.0/16 on purpose, because that overlaps Linode's private network. So jobs on 192.168.x networks have their cache requests silently dropped, and give up after about 20 seconds.
  • The result is a cycle: about 14 jobs that reach the cache, then about 16 that don't, and so on. A Docker restart resets the cycle, which is why fresh boxes start out fast.

Evidence

  • Job timings fit the cycle on both boxes. Old box: jobs 4–14 fast, 15–29 slow, 30–44 fast, 45–55 slow. New box: fast for its first ~14 jobs, slow from job 71.
  • While a slow job was running, docker network inspect on runner.openneo.net showed its network as 192.168.32.0/20, with no leftover networks from earlier jobs.

Fix

setup-actions.yml writes /etc/docker/daemon.json, which limits Docker's pool to 172.20.0.0/14 in /24s. That's 1,024 job networks, all inside the range ufw allows, so the rotation no longer matters. The ufw rule stays as it is, since it already covers this range. The new task restarts Docker (and the runner with it) only when the file changes.

Rollout

At a quiet moment (restarting Docker kills a running job):

ansible-playbook setup-actions.yml --limit runner

During the next job, sudo docker network ls and inspect on runner.openneo.net should show a 172.20.x.0/24 network, and Restore cached gems should take a second or two, not about 21 seconds.

Testing

ansible-playbook --syntax-check passes. 172.20.0.0/14 is within 172.16.0.0/12. Not yet run against the box.

🤖 Generated with Claude Code

https://claude.ai/code/session_014PWWd1c4UFRKce9t7XXiaP

CI has been flipping between fast runs (about 2 minutes, cache hits) and slow ones (8–12 minutes, where `actions/cache` times out and every native gem compiles from scratch). Moving the runner to its own box didn't fix it, because the box was never the problem. ## Cause - The runner creates a fresh Docker network for every job. - **Docker 26 and older don't reuse the lowest free subnet: they rotate through their whole default list.** That's `predefinedStartIndex` in `libnetwork/ipam/allocator.go`, and the list is in `ipamutils/utils.go`: `172.17`–`172.31` as /16s, then `192.168.0.0/16` as /20s. - ufw only lets **`172.16.0.0/12`** reach the cache proxy. It leaves out `192.168.0.0/16` on purpose, because that overlaps Linode's private network. So jobs on `192.168.x` networks have their cache requests silently dropped, and give up after about 20 seconds. - The result is a cycle: **about 14 jobs that reach the cache, then about 16 that don't**, and so on. A Docker restart resets the cycle, which is why fresh boxes start out fast. **Evidence** - Job timings fit the cycle on both boxes. Old box: jobs 4–14 fast, 15–29 slow, 30–44 fast, 45–55 slow. New box: fast for its first ~14 jobs, slow from job 71. - While a slow job was running, `docker network inspect` on runner.openneo.net showed its network as **`192.168.32.0/20`**, with no leftover networks from earlier jobs. ## Fix `setup-actions.yml` writes `/etc/docker/daemon.json`, which limits Docker's pool to **`172.20.0.0/14` in /24s**. That's 1,024 job networks, all inside the range ufw allows, so the rotation no longer matters. The ufw rule stays as it is, since it already covers this range. The new task restarts Docker (and the runner with it) only when the file changes. ## Rollout At a quiet moment (restarting Docker kills a running job): ```bash ansible-playbook setup-actions.yml --limit runner ``` During the next job, `sudo docker network ls` and `inspect` on runner.openneo.net should show a `172.20.x.0/24` network, and **Restore cached gems** should take a second or two, not about 21 seconds. ## Testing `ansible-playbook --syntax-check` passes. `172.20.0.0/14` is within `172.16.0.0/12`. Not yet run against the box. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014PWWd1c4UFRKce9t7XXiaP
Docker 26 and older rotate through their whole default address pool list
for new networks: 172.17-31 as /16s, then 192.168.0.0/16 as /20s. ufw only
lets 172.16.0.0/12 reach the runner's cache proxy, so CI alternated between
about 14 jobs that hit the cache and about 16 that timed out and rebuilt
every gem from scratch.

Pin Docker's pool to 172.20.0.0/14 in /24s (about 1000 networks), all
inside the allowed range.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PWWd1c4UFRKce9t7XXiaP
matchu merged commit 2a522e5f5b into main 2026-11-17 12:39:04 -08:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
OpenNeo/openneo-code!6
No description provided.