Move the Forgejo Actions runner to its own machine #5

Merged
matchu merged 2 commits from armin-bot/openneo-code:claude/separate-runner-box into main 2026-11-16 21:36:11 -08:00
Contributor

CI on the shared box has been starving itself: in a recent deploy run, actions/cache timed out talking to the runner's own cache proxy (getCacheEntry failed: Request timeout), so Install gems recompiled every native gem from scratch (7 minutes). On one vCPU and 2GB, Forgejo plus a CI job pushes the machine into swap, and the cache proxy can't answer in time.

This moves the runner to its own machine, a shared Linode 2GB: runner.openneo.net. CI can no longer slow Forgejo down (or vice versa). Jobs, which run in Docker as a root-equivalent user, also no longer share a machine with Forgejo's repos and secrets.

The impress side is OpenNeo/impress#33, which lets deploy-bot's key work from the new box's IP.

Changes

  • inventory.cfg splits into two groups: forgejo (code.openneo.net) and runner (runner.openneo.net).
  • setup-machine.yml and setup-security.yml apply to both machines, and the hostname now comes from the inventory. The swap file moves here from setup-actions.yml, so both machines get one. The Forgejo box already has it, so nothing changes there.
  • setup-actions.yml targets runner. Registration runs Forgejo's own CLI, so that one task uses delegate_to to run on the Forgejo box. The runner's name comes from its host: runner-openneo-net.
  • The runner config is unchanged, apart from comments: capacity: 1 and the 1GB container caps still fit a 2GB box.
  • remove-colocated-runner.yml (new, one-off) removes the old runner from the Forgejo box: the service, binary, runner user (and its cache data), Docker and its data, the weekly prune cron and the port-4000 ufw rule. It can be deleted from the repo once it has run.
  • setup-security.yml now reloads sshd when its settings change (before, they waited for a reboot). It also checks sshd's effective config, because an override in sshd_config.d/ would win.
  • README: explains the two machines, how to bootstrap a new one (see below), and the runner's Linode Cloud Firewall.

Reusing the runner's identity

The new box registers with the same runner_secret. I checked RegisterRunner in Forgejo v17.0's models/actions/forgejo.go: the first 16 characters of the secret pick the runner's UUID, and re-registering an existing UUID updates its name, scope and labels in place. So the new box takes over the existing runner (renamed runner-openneo-net), and there's nothing to deregister. Both daemons can run at once during the move. Each just takes jobs as the same runner.

Rollout

Order matters, so that deploys never run from a machine deploy-bot doesn't accept, and the old runner never gets cut off mid-deploy:

  1. ✅ Create the Linode (shared 2GB, 74.207.234.34) with its Cloud Firewall: inbound Drop, except TCP 22; outbound Accept.
  2. DNS: point runner.openneo.net at it (A 74.207.234.34, AAAA 2600:3c02::2000:ecff:fe40:eada).
  3. Bootstrap your user (as root on the new box). Use the same sudo password as on code.openneo.net, because one setup-actions.yml run uses sudo on both machines. (A check before registration catches a mismatch.)
    adduser matchu
    usermod -aG sudo matchu
    cp -r ~/.ssh /home/matchu/
    chown -R matchu:matchu /home/matchu/.ssh
    
  4. impress: run bin/deploy:setup from OpenNeo/impress#33's branch, then merge it. deploy-bot now accepts both machines.
  5. Set up the runner box:
    ansible-playbook setup-all.yml --limit runner
    
    OpenNeo/impress → Settings → Actions → Runners should show runner-openneo-net. This also checks that sshd really has root and password login off: a fresh Linode can override them in /etc/ssh/sshd_config.d/.
  6. Stop the old runner, at a quiet moment. While both daemons run as the same runner, a passing job doesn't tell you which machine ran it. And stopping the old one cancels its current job after 5 minutes, which could interrupt a deploy. So, when the Actions tab shows nothing running, on code.openneo.net:
    sudo systemctl stop forgejo-runner
    
    Then re-run a CI job (or push something) and check that it passes. It's now definitely on the new box. Its cache starts cold, so expect one more slow gem install. If it fails, sudo systemctl start forgejo-runner brings the old runner back.
  7. Remove the old runner for good:
    ansible-playbook remove-colocated-runner.yml
    
    Then reboot code.openneo.net, to clear the network rules and mounts Docker leaves behind. (If deleting /var/lib/docker fails with "Device or resource busy", reboot first, then re-run the playbook.)
  8. impress follow-up: drop 45.79.220.14 from deploy_bot_allowed_ips, and run bin/deploy:setup again.

Testing

  • ansible-playbook --syntax-check passes for setup-all.yml, remove-colocated-runner.yml, and impress's deploy/setup.yml.
  • --list-hosts confirms that machine/security target both boxes, Forgejo only code.openneo.net, and the runner only runner.openneo.net.
  • Fresh-eyes review (a separate agent) checked delegate_to under --limit runner with a local two-host test: the delegated task runs on the Forgejo box even though it's outside the limit, and become_user, environment, chdir and stdin apply there. It also confirmed that two daemons sharing one runner identity is safe (each task is handed out once), and that the ufw delete matches the original commented rule. Its rollout catches are fixed: steps 6–7, the sudo check, and the sshd reload and effective-config check.
  • Not yet run against real machines.

🤖 Generated with Claude Code

https://claude.ai/code/session_014PWWd1c4UFRKce9t7XXiaP

CI on the shared box has been starving itself: in a recent deploy run, `actions/cache` timed out talking to the runner's own cache proxy (`getCacheEntry failed: Request timeout`), so **Install gems** recompiled every native gem from scratch (7 minutes). On one vCPU and 2GB, Forgejo plus a CI job pushes the machine into swap, and the cache proxy can't answer in time. This moves the runner to its own machine, a shared Linode 2GB: `runner.openneo.net`. CI can no longer slow Forgejo down (or vice versa). Jobs, which run in Docker as a root-equivalent user, also no longer share a machine with Forgejo's repos and secrets. The impress side is OpenNeo/impress#33, which lets `deploy-bot`'s key work from the new box's IP. ## Changes - **`inventory.cfg`** splits into two groups: `forgejo` (code.openneo.net) and `runner` (runner.openneo.net). - **`setup-machine.yml`** and **`setup-security.yml`** apply to both machines, and the hostname now comes from the inventory. The **swap file** moves here from `setup-actions.yml`, so both machines get one. The Forgejo box already has it, so nothing changes there. - **`setup-actions.yml`** targets `runner`. Registration runs Forgejo's own CLI, so that one task uses `delegate_to` to run on the Forgejo box. The runner's name comes from its host: `runner-openneo-net`. - **The runner config** is unchanged, apart from comments: `capacity: 1` and the 1GB container caps still fit a 2GB box. - **`remove-colocated-runner.yml`** (new, one-off) removes the old runner from the Forgejo box: the service, binary, `runner` user (and its cache data), Docker and its data, the weekly prune cron and the port-4000 ufw rule. It can be deleted from the repo once it has run. - **`setup-security.yml`** now reloads sshd when its settings change (before, they waited for a reboot). It also checks sshd's *effective* config, because an override in `sshd_config.d/` would win. - **README**: explains the two machines, how to bootstrap a new one (see below), and the runner's Linode Cloud Firewall. ## Reusing the runner's identity The new box registers with the **same `runner_secret`**. I checked `RegisterRunner` in Forgejo v17.0's `models/actions/forgejo.go`: the first 16 characters of the secret pick the runner's UUID, and re-registering an existing UUID updates its name, scope and labels in place. So the new box takes over the existing runner (renamed `runner-openneo-net`), and there's nothing to deregister. Both daemons can run at once during the move. Each just takes jobs as the same runner. ## Rollout Order matters, so that deploys never run from a machine `deploy-bot` doesn't accept, and the old runner never gets cut off mid-deploy: 1. ✅ **Create the Linode** (shared 2GB, `74.207.234.34`) with its **Cloud Firewall**: inbound Drop, except TCP 22; outbound Accept. 2. **DNS**: point `runner.openneo.net` at it (A `74.207.234.34`, AAAA `2600:3c02::2000:ecff:fe40:eada`). 3. **Bootstrap your user** (as `root` on the new box). Use the **same sudo password** as on code.openneo.net, because one `setup-actions.yml` run uses `sudo` on both machines. (A check before registration catches a mismatch.) ```bash adduser matchu usermod -aG sudo matchu cp -r ~/.ssh /home/matchu/ chown -R matchu:matchu /home/matchu/.ssh ``` 4. **impress**: run `bin/deploy:setup` from OpenNeo/impress#33's branch, then merge it. `deploy-bot` now accepts both machines. 5. **Set up the runner box**: ```bash ansible-playbook setup-all.yml --limit runner ``` OpenNeo/impress → Settings → Actions → Runners should show `runner-openneo-net`. This also checks that sshd really has root and password login off: a fresh Linode can override them in `/etc/ssh/sshd_config.d/`. 6. **Stop the old runner, at a quiet moment.** While both daemons run as the same runner, a passing job doesn't tell you which machine ran it. And stopping the old one cancels its current job after 5 minutes, which could interrupt a deploy. So, when the Actions tab shows nothing running, on code.openneo.net: ```bash sudo systemctl stop forgejo-runner ``` Then re-run a CI job (or push something) and check that it passes. It's now definitely on the new box. Its cache starts cold, so expect one more slow gem install. If it fails, `sudo systemctl start forgejo-runner` brings the old runner back. 7. **Remove the old runner** for good: ```bash ansible-playbook remove-colocated-runner.yml ``` Then reboot code.openneo.net, to clear the network rules and mounts Docker leaves behind. (If deleting `/var/lib/docker` fails with "Device or resource busy", reboot first, then re-run the playbook.) 8. **impress follow-up**: drop `45.79.220.14` from `deploy_bot_allowed_ips`, and run `bin/deploy:setup` again. ## Testing - `ansible-playbook --syntax-check` passes for `setup-all.yml`, `remove-colocated-runner.yml`, and impress's `deploy/setup.yml`. - `--list-hosts` confirms that machine/security target both boxes, Forgejo only code.openneo.net, and the runner only runner.openneo.net. - **Fresh-eyes review** (a separate agent) checked `delegate_to` under `--limit runner` with a local two-host test: the delegated task runs on the Forgejo box even though it's outside the limit, and `become_user`, `environment`, `chdir` and `stdin` apply there. It also confirmed that two daemons sharing one runner identity is safe (each task is handed out once), and that the ufw delete matches the original commented rule. Its rollout catches are fixed: steps 6–7, the sudo check, and the sshd reload and effective-config check. - **Not yet run against real machines.** 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014PWWd1c4UFRKce9t7XXiaP
CI jobs on the shared 1-vCPU, 2GB box were starving Forgejo and the runner's
own cache proxy: actions/cache restores timed out, so every run recompiled
native gems from scratch. A separate box also keeps jobs (which run in Docker,
as a root-equivalent user) away from Forgejo's data and secrets.

- inventory.cfg splits into `forgejo` (code.openneo.net) and `runner`
  (runner.openneo.net) groups.
- setup-machine.yml and setup-security.yml apply to both, and set each
  machine's hostname from the inventory. The swap file moves into
  setup-machine.yml, so both machines get one.
- setup-actions.yml targets the runner box, and delegates the registration
  command to the Forgejo box. The runner is named after its host.
- remove-colocated-runner.yml is a one-off, to remove the old runner and
  Docker from the Forgejo box once the new one works.
- The README explains the two machines, how to bootstrap a new one, and the
  runner's Linode Cloud Firewall.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PWWd1c4UFRKce9t7XXiaP
- Stop the old runner by hand at a quiet moment before verifying the new
  box: while both run as the same runner, a passing job doesn't show which
  machine ran it, and the teardown could otherwise cancel a deploy midway.
- Check that sudo works as git on the Forgejo box before registering the
  runner, since the registration's no_log would hide a password mismatch.
- setup-security.yml now reloads sshd when its config changes, and checks
  sshd's effective config, so an override in sshd_config.d can't quietly
  leave root or password login on.
- Bootstrap with cp + chown instead of rsync, which a minimal image may lack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PWWd1c4UFRKce9t7XXiaP
matchu merged commit bb1157eb3f into main 2026-11-16 21:36:11 -08:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
OpenNeo/openneo-code!5
No description provided.