Deploy automatically when changes land in main #18

Merged
matchu merged 3 commits from armin-bot/impress:claude/deploy-on-merge into main 2026-11-09 08:10:30 -08:00
Contributor

Deploys main automatically: when a change lands and its specs pass, CI runs the existing deploy/deploy.yml playbook as a new deploy-bot user. Deploys now run database migrations too, with an escape hatch for migrations that need a person.

⚠️ Don't merge until the setup checklist below is done. Merging triggers a deploy right away.

How it works

CI (.forgejo/workflows/test.yml → ci.yml): a new deploy job, needs: rspec, runs only on pushes to main.

  • Precompiles assets with placeholder secrets. A production asset build boots the app but never connects to the database, so CI never holds production's real secrets. The only secret is DEPLOY_SSH_KEY, and only the "Set up SSH" step sees it.
  • Runs on main queue instead of cancelling. That's the concurrency block. Without it, Forgejo cancels older runs on every push (ConfigureActionRunConcurrency in services/actions/workflows.go), which could kill a deploy mid-migration. PR runs still cancel stale ones.
  • Only the newest commit deploys. Each deploy first checks that its commit is still the head of main, and skips itself if not. So a queued older run, or a re-run from the Actions page, can't roll production back.
  • The server's host key is pinned in deploy/files/known_hosts, which is committed, so reviewers see any change.

Migrations (deploy/deploy.yml) run after bundle install and before the current symlink switches. If one fails, the deploy stops and the old version keeps serving.

  • Escape hatch: a migration file containing # deploy: run manually.
    • While one is pending, the deploy skips all migrations, ships the code anyway, and then ends red, with instructions for running them by hand.
    • Blocking rule: while a marked migration is pending, an unmarked pending migration can't run either. In that case the deploy stops before switching versions, because the new code might depend on it.
    • The check is bin/rails deploy:check_migrations, backed by lib/manual_migrations.rb. It uses the same Active Record helpers as db:migrate:status, and the spec covers both databases, so a Rails upgrade that changes those helpers fails loudly.
  • bin/deploy -e skip_migrations=yes skips migrations for a manual deploy. bin/deploy and bin/deploy:push now pass extra arguments through to Ansible.

Server (deploy/setup.yml): a deploy-bot user in impress-deployers, so it has the same permissions as a person deploying. Its key only works from code.openneo.net (from= in authorized_keys; job containers have no IPv6, so they always connect over IPv4), and port, agent and X11 forwarding are off.

Fixes to the existing playbook:

  • synchronize defaulted to verify_host: false, which skips host key checking on the rsync connection. It now verifies.
  • rsync now skips vendor/bundle, where CI keeps its gems. Without that, about 110MB of compiled gems would ship with every deploy.

Docs:

  • New: docs/database-migrations.md. The rules (compatible with the running code and Impress 2020, fast, safe to re-run, quick to give up on locks), plus the escape hatch.
  • Updated: docs/deployment-setup.md, which covers automatic deploys, how deploy-bot works, and rotating its key.
  • Updated: README's deployment section.

Setup checklist (before merging)

  • Generate the key with ssh-keygen -t ed25519 -N '' -C deploy-bot@code.openneo.net -f deploy-bot. Commit deploy-bot.pub as deploy/files/deploy-bot-ssh-key.pub; Armin can do this if you send it over.
  • Put the private key in OpenNeo/impress → Settings → Actions → Secrets as DEPLOY_SSH_KEY, then delete both key files locally.
  • Fill in deploy/files/known_hosts with ssh-keyscan -t ed25519 impress.openneo.net, after checking the fingerprint.
  • Run bin/deploy:setup to create deploy-bot.
  • Check which migrations are pending in production (bin/rails db:migrate:status as impress). The first automatic deploy will run them. 20261108120000_add_user_lookup_indexes may be one of them; it's written to be safe online (algorithm: :inplace, a 10-second lock wait).

Testing

  • Specs: 416 examples, 0 failures, including the new spec/lib/manual_migrations_spec.rb.

  • Asset build: precompile succeeds in a clean checkout with only the placeholder env.

  • Playbooks: --syntax-check passes for deploy.yml and setup.yml, on current Ansible and on Ansible 7 / core 2.14, which is what CI's Debian bookworm installs.

  • Migration gating: probed on Ansible 2.14, with all four outcomes correct:

    Scenario Result
    Nothing marked migrate, then switch
    Marked only switch, then red
    Marked + unmarked stop before switching
    skip_migrations switch, no migrations
  • Rake task: exercised end to end against real migration files.

  • rsync rules: checked that they ship public/assets (with the sprockets manifest), but not vendor/bundle, node_modules or .bundle.

  • Forgejo behavior: confirmed in source that it evaluates expressions in concurrency, and that synchronize + become becomes sudo -u impress rsync, which the existing sudoers rule allows.

  • Fresh-eyes review:

    • It found the missing libvips, the cancel-on-push default, the rollback-by-rerun, and the silently blocked migrations. All fixed in the second commit.
    • Things it noted that are fine as-is:
      • The deploy restores the same caches as the tests. That's safe, because the runner keeps PR cache writes isolated, and job containers can only reach the cache through its proxy, which signs every request.
      • Actions logs on this public repo are public. Nothing secret gets printed today.
      • Failed deploys count toward the 5 old versions the playbook keeps.

🤖 Generated with Claude Code

https://claude.ai/code/session_014PWWczf6WkuBDdUASWtRcZ

Deploys `main` automatically: when a change lands and its specs pass, CI runs the existing `deploy/deploy.yml` playbook as a new `deploy-bot` user. Deploys now run database migrations too, with an escape hatch for migrations that need a person. > ⚠️ **Don't merge until the setup checklist below is done.** Merging triggers a deploy right away. ## How it works **CI** (`.forgejo/workflows/test.yml` → `ci.yml`): a new `deploy` job, `needs: rspec`, runs only on pushes to `main`. - **Precompiles assets with placeholder secrets.** A production asset build boots the app but never connects to the database, so CI never holds production's real secrets. The only secret is `DEPLOY_SSH_KEY`, and only the "Set up SSH" step sees it. - **Runs on `main` queue instead of cancelling.** That's the `concurrency` block. Without it, Forgejo cancels older runs on every push (`ConfigureActionRunConcurrency` in `services/actions/workflows.go`), which could kill a deploy mid-migration. PR runs still cancel stale ones. - **Only the newest commit deploys.** Each deploy first checks that its commit is still the head of `main`, and skips itself if not. So a queued older run, or a re-run from the Actions page, can't roll production back. - **The server's host key is pinned** in `deploy/files/known_hosts`, which is committed, so reviewers see any change. **Migrations** (`deploy/deploy.yml`) run after `bundle install` and before the `current` symlink switches. If one fails, the deploy stops and the old version keeps serving. - **Escape hatch:** a migration file containing `# deploy: run manually`. - While one is pending, the deploy skips all migrations, ships the code anyway, and then **ends red**, with instructions for running them by hand. - **Blocking rule:** while a marked migration is pending, an unmarked pending migration can't run either. In that case the deploy **stops before switching versions**, because the new code might depend on it. - The check is `bin/rails deploy:check_migrations`, backed by `lib/manual_migrations.rb`. It uses the same Active Record helpers as `db:migrate:status`, and the spec covers both databases, so a Rails upgrade that changes those helpers fails loudly. - `bin/deploy -e skip_migrations=yes` skips migrations for a manual deploy. `bin/deploy` and `bin/deploy:push` now pass extra arguments through to Ansible. **Server** (`deploy/setup.yml`): a `deploy-bot` user in `impress-deployers`, so it has the same permissions as a person deploying. Its key only works from `code.openneo.net` (`from=` in authorized_keys; job containers have no IPv6, so they always connect over IPv4), and port, agent and X11 forwarding are off. **Fixes to the existing playbook:** - `synchronize` defaulted to `verify_host: false`, which skips host key checking on the rsync connection. It now verifies. - rsync now skips `vendor/bundle`, where CI keeps its gems. Without that, about 110MB of compiled gems would ship with every deploy. **Docs:** - New: `docs/database-migrations.md`. The rules (compatible with the running code and Impress 2020, fast, safe to re-run, quick to give up on locks), plus the escape hatch. - Updated: `docs/deployment-setup.md`, which covers automatic deploys, how `deploy-bot` works, and rotating its key. - Updated: README's deployment section. ## Setup checklist (before merging) - [x] Generate the key with `ssh-keygen -t ed25519 -N '' -C deploy-bot@code.openneo.net -f deploy-bot`. Commit `deploy-bot.pub` as `deploy/files/deploy-bot-ssh-key.pub`; Armin can do this if you send it over. - [x] Put the private key in OpenNeo/impress → Settings → Actions → Secrets as `DEPLOY_SSH_KEY`, then delete both key files locally. - [x] Fill in `deploy/files/known_hosts` with `ssh-keyscan -t ed25519 impress.openneo.net`, after checking the fingerprint. - [x] Run `bin/deploy:setup` to create `deploy-bot`. - [x] Check which migrations are pending in production (`bin/rails db:migrate:status` as `impress`). The first automatic deploy will run them. `20261108120000_add_user_lookup_indexes` may be one of them; it's written to be safe online (`algorithm: :inplace`, a 10-second lock wait). ## Testing - **Specs:** 416 examples, 0 failures, including the new `spec/lib/manual_migrations_spec.rb`. - **Asset build:** precompile succeeds in a clean checkout with only the placeholder env. - **Playbooks:** `--syntax-check` passes for `deploy.yml` and `setup.yml`, on current Ansible and on Ansible 7 / core 2.14, which is what CI's Debian bookworm installs. - **Migration gating:** probed on Ansible 2.14, with all four outcomes correct: | Scenario | Result | |---|---| | Nothing marked | migrate, then switch | | Marked only | switch, then red | | Marked + unmarked | stop before switching | | `skip_migrations` | switch, no migrations | - **Rake task:** exercised end to end against real migration files. - **rsync rules:** checked that they ship `public/assets` (with the sprockets manifest), but not `vendor/bundle`, `node_modules` or `.bundle`. - **Forgejo behavior:** confirmed in source that it evaluates expressions in `concurrency`, and that `synchronize` + become becomes `sudo -u impress rsync`, which the existing sudoers rule allows. - **Fresh-eyes review:** - It found the missing libvips, the cancel-on-push default, the rollback-by-rerun, and the silently blocked migrations. All fixed in the second commit. - Things it noted that are fine as-is: - The deploy restores the same caches as the tests. That's safe, because the runner keeps PR cache writes isolated, and job containers can only reach the cache through its proxy, which signs every request. - Actions logs on this public repo are public. Nothing secret gets printed today. - Failed deploys count toward the 5 old versions the playbook keeps. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014PWWczf6WkuBDdUASWtRcZ
Adds a `deploy` job to CI (renamed from test.yml to ci.yml), which runs
on pushes to main once the specs pass. It precompiles assets with
placeholder secrets, then runs the existing deploy/deploy.yml playbook as
a new `deploy-bot` user, whose SSH key is a Forgejo secret.

Deploys now run database migrations, before switching to the new code.
If a pending migration is marked `# deploy: run manually`, the deploy
skips migrations, ships the code anyway, and warns in its log. The
`skip_migrations` var skips them for manual deploys. See the new
docs/database-migrations.md for the rules migrations need to follow.

Also:
- deploy/setup.yml creates `deploy-bot`, in `impress-deployers`, with
  its key only accepted from the Forgejo runner's machine.
- The deploy's rsync skips vendor/bundle, where CI keeps its gems, and
  now checks the server's SSH host key (synchronize skipped it by
  default).
- bin/deploy and bin/deploy:push pass extra arguments to Ansible.

Still needs deploy-bot's public key and the server's host key before
it can run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PWWczf6WkuBDdUASWtRcZ
Harden automatic deploys after review
All checks were successful
CI / rspec (pull_request) Successful in 2m18s
CI / deploy (pull_request) Has been skipped
b5bfd24829
- Install libvips in the deploy job. Booting the app to precompile
  assets loads ruby-vips, which fails without it.
- Add a `concurrency` block. Without one, Forgejo cancels older runs on
  every push, which could stop a deploy mid-migration. Now runs on main
  wait for each other, and PR runs still cancel stale ones.
- Before deploying, check that the commit is still the newest on main,
  and skip the deploy if not. Otherwise a queued older run, or a re-run
  from the Actions page, would roll production back.
- While a migration marked to run by hand is pending, every other pending
  migration is blocked too, since deploys skip them all. Now the deploy
  stops before switching versions if any unmarked migration is blocked,
  since its code might depend on it. And a deploy that skips marked
  migrations ends red, instead of only printing a warning, so someone
  notices. (`deploy:manual_migrations` is now `deploy:check_migrations`.)
- Docs: the 30-minute job limit, and the new blocking behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PWWczf6WkuBDdUASWtRcZ
matchu approved these changes 2026-11-09 08:02:13 -08:00
matchu force-pushed claude/deploy-on-merge from b5bfd24829
All checks were successful
CI / rspec (pull_request) Successful in 2m18s
CI / deploy (pull_request) Has been skipped
to 4055084d20
All checks were successful
CI / rspec (pull_request) Successful in 1m40s
CI / deploy (pull_request) Has been skipped
2026-11-09 08:02:28 -08:00
Compare
Add deploy-bot's public key and the server's SSH host key
All checks were successful
CI / rspec (pull_request) Successful in 1m40s
CI / deploy (pull_request) Has been skipped
100511b6e5
deploy-bot's key: SHA256:8jZ7drByLpCD4NJiX9bnIqI+8u+acRJyggkp67kidkg
impress.openneo.net's ed25519 host key:
SHA256:xUtp7jiH3RLAX4l8mebBxrfj9Y7x0Mzyn5noPPQkJZw

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PWWczf6WkuBDdUASWtRcZ
matchu merged commit 02538fc55f into main 2026-11-09 08:10:30 -08:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
OpenNeo/impress!18
No description provided.