Install rustc so Ruby is built with YJIT #14
Loading…
Reference in a new issue
No description provided.
Delete branch "armin-bot/impress:claude/amazing-ritchie-2us66j"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Production Ruby has no YJIT:
ruby-buildonly compiles it in whenrustcis installed at build time, anddeploy/setup.ymlnever installed it. Rails already enables YJIT in production on its own (config.load_defaults 8.1setsconfig.yjit = !Rails.env.local?), so no app config change is needed: I booted the app in production mode locally and gotRubyVM::YJIT.enabled? == true(andfalsein development).This PR:
rustcindeploy/setup.yml, before the "Install Ruby" task. The server runs Ubuntu 22.04, where therustcapt package is 1.58.1 in the release pocket and 1.75.0 fromjammy-updates/jammy-security(Ruby 3.4 needs >= 1.58), so plainaptworks. I built Ruby 3.4.5 with rustc 1.75.0 and nocargoonPATH(which is what the apt package gives you) and it came out with YJIT. I did not try 1.58.1, which only just meets the minimum; step 2 of the procedure prints which version you actually get./opt/ruby-<version>already exists, so addingrustcalone won't rebuild production's current Ruby.This PR does not change production by itself. The existing Ruby needs a one-time manual rebuild; the procedure is below. It's in this description rather than in
docs/, since it only needs to be run once.Memory and ordering (see #19)
#19 changed the memory picture since I first wrote this, so this supersedes my earlier cap arithmetic:
FALCON_COUNT=6) underMemoryMax=4G, and measured YJIT at about 75MB per worker (≈1.8GB for 6 workers). That fits under 4G with room for spikes, so noMemoryMaxchange or lower--yjit-mem-sizeis needed here. (I had suggested both, based on the old 1.2G cap. That cap turned out to be below healthy steady-state usage, not a leak symptom.)bin/deploy:setup(it changes the service file and installslibjemalloc2), not just a deploy.main(#18) and restart the service, so don't merge anything during the swap window below.Testing
RbConfig::CONFIGdifferences are theYJIT_*entries andconfigure_args; the extension ABI (3.4.0,x86_64-linux, shared libruby) is identical.bcrypt,msgpack,nio4r) compiled by the no-YJIT build load and run under the YJIT build with YJIT enabled.-noyjitdirectory makes the chain stop without nesting directories). A renamed built Ruby does not work (librubyRUNPATH and shebangs break), which is why the procedure uses a symlink.--check.systemctloutput, the cron jobs.One-time procedure for the production box: swap in a YJIT-enabled Ruby 3.4.5
YJIT is Ruby's JIT compiler. Rails turns it on by itself in production
(
config.load_defaults 8.1setsconfig.yjit = !Rails.env.local?), if theRuby was built with it. ruby-build only compiles YJIT in when
rustcisinstalled at build time, and the build succeeds silently without it.
setup.ymlnow installsrustcbefore building Ruby, so new serversand future Ruby upgrades get YJIT automatically. But the setup playbook skips
"Install Ruby" when
/opt/ruby-<version>already exists, so a server whose Rubywas built before this change keeps its YJIT-less Ruby until you rebuild it by
hand. Running
bin/deploy:setupinstallsrustcand prints a warning if thecurrent Ruby has no YJIT.
You can check any Ruby yourself:
One-time procedure: swap in a YJIT-enabled 3.4.5
Same Ruby version, so the gems in
/srv/impress/shared/bundleare reused as-is(see "Why the gems are safe to reuse" below), and
the downtime is just a service restart (under a minute, probably, counting the
Rails boot). This is not the Ruby-upgrade dance described in
deploy/deploy.yml, which is for changing versions.Run these on the server as a user with
sudo. Everything up to step 4 leavesthe live app untouched. Note that only
sudo -u impress ...andsudo systemctl {status,start,stop,restart} impressare passwordless (see thesudoers rules in
setup.yml); everything else below needs your own sudopassword.
1. Preflight. Record a baseline to compare against later, and check there's
room to compile:
nice(used in step 3) only protects CPU, not memory. Compiling Ruby withYJIT, plus the extra production Rails boot in the same step, runs next to the
live service, and if the box runs short of RAM the OOM killer could take out
falcon. We haven't measured the build's peak memory, so as a rough guess:
if the
availablecolumn offree -mis under about 1.5 GB, wait for a quietwindow or add temporary swap first.
2. Install rustc, either with this or by running
bin/deploy:setupfromyour machine. (Prefer the
apt-getroute: the full setup playbook alsore-runs the MariaDB, certbot and service-file tasks, and can restart the app.)
(On Ubuntu 22.04 you should get 1.75.0 from
jammy-updates, which is enabledby default. I only tested the YJIT build with 1.75; if you somehow only get
1.58.1, it meets the stated minimum but is untested.)
3. Build into a temporary prefix. The app keeps running meanwhile;
nicekeeps the build from starving it. This takes a few minutes (longer on a small
box):
Check that it has YJIT, and that it matches the live Ruby in everything that
affects compiled gems (the two lines printed by each loop should be identical;
the only differences expected in
configure_argsare the prefix and nothingelse, since YJIT is on by default):
Then try the new Ruby against the real app and the real production gems,
without touching the running service. This boots Rails in production mode
and loads every native extension:
Expect a description containing
+YJIT, thentrue. If anything looks wrong,stop here:
sudo rm -rf /opt/ruby-3.4.5-yjitand nothing has changed.4. Swap. Downtime starts here. Deploys run automatically when anything lands on
main(#18) and restart the service, so don't merge anything now, and check the Actions page for a deploy in progress. First refresh your sudo password cache, soyou aren't prompted while the site is down (the earlier
sudocalls may haveexpired). The
&&chain matters: if a step fails, the later ones are skipped,and the final
startjust brings the old, unchanged Ruby back up. If thathappens, work out why before retrying (for instance, a leftover
/opt/ruby-3.4.5-noyjitfrom an earlier attempt makes themvfail).The cron jobs (
items:auto_modelevery 2 minutes,neopets:importevery 10)also run through
/opt/ruby-3.4.5, so a run that fires in the instant betweenthe
mvand thelnfails, and the next one is fine. After the swap they runwith YJIT too; they run under cron rather than the
impressservice, soMemoryMaxdoesn't cover them.Note that
/opt/ruby-3.4.5becomes a symlink to the real new install, ratherthan the new install being renamed into place. That's deliberate: a built Ruby
has its own prefix baked in (the
rubybinary's library path, and the shebangsof
bundle,gem, etc.), so a renamed Ruby fails to start. Everything thatreferences
/opt/ruby-3.4.5(the systemd unit,/etc/profile.d, the cronjobs, the shebangs of gem binstubs in the shared bundle, and the Ansible
playbooks) keeps working unchanged through the symlink. Leave
/opt/ruby-3.4.5-yjitin place.5. Verify.
The
rails runnerone runs the same way the cron jobs do. It should print+YJITandtrue. "active (running)" only means the process started, notthat it's serving yet, so give it a few seconds after the start (Rails boot)
before the
curlshould showHTTP/2 200. Then click around the site a bit.Rollback (if the app misbehaves, or memory gets out of hand, see below):
(
rmwithout-ronly removes the symlink, and refuses if/opt/ruby-3.4.5is somehow a real directory. Don't add a trailing slash to that path.)
Once you're confident (a week or so),
sudo rm -rf /opt/ruby-3.4.5-noyjit.Why the gems are safe to reuse
Compiled gems live in
/srv/impress/shared/bundle/ruby/3.4.0/extensions/x86_64-linux/3.4.0/,keyed by Ruby ABI version and platform, which don't change when YJIT is added.
Comparing
RbConfig::CONFIGbetween a--disable-yjitand a default build of3.4.5, the only differences were the
YJIT_*entries andconfigure_args. Theextensions' ELF headers say
NEEDED libruby.so.3.4, which is satisfied by thelibrubythe running process already loaded, whichever install it came from.We tested this directly: gems with C extensions (
bcrypt,msgpack,nio4r)compiled by a YJIT-less 3.4.5 loaded and ran under the YJIT build, with YJIT
enabled and code compiled.
The one thing that would break this is building the new Ruby with different
configureoptions (e.g. a differentruby-buildversion orRUBY_CONFIGURE_OPTS).That's what the comparison in step 3 is for.
Memory: what to watch afterwards
With #19 deployed, the service runs 6 workers under
MemoryMax=4G, and YJIT is expected to add about 75MB per worker (≈450MB total, measured in #19, not on the live box). Hitting the cap is now a signal worth investigating, not routine. For the first few days:It's also worth watching response times, since that's the point. YJIT needs a warm-up period after every restart, so numbers are worse right after one.
If something goes wrong, the rollback above takes seconds. (If you ever need to cap YJIT specifically,
Environment="RUBYOPT=--yjit-mem-size=<MiB>"in the service file does it. Ruby 3.4'sRubyVM::YJIT.enable, which Rails calls, doesn't accept amem_size:option.)🤖 Generated with Claude Code
https://claude.ai/code/session_014PWWd2aRXHH83fsXVzrDQ2
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.