impress/deploy/files/impress.service

[Unit]
Description=Dress to Impress webapp

[Service]
User=impress
Restart=always
WorkingDirectory=/srv/impress/current
ExecStart=/opt/ruby-3.3.4/bin/bundle exec falcon host
Environment="RAILS_ENV=production"
; Set EXECJS_RUNTIME to save us from needing to install Node
Environment="EXECJS_RUNTIME=Disabled"
EnvironmentFile=/srv/impress/shared/production.env

; Kill the app at 1.2GB of RAM.
; NOTE: It's generally recommended to pair this with a `MemoryHigh` setting, to
; try throttling memory usage before killing altogether. But at time of writing,
; our app has a memory leak that causes it to always keep growing - and then the
; throttling *successfully* keeps it within the RAM limits, just running very
; slowly. Instead, let's always reboot it when it gets too high, and hopefully
; find and fix that leak sometime!
MemoryMax=1.2G

; Some security directives, adapted from Akkoma's service file, they seem like sensible defaults!
; Use private /tmp and /var/tmp folders inside a new file system namespace, which are discarded after the process stops.
PrivateTmp=true
; The /home, /root, and /run/user folders can not be accessed by this service anymore. If your Akkoma user has its home folder in one of the restricted places, or use one of these folders as its working directory, you have to set this to false.
ProtectHome=true
; Mount /usr, /boot, and /etc as read-only for processes invoked by this service.
ProtectSystem=full
; Sets up a new /dev mount for the process and only adds API pseudo devices like /dev/null, /dev/zero or /dev/random but not physical devices.
PrivateDevices=true
; Ensures that the service process and all its children can never gain new privileges through execve().
NoNewPrivileges=true
; Drops the sysadmin capability from the daemon.
CapabilityBoundingSet=~CAP_SYS_ADMIN

[Install]
WantedBy=multi-user.target
Extract some files out of the deploy script Okay, there's enough going on in here now that I don't like it inline anymore! Welcome to `files`! 2023-10-25 15:40:43 -07:00			`[Unit]`
			`Description=Dress to Impress webapp`

			`[Service]`
			`User=impress`
			`Restart=always`
			`WorkingDirectory=/srv/impress/current`
Upgrade to Ruby 3.3.4 in production Oh right, I did a dev-side version of this, and forgot prod needs it too! (Maybe a bit silly to bother for a patch-level but whatever!) 2024-08-31 12:49:50 -07:00			`ExecStart=/opt/ruby-3.3.4/bin/bundle exec falcon host`
Extract some files out of the deploy script Okay, there's enough going on in here now that I don't like it inline anymore! Welcome to `files`! 2023-10-25 15:40:43 -07:00			`Environment="RAILS_ENV=production"`
			`; Set EXECJS_RUNTIME to save us from needing to install Node`
			`Environment="EXECJS_RUNTIME=Disabled"`
			`EnvironmentFile=/srv/impress/shared/production.env`

Kill the app at high RAM, instead of trying to throttle it first Well, sitting at the `MemoryHigh` limit still grinds the app to a halt anyway, lmao. I guess it's a feature designed for well-behaved processes and not for outright leaking ones? Let's try just having systemd basically reset the app regularly when the RAM hits a certain threshold. I think that's what this config will do, we'll find out! 2023-10-27 17:03:08 -07:00			`; Kill the app at 1.2GB of RAM.`
			; NOTE: It's generally recommended to pair this with a `MemoryHigh` setting, to
			`; try throttling memory usage before killing altogether. But at time of writing,`
			`; our app has a memory leak that causes it to always keep growing - and then the`
			`; throttling successfully keeps it within the RAM limits, just running very`
			`; slowly. Instead, let's always reboot it when it gets too high, and hopefully`
			`; find and fix that leak sometime!`
Tighten the RAM limit bounds on the production impress service Lol ok, as I had kinda predicted, the memory bounds I set last time were not tight enough, and it stalled out again! (It was at 75% and fully just not working.) Let's try this tighter bound instead! 2023-10-27 10:32:33 -07:00			`MemoryMax=1.2G`
Add RAM constraints to impress service on in production I just restarted the impress app in production! First I logged in to see why it wasn't responding, and I saw that there was almost no free RAM left, and that the Rails app had grown to eat it all up! So in this change, we set a memory limit: if the impress app is taking up more than 75% of the machine's RAM, systemctl will try to shrink it down; if it can't, then it will kill the app at 80%. I'm not totally sure whether these bounds are tight enough? I didn't look closely enough at the numbers to see what the app's actual usage was according to systemctl at the time (`sudo systemctl status impress`), so my hope is this is enough. But if we run into a memory leak crash like that again, because it turns out even existing at 75% RAM freezes the machine when running alongside its other processes, we can decrease these numbers! I also don't know the nature of the memory leak, and that could be worth investigating—the app pretty cleanly fits into ~500–600MB when it starts up, but then does seem to slowly but steadily grow. If it could be kept at that size, it's possible we could downgrade the server and save some costs—but that's a question for another day, since making sure we handle memory leaks when they do happen is a more important robustness fix! 2023-10-26 13:51:47 -07:00
Extract some files out of the deploy script Okay, there's enough going on in here now that I don't like it inline anymore! Welcome to `files`! 2023-10-25 15:40:43 -07:00			`; Some security directives, adapted from Akkoma's service file, they seem like sensible defaults!`
			`; Use private /tmp and /var/tmp folders inside a new file system namespace, which are discarded after the process stops.`
			`PrivateTmp=true`
			`; The /home, /root, and /run/user folders can not be accessed by this service anymore. If your Akkoma user has its home folder in one of the restricted places, or use one of these folders as its working directory, you have to set this to false.`
			`ProtectHome=true`
			`; Mount /usr, /boot, and /etc as read-only for processes invoked by this service.`
			`ProtectSystem=full`
			`; Sets up a new /dev mount for the process and only adds API pseudo devices like /dev/null, /dev/zero or /dev/random but not physical devices.`
			`PrivateDevices=true`
			`; Ensures that the service process and all its children can never gain new privileges through execve().`
			`NoNewPrivileges=true`
			`; Drops the sysadmin capability from the daemon.`
			`CapabilityBoundingSet=~CAP_SYS_ADMIN`

			`[Install]`
			`WantedBy=multi-user.target`