Add RAM constraints to impress service on in production

I just restarted the impress app in production! First I logged in to see
why it wasn't responding, and I saw that there was almost no free RAM
left, and that the Rails app had grown to eat it all up!

So in this change, we set a memory limit: if the impress app is taking
up more than 75% of the machine's RAM, systemctl will try to shrink it
down; if it can't, then it will kill the app at 80%.

I'm not totally sure whether these bounds are tight enough? I didn't
look closely enough at the numbers to see what the app's actual usage
was according to systemctl at the time (`sudo systemctl status
impress`), so my hope is this is enough. But if we run into a memory
leak crash like that again, because it turns out even existing at 75%
RAM freezes the machine when running alongside its other processes, we
can decrease these numbers!

I also don't know the nature of the memory leak, and that could be worth
investigating—the app pretty cleanly fits into ~500–600MB when it starts
up, but then does seem to slowly but steadily grow. If it could be kept
at that size, it's possible we could downgrade the server and save some
costs—but that's a question for another day, since making sure we handle
memory leaks when they *do* happen is a more important robustness fix!
This commit is contained in:
Emi Matchu 2023-10-26 13:51:47 -07:00
parent 805521c289
commit 271d477110

View file

@ -11,6 +11,12 @@ Environment="RAILS_ENV=production"
Environment="EXECJS_RUNTIME=Disabled" Environment="EXECJS_RUNTIME=Disabled"
EnvironmentFile=/srv/impress/shared/production.env EnvironmentFile=/srv/impress/shared/production.env
; Try not to go over 75% of the server's RAM, and kill the app at 80%.
; NOTE: This assumes this is the main thing running on the box! If it were
; running alongside e.g. a database, you'd want to constrain it differently.
MemoryHigh=75%
MemoryMax=80%
; Some security directives, adapted from Akkoma's service file, they seem like sensible defaults! ; Some security directives, adapted from Akkoma's service file, they seem like sensible defaults!
; Use private /tmp and /var/tmp folders inside a new file system namespace, which are discarded after the process stops. ; Use private /tmp and /var/tmp folders inside a new file system namespace, which are discarded after the process stops.
PrivateTmp=true PrivateTmp=true