Raise the production memory cap, and use jemalloc #19

Merged
matchu merged 1 commit from armin-bot/impress:claude/memory-cap into main 2026-11-09 07:48:15 -08:00
Contributor

We set the 1.2GB MemoryMax in Oct 2023 to work around an apparent leak, back when production was one Puma process. Since Jan 2024, Falcon forks one worker per core with no preloading. On our 6-core box that's 6 × ~225MB ≈ 1.35GB of healthy steady-state usage, already over the cap, which explains the ~3 kills a week.

What I measured

I ran the app locally in production mode under Falcon, against the public data dump plus synthetic users, closets and outfits. I replayed a broad mix of request types and logged each worker's memory and Ruby heap stats.

  • No leak. In a 56k-request soak, every worker plateaued at 218–231MB from minute 5 to minute 22. Heap pages, old objects and malloc totals were all flat.
  • Spikes are real. One worker serving 6 concurrent 25k-hanger closet pages went from 241MB → 729MB peak, and settled at 366MB after 600 more normal requests (glibc keeps much of the high-water mark). The closet page is unpaginated, and Falcon has no per-worker concurrency cap.
  • jemalloc gives more back. In the same spike test, it settled at 298MB instead of 366MB (one run each, so treat it as suggestive).
  • YJIT adds ~75MB per worker (≈300MB each), so 6 workers ≈ 1.8GB.

Changes

  • falcon.rb reads FALCON_COUNT (unset → Falcon's default of one per core, so dev is unchanged).
  • The service file sets FALCON_COUNT=6 right next to MemoryMax, since they're coupled, and raises the cap to 4G: room for 6 YJIT workers plus spike headroom, on a 16GB box shared with MariaDB. I rewrote the comment to explain the history.
  • The service file preloads libjemalloc.so.2 (bare name, so it's arch-agnostic); setup.yml installs libjemalloc2. If the library is missing, the dynamic linker warns and falls back to glibc.

Testing

  • Locally, FALCON_COUNT=2 with jemalloc preloaded gave 2 workers, with jemalloc mapped into each and pages serving 200s. With the variable unset, it fell back to 4 workers (one per core here).
  • setup.yml still parses as YAML. I haven't run the playbook itself.

Deploying

This needs bin/deploy:setup (it changes the service file and installs a package), not just bin/deploy. Afterward, it's worth checking systemctl status impress and that grep jemalloc /proc/<worker pid>/maps shows up. If the cap ever gets hit now, that's a real signal worth investigating.

🤖 Generated with Claude Code

https://claude.ai/code/session_014PWWczFNVoGD1tQziaMfGe

We set the 1.2GB `MemoryMax` in Oct 2023 to work around an apparent leak, back when production was **one Puma process**. Since Jan 2024, Falcon forks one worker per core with no preloading. On our 6-core box that's **6 × ~225MB ≈ 1.35GB of healthy steady-state usage**, already over the cap, which explains the ~3 kills a week. ## What I measured I ran the app locally in production mode under Falcon, against the public data dump plus synthetic users, closets and outfits. I replayed a broad mix of request types and logged each worker's memory and Ruby heap stats. - **No leak.** In a 56k-request soak, every worker plateaued at 218–231MB from minute 5 to minute 22. Heap pages, old objects and malloc totals were all flat. - **Spikes are real.** One worker serving 6 concurrent 25k-hanger closet pages went from **241MB → 729MB peak**, and settled at **366MB** after 600 more normal requests (glibc keeps much of the high-water mark). The closet page is unpaginated, and Falcon has no per-worker concurrency cap. - **jemalloc gives more back.** In the same spike test, it settled at **298MB** instead of 366MB (one run each, so treat it as suggestive). - **YJIT adds ~75MB per worker** (≈300MB each), so 6 workers ≈ 1.8GB. ## Changes - `falcon.rb` reads `FALCON_COUNT` (unset → Falcon's default of one per core, so dev is unchanged). - The service file sets `FALCON_COUNT=6` right next to `MemoryMax`, since they're coupled, and raises the cap to **4G**: room for 6 YJIT workers plus spike headroom, on a 16GB box shared with MariaDB. I rewrote the comment to explain the history. - The service file preloads `libjemalloc.so.2` (bare name, so it's arch-agnostic); `setup.yml` installs `libjemalloc2`. If the library is missing, the dynamic linker warns and falls back to glibc. ## Testing - Locally, `FALCON_COUNT=2` with jemalloc preloaded gave 2 workers, with jemalloc mapped into each and pages serving 200s. With the variable unset, it fell back to 4 workers (one per core here). - `setup.yml` still parses as YAML. I haven't run the playbook itself. ## Deploying This needs `bin/deploy:setup` (it changes the service file and installs a package), not just `bin/deploy`. Afterward, it's worth checking `systemctl status impress` and that `grep jemalloc /proc/<worker pid>/maps` shows up. If the cap ever gets hit now, that's a real signal worth investigating. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014PWWczFNVoGD1tQziaMfGe
Raise the production memory cap, and use jemalloc
All checks were successful
Tests / rspec (pull_request) Successful in 2m33s
bc5d37c45a
We set a 1.2GB `MemoryMax` in 2023, to work around what looked like a
memory leak, back when production was a single Puma process. Since then,
Falcon started forking one worker per core: on our 6-core box, that's
~6 × 225MB ≈ 1.35GB of *healthy* steady-state usage, already above the
cap. That explains the few kills per week we still see.

A long production-mode load test (56k requests against the public data
dump) showed no leak: each worker's memory stayed flat once warm. But it
did find a real spike mechanism: big unpaginated pages (large closets,
the sitemap) served concurrently on one worker pushed it from ~240MB to
~730MB, and glibc only gave part of that back afterward.

So in this change:

- Pin Falcon's worker count (`FALCON_COUNT=6`) in the service file, right
  next to the memory cap, since the two need to be chosen together.
  (`falcon.rb` reads it; unset, Falcon still defaults to one per core.)
- Raise `MemoryMax` to 4GB: room for 6 workers with YJIT (~300MB each)
  plus spike headroom, on a 16GB box that also runs MariaDB.
- Preload jemalloc, which gave back more memory after the spike test
  than glibc did (settling at ~300MB vs ~365MB). If the library is
  missing, the dynamic linker warns and falls back to glibc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PWWczFNVoGD1tQziaMfGe
matchu merged commit 2deba0146b into main 2026-11-09 07:48:15 -08:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
OpenNeo/impress!19
No description provided.