Fix intermittent "Trying to reuse unacquired resource" on HTTP/2 fetches #22

Merged
matchu merged 2 commits from armin-bot/impress:claude/intelligent-goodall-ucr48y into main 2026-11-09 08:18:25 -08:00
Contributor

Fixes the intermittent 500 from the production load test:

RuntimeError (Trying to reuse unacquired resource: #<Async::HTTP::Protocol::HTTP2::Client 0 active streams>!)

raised from DTIRequests.get while fetching manifests from images.neopets.com (7 times in ~56k requests).

Root cause

It's a bug in async-pool 0.11.1 (the connection pool under async-http), which upstream has already fixed. The fix here is a lockfile bump to async-pool 0.12.0. async-http 0.89's ~> 0.9 constraint already allows it, so nothing else moves.

The path:

  1. Concurrent Falcon requests (fibers) share Async::HTTP::Internet.instance, so their manifest fetches multiplex streams over one shared HTTP/2 connection to images.neopets.com.
  2. Fiber F has a stream in flight on that connection. Meanwhile, the server hangs up its end of the socket (a FIN arrives), and the client's background reader hasn't processed that yet.
  3. Fiber G asks the pool for a connection. acquire_or_create_resource checks viable?, which for HTTP/2 peeks the socket, sees the FIN, and returns false. In 0.11.1 the pool then retires the connection on the spot, even though F is still using it: it deletes the connection from @resources, then starts closing it.
  4. F's stream finishes and releases the connection. The close hasn't completed yet, so reusable? is still true, and reuse looks it up in @resources, finds nothing, and raises. ("0 active streams": F's own stream had already finished.)

async-pool 0.11.2 made reuse tolerate an already-retired resource. 0.12.0 goes further: it stops handing out a non-viable resource, but only retires it once its last user releases it, which removes the race outright.

The brief's first theory (the pool's "gardener" task is parented to a request, and gets stopped and drains the pool when that request ends) doesn't hold up. The gardener is transient, and async's stop_children skips transient tasks. When its parent finishes, it's reparented upward rather than stopped (async 2.35 node.rb). So I left DTIRequests and the shared Internet instance alone: cross-request connection reuse wasn't the problem, it just made the race reachable.

What's confirmed: the error and the code path, both variants of it:

  • "1 active streams": the new spec makes the retire-while-in-use happen against a real local HTTP/2 server. Closing the connection cuts off the in-flight stream, which then releases it. It raises Trying to reuse unacquired resource on 0.11.1 and passes on 0.12.0.
  • "0 active streams" (the production message): a review script reproduced this too. The other fiber's stream has already finished, and it releases the connection while the retiring fiber is still partway through closing it (close yields while stopping the reader task). That script drives pool internals directly, so it isn't checked in as a spec. 0.12.0 fixes both.

What's inferred: the trigger in production, i.e. why images.neopets.com connections stop looking viable. My guess is the server or CDN recycling connections (e.g. a per-connection request cap like nginx's keepalive_requests, or an idle timeout). I couldn't make that happen organically over localhost: the reader always processes the hang-up before another fiber checks the connection. A narrow, latency-dependent window fits a 7-in-56k rate. Even so, the spec stubs viable? rather than reproducing the hang-up for real.

Changes

  • Gemfile.lock / vendor/cache: async-pool 0.11.1 → 0.12.0.
  • spec/lib/http2_connection_pool_spec.rb: regression spec. It starts a local HTTP/2 server, puts one slow response mid-body, marks its connection non-viable, and checks that a second request gets a fresh connection while the first one finishes cleanly.

Testing

  • bundle exec rspec: 407 examples, 0 failures (2 pending, as before).
  • The new spec fails on the old lockfile with the production error, and passes on the new one.

🤖 Generated with Claude Code

https://claude.ai/code/session_014PWWcywsHc4nxSKJGptZa6

Fixes the intermittent 500 from the production load test: ``` RuntimeError (Trying to reuse unacquired resource: #<Async::HTTP::Protocol::HTTP2::Client 0 active streams>!) ``` raised from `DTIRequests.get` while fetching manifests from images.neopets.com (7 times in ~56k requests). ## Root cause It's a bug in `async-pool` 0.11.1 (the connection pool under async-http), which upstream has already fixed. The fix here is a lockfile bump to **async-pool 0.12.0**. async-http 0.89's `~> 0.9` constraint already allows it, so nothing else moves. The path: 1. Concurrent Falcon requests (fibers) share `Async::HTTP::Internet.instance`, so their manifest fetches multiplex streams over **one shared HTTP/2 connection** to images.neopets.com. 2. Fiber F has a stream in flight on that connection. Meanwhile, the server hangs up its end of the socket (a FIN arrives), and the client's background reader hasn't processed that yet. 3. Fiber G asks the pool for a connection. `acquire_or_create_resource` checks `viable?`, which for HTTP/2 peeks the socket, sees the FIN, and returns false. In 0.11.1 the pool then **retires the connection on the spot, even though F is still using it**: it deletes the connection from `@resources`, then starts closing it. 4. F's stream finishes and releases the connection. The close hasn't completed yet, so `reusable?` is still true, and `reuse` looks it up in `@resources`, finds nothing, and raises. ("0 active streams": F's own stream had already finished.) async-pool 0.11.2 made `reuse` tolerate an already-retired resource. 0.12.0 goes further: it stops handing out a non-viable resource, but **only retires it once its last user releases it**, which removes the race outright. The brief's first theory (the pool's "gardener" task is parented to a request, and gets stopped and drains the pool when that request ends) doesn't hold up. The gardener is `transient`, and async's `stop_children` skips transient tasks. When its parent finishes, it's reparented upward rather than stopped (async 2.35 `node.rb`). So I left `DTIRequests` and the shared `Internet` instance alone: cross-request connection reuse wasn't the problem, it just made the race reachable. **What's confirmed:** the error and the code path, both variants of it: - **"1 active streams":** the new spec makes the retire-while-in-use happen against a real local HTTP/2 server. Closing the connection cuts off the in-flight stream, which then releases it. It raises `Trying to reuse unacquired resource` on 0.11.1 and passes on 0.12.0. - **"0 active streams"** (the production message): a review script reproduced this too. The other fiber's stream has already finished, and it releases the connection while the retiring fiber is still partway through closing it (`close` yields while stopping the reader task). That script drives pool internals directly, so it isn't checked in as a spec. 0.12.0 fixes both. **What's inferred:** the trigger in production, i.e. *why* images.neopets.com connections stop looking viable. My guess is the server or CDN recycling connections (e.g. a per-connection request cap like nginx's `keepalive_requests`, or an idle timeout). I couldn't make that happen organically over localhost: the reader always processes the hang-up before another fiber checks the connection. A narrow, latency-dependent window fits a 7-in-56k rate. Even so, the spec stubs `viable?` rather than reproducing the hang-up for real. ## Changes - `Gemfile.lock` / `vendor/cache`: async-pool 0.11.1 → 0.12.0. - `spec/lib/http2_connection_pool_spec.rb`: regression spec. It starts a local HTTP/2 server, puts one slow response mid-body, marks its connection non-viable, and checks that a second request gets a fresh connection while the first one finishes cleanly. ## Testing - `bundle exec rspec`: 407 examples, 0 failures (2 pending, as before). - The new spec fails on the old lockfile with the production error, and passes on the new one. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014PWWcywsHc4nxSKJGptZa6
Bump async-pool 0.11.1 -> 0.12.0. In 0.11.1, when a request found the
shared HTTP/2 connection to images.neopets.com no longer viable (e.g. the
server had hung up), the pool retired it immediately, even with another
fiber still mid-response on it. When that fiber's stream then released
the connection, the pool no longer knew about it and raised. 0.12.0 only
retires a non-viable connection once its last user releases it.

Add a regression spec that reproduces the error against a local HTTP/2
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PWWcywsHc4nxSKJGptZa6
Tighten the HTTP/2 pool spec: no WebMock toggling, add timeouts
All checks were successful
Tests / rspec (pull_request) Successful in 2m5s
a4207b30c9
Use WebMock's saved original client class instead of disabling and
re-enabling its async-http adapter, which could leave the adapter on in
runs that never loaded webmock/rspec. Bound the example and its cleanup
with timeouts so a regression fails fast instead of hanging, and say
which variant of the error the spec reproduces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PWWcywsHc4nxSKJGptZa6
matchu merged commit 91bcdf135e into main 2026-11-09 08:18:25 -08:00
matchu referenced this pull request from a commit 2026-11-15 19:06:47 -08:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
OpenNeo/impress!22
No description provided.