Commit graph

5 commits

Author SHA1 Message Date
be560e4595 Upgrade async and related gems, and fix async-http response handling
When playing with a Rainbow Pool syncing task, I noticed that error
handling wasn't working correctly for requests using `async-http`: if
the block raised an error, the `Sync` block would never return.

My suspicion is that this is because we were never reading or releasing
the request body.

In this change, I upgrade all the relevant gems for good measure, and
switch to using the response object yielded by the _block_, so we can
know it's being resource-managed correctly. Now, failures raise errors
as expected!

(I tested all these relevant service calls, too!)
2024-09-07 12:14:12 -07:00
46d3325144 Load *all* NC Mall pages in nc_mall:sync
Ta da! Now I can run this and pull 481 records into our database, and
then turn around and run it again and have them all correctly say
"skipped"!
2024-05-10 17:39:40 -07:00
b6e18e10a5 Add bare-bones rails nc_mall:sync task, incl. NCMallRecord model
Currently we only load the homepage, so there's only actually one
wearable item to sync up! But here's the task to do it!

To do this, we also created the backing model NCMallRecord, where we'll
save the current NC Mall state!
2024-05-07 17:40:14 -07:00
1f157b49da Load additional pages via NC Mall scraper service
This is for URLs like this! https://ncmall.neopets.com/mall/ajax/load_page.phtml?type=browse&cat=43&lang=en
2024-05-07 17:38:48 -07:00
7b0b6b70d2 Initial NC Mall scraper service
This doesn't connect to anything yet, I'm just doing the beginnings of
loading NC Mall item data!

My intent is to run this regularly to keep our own NC info in the
database too, primarily for use in the Item Getting Guide. (Could be
useful to surface in other places too though!) This will help us split
items into those that can be one-click purchased with the NC Mall
integration, vs NC items that need to be acquired by other means.
2024-05-07 16:06:37 -07:00