Commit graph

136 commits

Author SHA1 Message Date
be560e4595 Upgrade async and related gems, and fix async-http response handling
When playing with a Rainbow Pool syncing task, I noticed that error
handling wasn't working correctly for requests using `async-http`: if
the block raised an error, the `Sync` block would never return.

My suspicion is that this is because we were never reading or releasing
the request body.

In this change, I upgrade all the relevant gems for good measure, and
switch to using the response object yielded by the _block_, so we can
know it's being resource-managed correctly. Now, failures raise errors
as expected!

(I tested all these relevant service calls, too!)
2024-09-07 12:14:12 -07:00
30e757b050 Add x86_64 versions of some of the cached gems
Two workstations with different chipsets, wowie!
2024-08-31 12:08:14 -07:00
acade360a8 Update Ruby to 3.3.4, and update gems
For the gems, I mostly just ran `bundle update`; with the exception of
`httparty`, because latest Ruby throws a deprecation warning about its
use of the deprecated `csv` stdlib, which the latest version resolves.

One other little thing: this is on my new Fedora workstation, and I had
to deal with a known bug where the `sassc` gem compiles a `libsass.so`
file, but saves it in the wrong place somehow.

Here's the known bug, and the comment that helped me:
https://github.com/sass/sassc-ruby/issues/146#issuecomment-2028974524

And here's what I ran to get it into the right place:

```shell
ln -s ~/.local/share/gem/ruby/3.3.0/extensions/aarch64-linux/3.3.0/sassc-2.4.0/sassc/libsass.so \
      ~/.local/share/gem/ruby/3.3.0/gems/sassc-2.4.0/lib/sassc/libsass.so
```

This thing about `libsass` isn't reflected in the code changes anywhere
in this commit! I'm just mentioning it so that it's literally written
down anywhere. (I did try other comments' advice to use an older
version of `sassc` first, but I ran into compilation errors, so figured
this machine-side hack was better than untangling that mess.)
2024-08-27 17:06:22 -07:00
a37dda6af7 Add Solargraph autocomplete while in development
This is a Ruby language server that integrates with my editor! Static
analysis of Ruby and Rails is pretty tricky, but it's working and I
think that's neat!!
2024-07-01 15:35:39 -07:00
e1c598e591 Remove unused sqlite3 gem
Oh huh, I guess we used to use this for automated testing, but since
then I've moved the test database to just be in MySQL like everything
else, so I think we don't need this adapter anymore! Goodbye!
2024-06-04 12:23:39 -07:00
3fa724bbc4 Upgrade to Rails 7.1.3.4
Motivated by a security patch, I don't think we're affected by it but
whatever, it's a convenient time to get it done!
2024-06-04 11:32:51 -07:00
3d0c506370 Update some cached gems for macOS
I'm back to working on my Mac laptop again! Running `bundle install`
updated some gems' cached built versions for this OS, so, ta da!
2024-05-02 12:19:08 -07:00
9cbeee0acd Refactor to use OpenID Connect OmniAuth gem instead of plain OAuth2
Right, I didn't totally connect the dots that there's some OpenID
features in the mix here for how we expect to identify the user once
they authenticate. It requires looking up the provider's public key,
and validating the JWT they sent us. This gem does all that for us!

I don't actually know what a real NeoPass `id_token` looks like yet?
But I'll fill in some placeholder stuff for now, and use that for
initializing the account!
2024-03-14 18:11:40 -07:00
f483722af4 NeoPass strategy interacts with dev NeoPass server, which is still WIP
In this change, we wire up a new NeoPass OAuth2 strategy for OmniAuth,
and hook up the "Log in with NeoPass" button to use it!

The authentication currently fails with `invalid_credentials`, and
shows the `owo` response we hardcoded into the NeoPass server's token
response. We need to finally follow up on the little `TODO` written in
there!
2024-03-14 16:13:31 -07:00
08b1b9e83b Add OmniAuth plugin to AuthUser
This is setting us up for NeoPass, but first we're just gonna try stuff
with the "developer" strategy that's built in for testing, rather than
using the NeoPass dev server!
2024-03-14 15:06:13 -07:00
684dcb53ba Add Turbo to speed up the app, and set up for missing UJS features
Oh right, we don't have Rails UJS going on anymore, which is what
handled the confirmation prompts for deleting lists. Turbo is the more
standard modern solution to that, and should speed up certain
pageloads, so let's do it!

Here I install the `turbo-rails` gem, then run `rails turbo:install` to
install the `@hotwired/turbo-rails` npm package. Then I move
`application.js` that's run all on pages but the outfit editor into our
section of JS that gets run through the bundler, and add Turbo to it.

I had to fix a couple tricky things:

1. The outfit editor page doesn't play nice with being swapped into the
   document, so I make it require a full page reload instead.
2. Prefetching the Sign In link can cause the wrong `return_to` address
   to be written to the `session`. (It's a GET request that does, ever
   so slightly, take its own actions, oops!) As a simple hacky answer,
   we disallow prefetching on that link.

Haven't fixed up the UJS stuff for confirm prompts to use Turbo yet,
that's next!
2024-03-13 13:43:48 -07:00
8dc11f9940 Create rails public_data:commit task, to share public data dumps
I'm starting to port over the functionality that was previously just,
me running `yarn db:export:public-data` in `impress-2020` and
committing it to Git LFS every time.

My immediate motivation is that the `impress-2020` git repository is
getting weirdly large?? Idk how these 40MB files have blown up to a
solid 16GB of Git LFS data (we don't have THAT many!!!), but I guess
there's something about Git LFS's architecture and disk usage that I'm
not understanding.

So, let's move to a simpler system in which we don't bind the public
data to the codebase, but instead just regularly dump it in production
and make it available for download.

This change adds the `rails public_data:commit` task, which when run in
production will make the latest available at
`https://impress.openneo.net/public-data/latest.sql.gz`, and will also
store a running log of previous dumps, viewable at
`https://impress.openneo.net/public-data/`.

Things left to do:
1. Create a `rails public_data:pull` task, to download `latest.sql.gz`
   and import it into the local development database.
2. Set up a cron job to dump this out regularly, idk maybe weekly? That
   will grow, but not very fast (about 2GB per year), and we can add
   logic to rotate out old ones if it starts to grow too far. (If we
   wanted to get really intricate, we could do like, daily for the past
   week, then weekly for the past 3 months, then monthly for the past
   year, idk. There must be tools that do this!)
2024-02-29 14:30:33 -08:00
45579167dc Run bundle update
I also put in a manual bump for `falcon`!

The motivation is that I'm working on a Ruby 3.3.0 upgrade in another
branch, and I'm getting deprecation warnings from the `async` gem,
which I think are resolved in the latest version, so I figure, hey,
good time for an update!
2024-02-22 11:48:32 -08:00
ec1cd65a2e Upgrade to Rails 7.1.3.2
In response to:
- https://discuss.rubyonrails.org/t/possible-xss-vulnerability-in-action-controller/84947
- https://discuss.rubyonrails.org/t/possible-redos-vulnerability-in-accept-header-parsing-in-action-dispatch/84946

No evidence that our app has been affected, but I like to move quickly
on these!
2024-02-22 11:25:52 -08:00
f85703d2cc Remove now-unused globalize dependency
We previously used this to translate fields on some models: Color,
Species, Zone, and Item.

We no longer do, so we can safely remove the gem!
2024-02-20 16:58:39 -08:00
7cf3786023 Uninstall puma gem
Falcon works in development too, puma no longer needed!
2024-01-29 01:04:31 -08:00
76af587e7c Replace falcon server with puma
Been wanting this for a while in theory, gonna actually do it now!

The motivation is that I want to turn up the timeout for loading pets,
because the Neopets endpoints are slower today with the NC UC release -
but I can already predict that under our current architecture that will
be a problem, because it'll block up our request queue!

Falcon uses Ruby's relatively-new async system to *not* have requests
block on upstream requests, and my understanding is that this behavior
is plug-and-play. Let's see how it goes!
2024-01-23 21:55:26 -08:00
470c805880 Save last trade activity time onto User
In impress-2020, we do a big slow query to figure out which users have
been active in trades recently. Now, we cache that timestamp on the
User model.

This won't have any immediate effect; it's to clear the way for Classic
DTI to receive the better trade ratios feature people like from 2020.

I also added some unit testing infra because I finally wanted it! for
all the ways you can trigger this timestamp lol

Note too that this is a bit of an unusually complex migration, but my
hope is that the batching and query structure and such helps it run
surprisingly fast! 🤞
2024-01-19 00:00:46 -08:00
d2de971a60 Delete more rake tasks
I tried to port the Rainbow Pool ones forward, but ran into issues with the
service that uses browser-specific stuff to check that traffic is valid :/

Incidentally, those were the only places we were using `rest-client`.
Goodbye!
2023-11-10 18:59:46 -08:00
dc22a458bf Move manifest backfill to swf_assets:manifests task
Okay, I've simplified the migration to *just* add the column, and
instead added a task to find assets without manifest URLs and backfill
them.

Performance is a lot better now, using the `async-http` library, which
as I understand it supports both persistent connections when invoked
like this, and maybe also HTTP/2 multiplexing?? (Though I'm not
actually sure images.neopets.com does lol)

I'm not sure about the number of concurrent tasks I picked here, 100
seems okay for an internet thing and for such small requests, but I
worry that the CDN is gonna get annoyed or something. Well, we'll see!
This task is very resumable if it turns out we get frozen out or
something.
2023-11-10 16:52:50 -08:00
18ff22f211 Add Sentry to Rails
Now we're tracking both JS and Rails errors, phew!
2023-11-06 12:37:40 -08:00
c496e33c37 Add mini profiler to each page
It shows up in development always, and if you're logged in as Me
Specifically in production!

I'm using this to poke at memory usage for pages that seem suspicious.
I don't know why our app reliably grows so large in RAM, but my hunch is
that maybe there are some pages that just use a truly large amount to
begin with - and I've learned Ruby doesn't release memory back after
it's GC'd, it just grows the process and keeps the free space to itself
in its own heap!

So I'm just eyeing pages that I know *can* have a lot going on, and
seeing what I find!
2023-10-27 19:38:49 -07:00
793c2c0451 Merge branch 'main' into devcontainer 2023-10-25 16:47:23 -07:00
13371e3cf2 Remove unused automated testing files & gems
Look, I'll be real, I have literally not run these automated tests in
probably like a whole decade. Most of these files are empty, the ones
that aren't seem basically trivial, and I bet half of it would fail
anyway.

If I wanted to do real automated testing, I would basically want to
start from scratch anyway, and apply coverage I can trust to the areas
I actually care about.

Until then, I feel like these gems and files are mostly just clutter,
and I don't like them being One More Barrier To Entry. Goodbye, unused
complexity!
2023-10-25 16:09:01 -07:00
56ce32b6cb Upgrade to Rails 7.1.1
The usual stuff! Installed the new gem and its new deps, ran
`bin/rails app:update` and did my best to manually merge the dev/prod
config files with the new canonical defaults, deleted some migrations I
don't think are relevant to us, and yeah!

Also, Rails 7.1 seems to need `libyaml-dev` installed, so I added that
to the `deploy/setup.yml` playbook!

One thing to note is that, while I was here, I turned on some settings
relating to our use of SSL that technically weren't on before. This
should be fine and helpful? But if stuff breaks, well, check those!
2023-10-25 15:05:31 -07:00
2f3294b899 [WIP] Build a dev container
Idk why, but unlike my previous experience with Rails devcontainers, this time the setup process is running so wildly slowly?

Might just be a transient issue on my machine, maybe something that would be improved with a restart and trying again another time? Or could be something about the MySQL image that doesn't run great in this context?

In any case, I'm just gonna set this down for now!
2023-10-25 11:46:14 -07:00
661fbd2d03 Security updates
Resolves CVE-2023-40175 and CVE-2023-38037.
2023-10-23 19:08:56 -07:00
de245f96f3 Delete some unused public/assets files
Just cleaning up a bit! I'm sure there's more to remove, these were just some clear candidates: old wardrobe code, and stuff in `public` that I just fully don't recognize and don't think is doing anything? (We'll find out if something crashes though lol!)
2023-10-23 19:05:09 -07:00
8bd7ca167c Remove leftover Capistrano stuff
We were using that for deployment before, and now we're using my Ansible recipes!
2023-10-23 19:05:09 -07:00
74197a6e9f Upgrade to latest Sass and Sprockets
This required a buncha fixes to how SASS scoping works! Needed to add a bunch of imports for stuff that previously would get read from the global scope by being imported *after* the constants and mixins etc.

There's clearly a lot of refactor opportunity here, but I'm not gonna worry about it!!
2023-10-23 19:05:09 -07:00
15002d19db Remove compass-rails
I wasn't sure what we were actually using it for, turns out it was mostly polyfills for CSS features that are very standard now!

I didn't audit these changes very carefully tbqh because they seemed pretty simple? Fingers crossed!
2023-10-23 19:05:09 -07:00
9b68e982e7 Precompile assets when deploying new version
I did some refactoring while here too, of pulling the deploy scripts out of `package.json` and into `bin`, to be a bit more canonically Rails-y. (idk how canonical the colon thing is but, probably fine??)
2023-10-23 19:05:09 -07:00
4ddcb005ea Remove memcache from production config
I don't know enough about our caching situation to know where memcache performs meaningfully better than Rails's in-memory cache. Let's delete it for now and see if there's a problem, to simplify the deploy environment!
2023-10-23 19:05:09 -07:00
6b8fc6407e Use puma web server
Works in dev, and people seem to like it for prod! (I'm working on some deploy stuff is what.)
2023-10-23 19:05:09 -07:00
81b2a2b4a2 Bundle wardrobe-2020 into the app
We add jsbuilding-rails to get esbuild running in the app, and then we copy-paste the files we need from impress-2020 into here!

I stopped at the point where it was building successfully, but it's not running correctly: it's not sure about `process.env` in `next`, and I think the right next step is to delete the NextJS deps altogether and use React Router instead.
2023-10-23 19:05:08 -07:00
83f80facda Can log into OpenNeo ID accounts directly!
A lot of rough edges here (e.g. no styles on the flash messages), but it's working and that's good!!

I tested this by temporarily switching to the production database and logging in as matchu!

Still missing a lot of big features too, like registration, password resets, settings page, etc.
2023-10-23 19:05:07 -07:00
700e26d7df Remove old OpenNeo ID auth code
This removes login/logout/session logic for integrating with OpenNeo ID, replacing them with stubs that just redirect to `/?TODO` when you click login, and helpers that act as if you're not logged in.

This gives us a clean slate to plug in new Devise logic to integrate with the `openneo_id` database directly!
2023-10-23 19:05:07 -07:00
7e922503b5 Upgrade to Rails 7.0.6
Whew! Seems like a pretty clean one? Ran `rails app:upgrade` and stuff, and made some corrections to keyword arguments for `translate` calls. There might be more such problems elsewhere? But that's hard to search for, and we'll have to see.
2023-10-23 19:05:07 -07:00
02a9afbb74 Upgrade to Ruby 3.0.6
Hey nice! We have to add webrick now because it's not included in Ruby 3, but hey just drop it right back in.

Idk how to choose between this or puma or whatever, but in the absence of a specific reason let's just pick the one whose name I know best.
2023-10-23 19:05:07 -07:00
59efb49419 Upgrade to Rails 6.1.7.4
This one was pretty straightforward yaay! Main thing was the change from `render file` to `render template` in a couple places, oh and a thing with complex `order()` clauses.
2023-10-23 19:05:07 -07:00
eaf43128ba Add web console support for Vagrant users 2023-10-23 19:05:06 -07:00
be7e11a0d0 Upgrade to Rails 6.0.6.1
Another pretty easy one! We have the `rails app:update` changes in here too.
2023-10-23 19:05:06 -07:00
bb5065deed Update all our gems as far as we can
Got pretty far, there's a couple still held back by Rails 6+, but yeah doing great!
2023-10-23 19:05:06 -07:00
bc3a9bc33b Fix gems we'd locked to old versions
Now that we're on Ruby 2.6, the constraints on the latest versions of these dependencies are satisfied, so we can stop holding them back!
2023-10-23 19:05:06 -07:00
fc34208db9 Upgrade web-console
Huh, error pages weren't actually working on this older version of web-console, but now they are!

Solution found here: https://discuss.rubyonrails.org/t/question-dev-mode-500-errors-not-displayed-config-consider-all-requests-local-not-honored/78453/7
2023-10-23 19:05:05 -07:00
86edc8584f Run rails app:update
We accepted some changes as-is, but for development.rb and production.rb we read the diff and manually edited them!
2023-10-23 19:05:05 -07:00
d97c32b5da Upgrade to Rails 5.2.8.1
Some important little upgrades but mostly straightforward!

Note that there's still a known issue where item searches crash, I was hoping that this was a bug in Rails 4.2 that would be fixed on upgading to 5, but nope, oh well!

Also uhh I just got a bit silly and didn't actually mean to go all the way to 5.2 in one go, I had meant to start at 5.0… but tbh the 5.1 and 5.2 changes seem small, and this seems to be working, so. Yeah ok let's roll!
2023-10-23 19:05:05 -07:00
685ac2b4ad Make 4.1 -> 4.2 changes
Just following the recommended stuff in the upgrade guide!
2023-10-23 19:05:05 -07:00
f80d220a62 Upgrade to Rails 4.2.11.3 and Ruby 2.4.10
Some tricks required here to get the dependencies to work out, but we got it!!

Oh also, we move away from the rbenv in Ubuntu's package manager, because it doesn't support more recent Rubies like 2.4.10.
2023-10-23 19:05:05 -07:00
4091ce2a5b Remove dotenv-deployment, upgrade dotenv-rails
Idk exactly what's going on with dotenv-deployment, if it turns out it was critical to our deploy process then we'll change the deploy process! It's deprecated and conflicts with gem deps for `dotenv-rails`.
2023-10-23 19:05:05 -07:00