1
0
Fork 1
Commit graph

103 commits

Author SHA1 Message Date
1d4771ecc5 Create new DTIRequests.load_many helper, to make parallel requests
Just a wrapper for the barrier/semaphore thing we're doing constantly!

I only applied it in the importer rake tasks for now. There's other
call sites to move over to it, too!
2024-12-16 13:16:03 -08:00
acb52cb870 Move NCMall and NeoPass services into a Neopets module
Just a bit more clarity of grouping! I'm also thinking about extracting
modeling APIs into a service file like this too, in which case I think
this would help clarify what it is.
2024-10-18 17:27:15 -07:00
4f9fbc1ac0 Improve pet type "Added" timestamp styles 2024-10-11 16:24:47 -07:00
5bf2ef42a0 Move JS libraries to vendor/javascript
The silly motivation is that I wanted to remove `.prettierignore`,
which just exists to omit that one folder from `npm run format`. But it
also seems like this is the standard place to put them—a standard
created long after we first set this up lol
2024-09-13 21:16:46 -07:00
0a5d369735 Remove i18n locale config complexity we don't use anymore
I forget what this was for, I think part of it was for managing item
names in different languages, and the "private" locale thing was
probably for WIP locales? But yeah, not used, delete!
2024-09-13 20:55:09 -07:00
ec476f4c65 List the specific Dyeworks end date in Item Getting Guide, if we know 2024-06-18 16:47:05 -07:00
a55a9b08d4 Disable sending performance traces to health.openneo.net
There's been some stability issues with our self-hosted
health.openneo.net service lately. I just upgraded to a new version
today, so my hope is that today's weird slowness is that it's doing the
expected GlitchTip 4.0 upgrade data migration process designed to save
disk space, and hopefully it will take care of itself by the end of the
day?

But just to help out, I'm disabling this feature to remove pressure on
the GlitchTip service. We don't actually use performance trace analysis
irl, and so it's just taking up disk space and processing power. If we
end up wanting to dig into something in the future, we can turn it back
on! (My hope had been that a sample rate of 5% was small enough to be a
good hedge to have "enough" data just in case something comes up, but
idk, I don't actually have a great sense of how much pressure even 5%
of our total volume is, and I'd rather just be certain it's out of the
picture altogether, at least while working on this.)
2024-05-19 19:14:06 -07:00
7b0b6b70d2 Initial NC Mall scraper service
This doesn't connect to anything yet, I'm just doing the beginnings of
loading NC Mall item data!

My intent is to run this regularly to keep our own NC info in the
database too, primarily for use in the Item Getting Guide. (Could be
useful to surface in other places too though!) This will help us split
items into those that can be one-click purchased with the NC Mall
integration, vs NC items that need to be acquired by other means.
2024-05-07 16:06:37 -07:00
644b181ed0 Use Neopets username as base name for new NeoPass accounts, if possible
Yay, we got the API endpoint for this! The `linkage` scope is the key.

Rather than pulling back the specific fallback behavior we had wrote
for usernames before, which was slightly different and involved
appending `neopass` in there too (e.g. `matchu-neopass-1234`), I
figured let's just use a lot of the same logic, and just use the
preferred name as the base name. (I figure the `neopass` suffix isn't
that useful anyway, `matchu-1234` kinda looks better tbh! And it's all
fallback stuff that I expect serious users to replace, anyway.)
2024-04-09 07:48:13 -07:00
e2d763e3c3 Fix NeoPass access token request to use POST data instead of Auth header
Ahh okay, the NeoPass spec says to pass the `client_id` and
`client_secret` as POST form data when exchanging the code for the
access token, but the default behavior of our client is to pass it as
an `Authorization` header instead.

In this change, we set an option to change that behavior, and also add
a lot of comments about this and the other options!
2024-04-01 05:08:47 -07:00
08986153df Add our client ID and client secret, to connect to NeoPass for real!
Wowie, it's starting to happen! :3

When you run this in production, though, you get back the auth failure
message, and the OmniAuth logs say the server returned the following:

> invalid_client: Client authentication failed (e.g., unknown client,
> no client authentication included, or unsupported authentication
> method). The OAuth 2.0 Client supports client authentication method
> 'client_secret_post', but method 'client_secret_basic' was requested.
> You must configure the OAuth 2.0 client's
> 'token_endpoint_auth_method' value to accept 'client_secret_basic'.

I'll add a fix for this in the next commit, with some explanations as
to why!
2024-04-01 04:55:42 -07:00
9cbeee0acd Refactor to use OpenID Connect OmniAuth gem instead of plain OAuth2
Right, I didn't totally connect the dots that there's some OpenID
features in the mix here for how we expect to identify the user once
they authenticate. It requires looking up the provider's public key,
and validating the JWT they sent us. This gem does all that for us!

I don't actually know what a real NeoPass `id_token` looks like yet?
But I'll fill in some placeholder stuff for now, and use that for
initializing the account!
2024-03-14 18:11:40 -07:00
f483722af4 NeoPass strategy interacts with dev NeoPass server, which is still WIP
In this change, we wire up a new NeoPass OAuth2 strategy for OmniAuth,
and hook up the "Log in with NeoPass" button to use it!

The authentication currently fails with `invalid_credentials`, and
shows the `owo` response we hardcoded into the NeoPass server's token
response. We need to finally follow up on the little `TODO` written in
there!
2024-03-14 16:13:31 -07:00
08b1b9e83b Add OmniAuth plugin to AuthUser
This is setting us up for NeoPass, but first we're just gonna try stuff
with the "developer" strategy that's built in for testing, rather than
using the NeoPass dev server!
2024-03-14 15:06:13 -07:00
09090d53ce Migrate from Sentry to self-hosted GlitchTip instance
Trying something new and lightweight and more data-controlled!

I also turned down the sample rate for the performance traces feature,
because we hardly use it right now, and Sentry is always getting mad at
us for vastly exceeding our free plan quota—and like, we're not on
Sentry anymore so I imagine we have more wiggle room with that, but I
figure let's turn down the volume anyway, until we decide we want it.
2024-02-28 13:14:32 -08:00
b7f4c5b6ac Remove asset_hosts initializer
It just contains one constant, used in one place. Inlined and deleted!
2024-02-24 15:05:37 -08:00
666394de25 Refactor Impress 2020 config
I've moved the support secret into the encrypted credentials file, and
moved the origin into a top-level custom config value in the
environment files, with different defaults per environment but still
the ability to override it. (I don't use this, but it feels polite to
not actually *demand* that people use port 4000, y'know?)
2024-02-22 13:07:43 -08:00
5b016673d7 Migrate secret key to Rails credentials file (and fix deprecation warn)
There's a bit happening behind the scenes of this change. Previously,
we kept a `SECRET_TOKEN` environment variable in `production.env`, and
used a `secret_token.rb` initializer to wire it up as the
`secret_key_base`.

In this change, we move to Rails's new-ish (two years old :p) encrypted
credentials system. Now, we set a `RAILS_MASTER_KEY` environment
variable in the deployed `production.env` instead (and in our local
`.env.production` in the project root for managing it), and we can run
`rails credentials:edit` to open the encrypted file in a text editor.

Inside, the content is just:
```yml
secret_key_base: "<OUR_SECRET_KEY>"
```

This indirection doesn't exactly do much for us functionally; it's just
the more standard way of achieving what our `secret_token.rb` situation
was achieving.

We could also migrate other secrets into there, and I just might! That
would simplify duplication between `/deploy/files/production.env` and
`/.env.production`, at any rate! The main notable one is
`MATCHU_EMAIL_PASSWORD` for sending auth emails from
`matchu@openneo.net` (and there's also a Stripe token that we don't
actually use in the app these days, those codepaths are old bones). Oh
and there's also the `IMPRESS_2020_SUPPORT_SECRET`!

Anyway, the motivation for this was to remove the warning when starting
the app that Devise is trying to use the deprecated
`Rails.application.secrets` method. I was expecting to have to do
[the workaround shared here](https://github.com/heartcombo/devise/issues/5644#issuecomment-1804626431),
but it turns out whatever default behavior Devise does under the hood
is happy enough with our new decision to use the credentials file, and
the deprecation warning is gone! Ok neat!
2024-02-22 12:36:30 -08:00
8aa6eb40a7 Disable advanced mini-profiler tools in production
I'm not really using this lately, and it _only_ creates vulnerability
surface area when not in use; so, while I'm pretty sure we locked this
down correctly to only admin accounts, I'm disabling it just as good
practice. We can add this back later if we need it again!
2024-02-03 08:21:14 -08:00
4fff8d88f2 Add support_staff flag to user record; they can use Support tools
A little architecture trick here! DTI 2020 authorizes support staff
requests by means of a secret token, instead of user account stuff. And
our support tools still all call DTI 2020 APIs.

So here, we bridge the gap: we copy DTI 2020's support secret to this
app's environment variables (I needed to update
`deploy/files/production.env` and run `bin/deploy:setup` for this!),
then users with the new `support_secret` flag have it added to their
HTML documents in the meta tags. Then, the JS reads the meta tag.

I also fixed an issue in the `deploy/setup.yml` playbook, where I had
temporarily commented some stuff out to skip steps one time, and forgot
to uncomment them after oops lol!
2024-01-29 04:21:19 -08:00
6763e9454e Optionally use local instance of impress-2020 during development
To activate this, I created a `.env.development` file in my project
root, with the following content:

```env
IMPRESS_2020_ORIGIN=http://localhost:4000
```

Then, I started impress-2020 with `yarn dev --port=4000`.

Now, the app loads from there, hooray!! It even fixes that obnoxious
pet state ID bug that happens when you run against the production db lol
2024-01-28 07:00:29 -08:00
1933046809 Try to fix the pet load limiter
This is hard to test directly, but this is my guess from what I'm
reading in this? https://stackoverflow.com/a/32958124/107415
2024-01-24 03:26:52 -08:00
470c805880 Save last trade activity time onto User
In impress-2020, we do a big slow query to figure out which users have
been active in trades recently. Now, we cache that timestamp on the
User model.

This won't have any immediate effect; it's to clear the way for Classic
DTI to receive the better trade ratios feature people like from 2020.

I also added some unit testing infra because I finally wanted it! for
all the ways you can trigger this timestamp lol

Note too that this is a bit of an unusually complex migration, but my
hope is that the batching and query structure and such helps it run
surprisingly fast! 🤞
2024-01-19 00:00:46 -08:00
18ff22f211 Add Sentry to Rails
Now we're tracking both JS and Rails errors, phew!
2023-11-06 12:37:40 -08:00
c496e33c37 Add mini profiler to each page
It shows up in development always, and if you're logged in as Me
Specifically in production!

I'm using this to poke at memory usage for pages that seem suspicious.
I don't know why our app reliably grows so large in RAM, but my hunch is
that maybe there are some pages that just use a truly large amount to
begin with - and I've learned Ruby doesn't release memory back after
it's GC'd, it just grows the process and keeps the free space to itself
in its own heap!

So I'm just eyeing pages that I know *can* have a lot going on, and
seeing what I find!
2023-10-27 19:38:49 -07:00
8ae1370cc9 Remove unused neopia_host config variable
I cleared the references to the Neopia server (old thing responsible
for pet loading without blocking the Rails app) out of here a while ago,
but didn't clear out this config value!
2023-10-25 16:20:28 -07:00
b756ae023e Use a hardcoded SECRET_TOKEN, in development only
Oh right, we intentionally fail if there's no SECRET_TOKEN provided, but
that's not really useful for development!

Here, we add a SECRET_TOKEN only used in development - which doesn't
need to be secret, because it doesn't guard actual user sessions!

In production, the behavior is unchanged.
2023-10-25 15:54:19 -07:00
56ce32b6cb Upgrade to Rails 7.1.1
The usual stuff! Installed the new gem and its new deps, ran
`bin/rails app:update` and did my best to manually merge the dev/prod
config files with the new canonical defaults, deleted some migrations I
don't think are relevant to us, and yeah!

Also, Rails 7.1 seems to need `libyaml-dev` installed, so I added that
to the `deploy/setup.yml` playbook!

One thing to note is that, while I was here, I turned on some settings
relating to our use of SSL that technically weren't on before. This
should be fine and helpful? But if stuff breaks, well, check those!
2023-10-25 15:05:31 -07:00
bb20925382 Fix deprecation warning in Rack::Attack
Looking at the docs, I think what changed is that `throttled_responder` gets the request as an argument instead of the `env`? And has the same return type for the lambda as before?

So uhhh I don't remember how to test this, but uhh it's not crashing when the server starts anymore, and I feel like the most likely problem here would be that you get a 500 instead of a useful response in the rate limit case, so like. ehh I'll just leave it be!
2023-10-23 19:05:09 -07:00
64e3702d6d Fix typography on wardrobe 2020 page
Add the green body color back in, and add the Delicious font!
2023-10-23 19:05:09 -07:00
83f80facda Can log into OpenNeo ID accounts directly!
A lot of rough edges here (e.g. no styles on the flash messages), but it's working and that's good!!

I tested this by temporarily switching to the production database and logging in as matchu!

Still missing a lot of big features too, like registration, password resets, settings page, etc.
2023-10-23 19:05:07 -07:00
700e26d7df Remove old OpenNeo ID auth code
This removes login/logout/session logic for integrating with OpenNeo ID, replacing them with stubs that just redirect to `/?TODO` when you click login, and helpers that act as if you're not logged in.

This gives us a clean slate to plug in new Devise logic to integrate with the `openneo_id` database directly!
2023-10-23 19:05:07 -07:00
7e922503b5 Upgrade to Rails 7.0.6
Whew! Seems like a pretty clean one? Ran `rails app:upgrade` and stuff, and made some corrections to keyword arguments for `translate` calls. There might be more such problems elsewhere? But that's hard to search for, and we'll have to see.
2023-10-23 19:05:07 -07:00
59efb49419 Upgrade to Rails 6.1.7.4
This one was pretty straightforward yaay! Main thing was the change from `render file` to `render template` in a couple places, oh and a thing with complex `order()` clauses.
2023-10-23 19:05:07 -07:00
7d582be79a Invalidate old session cookies
The session format changed, so we change the session cookie name rather than have things crash about it! (I hope the actual prod behavior is to ignore bad cookies rather than crash? But I figure this is more reliable anyway.)
2023-10-23 19:05:06 -07:00
be7e11a0d0 Upgrade to Rails 6.0.6.1
Another pretty easy one! We have the `rails app:update` changes in here too.
2023-10-23 19:05:06 -07:00
86edc8584f Run rails app:update
We accepted some changes as-is, but for development.rb and production.rb we read the diff and manually edited them!
2023-10-23 19:05:05 -07:00
c9a7f0c4d4 Delete ~empty initializer files
Idk I guess these are the default place to put certain settings, but idk if they're still canonical, and I'd rather just not have files that don't mean anything rn!
2023-10-23 19:05:05 -07:00
a8ee091f98 Delete unused Pledgie files
This was back when we used a third-party campaign tool! Haven't run this code in ages!
2023-10-23 19:05:05 -07:00
f159091f0e Delete unused bullet.rb config file 2023-10-23 19:05:05 -07:00
c4f3a472ff Remove RightAWS gem
We're not connecting to AWS directly anymore, now that we deleted the SWF conversion stuff, so we can finally clear this out!
2023-10-23 19:05:05 -07:00
Matchu
d0616b6dfd Delete Camo references & Addressable gem
At one point we piloted a "Camo" service to proxy HTTPS image urls for us, but it doesn't exist anymore.

We already have proxies and stuff for this, so I left `Image` as a placeholder for this, but it's not working yet!

This also deletes our final reference to the Addressable gem, so we can remove it!
2023-10-23 19:05:05 -07:00
Matchu
b0cceb8ce7 Uninstall Airbrake
Haven't checked it in forever, if we want an error reporting solution we'll create it fresh!
2023-10-23 19:05:05 -07:00
Matchu
31ebf7d4eb Remove OutfitImageUploader and Carrierwave
Ok right, missed this in the outfit image stuff deletion, get rid of the code and library we were using to push those images up to S3!
2023-10-23 19:05:05 -07:00
Matchu
1195a6190b Uninstall resque
Yay, we've deleted all our background tasks!

We'll probably want to replace some of the basic functionality like certain caching? But we can deal with that as we run into it.

The direct motivation here was a seeming version conflict between Rails 4.2's rack dependency and latest Resque's rack dependency... but this is just nice complexity elimination regardless, we want this anyway :3
2023-10-23 19:05:04 -07:00
Matchu
e121d8bba2 Remove SWF conversion
We've already swapped out the backend for this stuff to Impress 2020, so the resque task and the broken image report UI aren't actually relevant anymore. Delete them!

This helps us delete Resque soon too.
2023-10-23 19:05:04 -07:00
Matchu
b35c773be3 Remove flex from the app (breaks search!)
We'll need to replace the item search query stuff with direct MySQL queries, but that's not ready yet bc the app still isn't booting, so we're committing this in a known broken state for now!
2023-10-23 19:05:02 -07:00
Matchu
1ed973f319 Remove newrelic gem
I haven't logged into newrelic in a billion years, let's just stop sending them stuff

(This is a precursor to an attempt to delete flex stuff too and replace our elasticsearch stuff with direct mysql queries like Impress 2020 does, but that'll be more work!)
2023-10-23 19:05:02 -07:00
Matchu
ca858f1350 Remove unused Devise initializer settings
I guess the APIs changed here, but these were placeholder settings we weren't actually using anyway (cuz we use the OpenNeo ID integration), so I just commented them out and it seems fine for now!
2023-10-23 19:05:02 -07:00
Matchu
72a08901c8 Upgrade to Ruby 2.2.4, Rails 4.0.13
NOTE: This doesn't boot yet! There's something changed in the `devise` API that we'll need to fix!

```
/vagrant/config/initializers/devise.rb:46:in `block in <top (required)>': undefined method `encryptor=' for Devise:Module (NoMethodError)
```

But yeah, we navigated the gem upgrades, and also I ran `rake rails:update` and hand-processed the suggestions it had for our config files.
2023-10-23 19:05:02 -07:00