There's been some stability issues with our self-hosted
health.openneo.net service lately. I just upgraded to a new version
today, so my hope is that today's weird slowness is that it's doing the
expected GlitchTip 4.0 upgrade data migration process designed to save
disk space, and hopefully it will take care of itself by the end of the
day?
But just to help out, I'm disabling this feature to remove pressure on
the GlitchTip service. We don't actually use performance trace analysis
irl, and so it's just taking up disk space and processing power. If we
end up wanting to dig into something in the future, we can turn it back
on! (My hope had been that a sample rate of 5% was small enough to be a
good hedge to have "enough" data just in case something comes up, but
idk, I don't actually have a great sense of how much pressure even 5%
of our total volume is, and I'd rather just be certain it's out of the
picture altogether, at least while working on this.)
This doesn't connect to anything yet, I'm just doing the beginnings of
loading NC Mall item data!
My intent is to run this regularly to keep our own NC info in the
database too, primarily for use in the Item Getting Guide. (Could be
useful to surface in other places too though!) This will help us split
items into those that can be one-click purchased with the NC Mall
integration, vs NC items that need to be acquired by other means.
Yay, we got the API endpoint for this! The `linkage` scope is the key.
Rather than pulling back the specific fallback behavior we had wrote
for usernames before, which was slightly different and involved
appending `neopass` in there too (e.g. `matchu-neopass-1234`), I
figured let's just use a lot of the same logic, and just use the
preferred name as the base name. (I figure the `neopass` suffix isn't
that useful anyway, `matchu-1234` kinda looks better tbh! And it's all
fallback stuff that I expect serious users to replace, anyway.)
Ahh okay, the NeoPass spec says to pass the `client_id` and
`client_secret` as POST form data when exchanging the code for the
access token, but the default behavior of our client is to pass it as
an `Authorization` header instead.
In this change, we set an option to change that behavior, and also add
a lot of comments about this and the other options!
Wowie, it's starting to happen! :3
When you run this in production, though, you get back the auth failure
message, and the OmniAuth logs say the server returned the following:
> invalid_client: Client authentication failed (e.g., unknown client,
> no client authentication included, or unsupported authentication
> method). The OAuth 2.0 Client supports client authentication method
> 'client_secret_post', but method 'client_secret_basic' was requested.
> You must configure the OAuth 2.0 client's
> 'token_endpoint_auth_method' value to accept 'client_secret_basic'.
I'll add a fix for this in the next commit, with some explanations as
to why!
Right, I didn't totally connect the dots that there's some OpenID
features in the mix here for how we expect to identify the user once
they authenticate. It requires looking up the provider's public key,
and validating the JWT they sent us. This gem does all that for us!
I don't actually know what a real NeoPass `id_token` looks like yet?
But I'll fill in some placeholder stuff for now, and use that for
initializing the account!
In this change, we wire up a new NeoPass OAuth2 strategy for OmniAuth,
and hook up the "Log in with NeoPass" button to use it!
The authentication currently fails with `invalid_credentials`, and
shows the `owo` response we hardcoded into the NeoPass server's token
response. We need to finally follow up on the little `TODO` written in
there!
This is setting us up for NeoPass, but first we're just gonna try stuff
with the "developer" strategy that's built in for testing, rather than
using the NeoPass dev server!
Trying something new and lightweight and more data-controlled!
I also turned down the sample rate for the performance traces feature,
because we hardly use it right now, and Sentry is always getting mad at
us for vastly exceeding our free plan quota—and like, we're not on
Sentry anymore so I imagine we have more wiggle room with that, but I
figure let's turn down the volume anyway, until we decide we want it.
I've moved the support secret into the encrypted credentials file, and
moved the origin into a top-level custom config value in the
environment files, with different defaults per environment but still
the ability to override it. (I don't use this, but it feels polite to
not actually *demand* that people use port 4000, y'know?)
There's a bit happening behind the scenes of this change. Previously,
we kept a `SECRET_TOKEN` environment variable in `production.env`, and
used a `secret_token.rb` initializer to wire it up as the
`secret_key_base`.
In this change, we move to Rails's new-ish (two years old :p) encrypted
credentials system. Now, we set a `RAILS_MASTER_KEY` environment
variable in the deployed `production.env` instead (and in our local
`.env.production` in the project root for managing it), and we can run
`rails credentials:edit` to open the encrypted file in a text editor.
Inside, the content is just:
```yml
secret_key_base: "<OUR_SECRET_KEY>"
```
This indirection doesn't exactly do much for us functionally; it's just
the more standard way of achieving what our `secret_token.rb` situation
was achieving.
We could also migrate other secrets into there, and I just might! That
would simplify duplication between `/deploy/files/production.env` and
`/.env.production`, at any rate! The main notable one is
`MATCHU_EMAIL_PASSWORD` for sending auth emails from
`matchu@openneo.net` (and there's also a Stripe token that we don't
actually use in the app these days, those codepaths are old bones). Oh
and there's also the `IMPRESS_2020_SUPPORT_SECRET`!
Anyway, the motivation for this was to remove the warning when starting
the app that Devise is trying to use the deprecated
`Rails.application.secrets` method. I was expecting to have to do
[the workaround shared here](https://github.com/heartcombo/devise/issues/5644#issuecomment-1804626431),
but it turns out whatever default behavior Devise does under the hood
is happy enough with our new decision to use the credentials file, and
the deprecation warning is gone! Ok neat!
I'm not really using this lately, and it _only_ creates vulnerability
surface area when not in use; so, while I'm pretty sure we locked this
down correctly to only admin accounts, I'm disabling it just as good
practice. We can add this back later if we need it again!
A little architecture trick here! DTI 2020 authorizes support staff
requests by means of a secret token, instead of user account stuff. And
our support tools still all call DTI 2020 APIs.
So here, we bridge the gap: we copy DTI 2020's support secret to this
app's environment variables (I needed to update
`deploy/files/production.env` and run `bin/deploy:setup` for this!),
then users with the new `support_secret` flag have it added to their
HTML documents in the meta tags. Then, the JS reads the meta tag.
I also fixed an issue in the `deploy/setup.yml` playbook, where I had
temporarily commented some stuff out to skip steps one time, and forgot
to uncomment them after oops lol!
To activate this, I created a `.env.development` file in my project
root, with the following content:
```env
IMPRESS_2020_ORIGIN=http://localhost:4000
```
Then, I started impress-2020 with `yarn dev --port=4000`.
Now, the app loads from there, hooray!! It even fixes that obnoxious
pet state ID bug that happens when you run against the production db lol
In impress-2020, we do a big slow query to figure out which users have
been active in trades recently. Now, we cache that timestamp on the
User model.
This won't have any immediate effect; it's to clear the way for Classic
DTI to receive the better trade ratios feature people like from 2020.
I also added some unit testing infra because I finally wanted it! for
all the ways you can trigger this timestamp lol
Note too that this is a bit of an unusually complex migration, but my
hope is that the batching and query structure and such helps it run
surprisingly fast! 🤞
It shows up in development always, and if you're logged in as Me
Specifically in production!
I'm using this to poke at memory usage for pages that seem suspicious.
I don't know why our app reliably grows so large in RAM, but my hunch is
that maybe there are some pages that just use a truly large amount to
begin with - and I've learned Ruby doesn't release memory back after
it's GC'd, it just grows the process and keeps the free space to itself
in its own heap!
So I'm just eyeing pages that I know *can* have a lot going on, and
seeing what I find!
I cleared the references to the Neopia server (old thing responsible
for pet loading without blocking the Rails app) out of here a while ago,
but didn't clear out this config value!
Oh right, we intentionally fail if there's no SECRET_TOKEN provided, but
that's not really useful for development!
Here, we add a SECRET_TOKEN only used in development - which doesn't
need to be secret, because it doesn't guard actual user sessions!
In production, the behavior is unchanged.
The usual stuff! Installed the new gem and its new deps, ran
`bin/rails app:update` and did my best to manually merge the dev/prod
config files with the new canonical defaults, deleted some migrations I
don't think are relevant to us, and yeah!
Also, Rails 7.1 seems to need `libyaml-dev` installed, so I added that
to the `deploy/setup.yml` playbook!
One thing to note is that, while I was here, I turned on some settings
relating to our use of SSL that technically weren't on before. This
should be fine and helpful? But if stuff breaks, well, check those!
Looking at the docs, I think what changed is that `throttled_responder` gets the request as an argument instead of the `env`? And has the same return type for the lambda as before?
So uhhh I don't remember how to test this, but uhh it's not crashing when the server starts anymore, and I feel like the most likely problem here would be that you get a 500 instead of a useful response in the rate limit case, so like. ehh I'll just leave it be!
A lot of rough edges here (e.g. no styles on the flash messages), but it's working and that's good!!
I tested this by temporarily switching to the production database and logging in as matchu!
Still missing a lot of big features too, like registration, password resets, settings page, etc.
This removes login/logout/session logic for integrating with OpenNeo ID, replacing them with stubs that just redirect to `/?TODO` when you click login, and helpers that act as if you're not logged in.
This gives us a clean slate to plug in new Devise logic to integrate with the `openneo_id` database directly!
Whew! Seems like a pretty clean one? Ran `rails app:upgrade` and stuff, and made some corrections to keyword arguments for `translate` calls. There might be more such problems elsewhere? But that's hard to search for, and we'll have to see.
This one was pretty straightforward yaay! Main thing was the change from `render file` to `render template` in a couple places, oh and a thing with complex `order()` clauses.
The session format changed, so we change the session cookie name rather than have things crash about it! (I hope the actual prod behavior is to ignore bad cookies rather than crash? But I figure this is more reliable anyway.)
Idk I guess these are the default place to put certain settings, but idk if they're still canonical, and I'd rather just not have files that don't mean anything rn!
At one point we piloted a "Camo" service to proxy HTTPS image urls for us, but it doesn't exist anymore.
We already have proxies and stuff for this, so I left `Image` as a placeholder for this, but it's not working yet!
This also deletes our final reference to the Addressable gem, so we can remove it!
Yay, we've deleted all our background tasks!
We'll probably want to replace some of the basic functionality like certain caching? But we can deal with that as we run into it.
The direct motivation here was a seeming version conflict between Rails 4.2's rack dependency and latest Resque's rack dependency... but this is just nice complexity elimination regardless, we want this anyway :3
We've already swapped out the backend for this stuff to Impress 2020, so the resque task and the broken image report UI aren't actually relevant anymore. Delete them!
This helps us delete Resque soon too.
We'll need to replace the item search query stuff with direct MySQL queries, but that's not ready yet bc the app still isn't booting, so we're committing this in a known broken state for now!
I haven't logged into newrelic in a billion years, let's just stop sending them stuff
(This is a precursor to an attempt to delete flex stuff too and replace our elasticsearch stuff with direct mysql queries like Impress 2020 does, but that'll be more work!)
I guess the APIs changed here, but these were placeholder settings we weren't actually using anyway (cuz we use the OpenNeo ID integration), so I just commented them out and it seems fine for now!
NOTE: This doesn't boot yet! There's something changed in the `devise` API that we'll need to fix!
```
/vagrant/config/initializers/devise.rb:46:in `block in <top (required)>': undefined method `encryptor=' for Devise:Module (NoMethodError)
```
But yeah, we navigated the gem upgrades, and also I ran `rake rails:update` and hand-processed the suggestions it had for our config files.
Rather than figure out how to upgrade the Stripe gem to be compatible with future Rails, I'd rather just delete the references, since it's currently unused.
I'm not so bold as to go in and fully trash all our donation code; I just want to ensure we're not sending people down broken codepaths, and that if they reach them, the error messages are clear enough.
We've been serving images directly from `impress-asset-images.s3.amazonaws.com` for a long time. While they serve with long-lasting HTTP cache headers, and the app requests them with the `updated_at` timestamp in the query string; each GET request still executes a full S3 ReadObject operation to get the latest version.
In the past, this was only relevant to users on Image Mode, not Flash Mode. But now that everyone's on Image Mode, this matters a lot more!
Now, we've configured a Fastly host at `impress-asset-images.openneo.net`, to sit in front of our S3 bucket. This should dramatically reduce the GET requests to S3 itself, as our cache warms up and gains copies of the most common asset PNGs.
That said, I'm not sure how much actual cost impact this change will have. Our AWS console isn't configured to differentiate cost by bucket yet—I've started this process, but it might take a few days to propagate. All I know is that our current costs are $35/mo data transfer + $20/mo storage, and that outfit images are responsible for most of the storage cost. I hypothesize that `impress-asset-images` is responsible for most of the reads and data transfers, but I'm not sure!
In the future, I think we'll be able to bring our AWS costs to near-zero, by:
- Obsolete `impress-asset-images`, by using the official Neopets PNGs instead, after the HTML5 conversion completes.
- Obsolete `impress-outfit-images`, by using a Node endpoint to generate the images, fronted by a CDN cache. (Transfer the actual data to a long-term storage backup, and replace the S3 objects with redirects, so that old S3 URLs will still work.)
I hope this will be a big slice of the costs though! 🤞
(Note: I'll be deploying this on a bit of a delay, because I want to see the DNS propagate across the globe before flipping to a new domain!)
