Preparing to finally move wardrobe-2020's item search to use the main
app's API endpoints instead!
One blocker I forgot about here: Impress 2020 has actual support for
knowing an item's true appearance, like by reading the manifest and
stuff, that we haven't really ported over. I feel like maybe I should
pause and work on the changes to manifest-archiving that I'd been
planning anyway? I'll think about it.
I changed the type of this tag without realizing the JS references it
by both class and `div`!
I think at the time this was a perf suggestion for jQuery, because the
best way to query by class name was to query by tag first then filter?
It's possible our jQuery still does this, but I don't imagine it's very
relevant today, so I'll just remove that for better guarding against
similar bugs in the future instead.
I've moved the support secret into the encrypted credentials file, and
moved the origin into a top-level custom config value in the
environment files, with different defaults per environment but still
the ability to override it. (I don't use this, but it feels polite to
not actually *demand* that people use port 4000, y'know?)
There's a bit happening behind the scenes of this change. Previously,
we kept a `SECRET_TOKEN` environment variable in `production.env`, and
used a `secret_token.rb` initializer to wire it up as the
`secret_key_base`.
In this change, we move to Rails's new-ish (two years old :p) encrypted
credentials system. Now, we set a `RAILS_MASTER_KEY` environment
variable in the deployed `production.env` instead (and in our local
`.env.production` in the project root for managing it), and we can run
`rails credentials:edit` to open the encrypted file in a text editor.
Inside, the content is just:
```yml
secret_key_base: "<OUR_SECRET_KEY>"
```
This indirection doesn't exactly do much for us functionally; it's just
the more standard way of achieving what our `secret_token.rb` situation
was achieving.
We could also migrate other secrets into there, and I just might! That
would simplify duplication between `/deploy/files/production.env` and
`/.env.production`, at any rate! The main notable one is
`MATCHU_EMAIL_PASSWORD` for sending auth emails from
`matchu@openneo.net` (and there's also a Stripe token that we don't
actually use in the app these days, those codepaths are old bones). Oh
and there's also the `IMPRESS_2020_SUPPORT_SECRET`!
Anyway, the motivation for this was to remove the warning when starting
the app that Devise is trying to use the deprecated
`Rails.application.secrets` method. I was expecting to have to do
[the workaround shared here](https://github.com/heartcombo/devise/issues/5644#issuecomment-1804626431),
but it turns out whatever default behavior Devise does under the hood
is happy enough with our new decision to use the credentials file, and
the deprecation warning is gone! Ok neat!
I hadn't realized for a while that we weren't already doing this lol, I
had noticed that `bundle install` in production was slower than I
expected when adding new stuff, but it was when we did this big recent
`bundle update` that I really noticed the difference.
Fixed now, I think! Though the real test will come when we actually
have a new gem to install, since this was a no-op case.
As the comment in `deploy.yml` explains, this was a multi-step process,
but it went very smoothly as planned, hooray!!
I noticed again while making this change that Bundler doesn't seem to
be availing itself of the checked-in dependencies in `vendor/cache`. I
think I know the fix for this, I'll toss it into an upcoming change and
see if it works!
Still need to test the app with it, and getting this to deploy right
will be a bit tricky! Here's my thinking for sequencing once the code
is ready:
1. Temporarily modify `deploy.yml` to push the version, but not set it
as `current` or restart the app.
2. Update the service file to use Ruby 3.3.0 and reference that version
directly (instead of `current`), and restart the app.
3. Once it's already running, link that version as `current`.
4. Update the service file to reference `current` as usual, and restart
the app.
I also put in a manual bump for `falcon`!
The motivation is that I'm working on a Ruby 3.3.0 upgrade in another
branch, and I'm getting deprecation warnings from the `async` gem,
which I think are resolved in the latest version, so I figure, hey,
good time for an update!
Okay, so I still don't know why rendering is just so slow (though
migrating away from item translations did help!), but I can at least
cache entire closet lists as a basic measure.
That way, the first user to see the latest version of a closet list
will still need just as much time to load it… but *only* the ones that
have changed since last time (rather than always the full page), and
then subsequent users get to reuse it too!
Should help a lot for high-traffic lists, which incidentally are likely
to be the big ones belonging to highly active traders!
One big change we needed to make was to extract the `user-owns` and
`user-wants` classes (which we use for trade matches for *the user
viewing the list right now*) out of the cached HTML, and apply them
after with Javascript instead. I always dislike moving stuff to JS, but
the wins here seem. truly very very good, all things considered!
From an era when we didn't have that! Now we do!
(My motivation is that I'm trying to add new JS to this page and errors
in stickUp are crashing the page early, womp womp!)
Okay cool, we're successfully migrated off translations, we can delete
the table now!
I'm not worried about backing up this data as such, because the
impress-2020 repo has a bunch of this data in its
`public-data-from-modeling.sql.gz` file history. Safe to remove from
the live app!
This one is important, I didn't notice that this is a way of setting
attributes that won't be written to both tables! `name` will only be
written to the translation table (which crashes the save), and the
other fields would only be written to the main table. Fixed! (I don't
like the super-dynamic this code was written before, anyway.)
Missed this at first - now that the `name` field is just a normal field
and is always English, it's now an error to provide the locale to it as
a parameter, like we used to for the translated version of the field!
Like with Species, Color, and Zone, we're moving the translation data
directly onto the model, and just using English. This will simplify
some of our queries a lot (way fewer joins!), and it's what Neopets
does now anyway, and I have a secret hope that removing the complexity
along the codepath for `item.name` might help speed up large item lists
if we're lucky?? 🤞
Anyway, this is the first step, performing the migration to copy the
data onto the `items` table, making sure to keep them in sync for the
2020 app for now!
I think this was to explain why `order` wasn't part of this query, and
we probably used to sort in the controller? But now the item search
module takes care of all that, this is just confusing to say now imo!
Impress 2020 has had this for a while, I've wanted it for reference on
occasion, let's bring it in!
Very similar logic, and Ruby & Rails's date affordances are super
helpful for simplifying how to express it!
The homepage used to point to old projects that don't work anymore
anyway! This is the only project that stuck, so just redirect here!
We also remove the openneo.net link from the footer, because there's
nothing useful to say there anymore!
It hasn't been updated in a long time, let's just be rid of it!
It's possible I'll replace it with another blog sometime if we get the
chance to do more development work, it could be a useful way to improve
communication—but not yet!
It's finally colocated onto this box, instead of being on the old
server! I think I'm noticing substantial perf improvements, probably
both from increased colocation (tho they were in the same house
before), and also from like ten years of performance optimizations LOL!
As part of this, I created a new `setup_secrets.yml` file that's
similar to `production.env`, but is for values that the setup script
itself needs access to, whereas `production.env` is for values that the
app needs at runtime. (Though they have some things in common, like the
MySQL user password!) It's gitignored for security, as per usual!
Initially I put this all in the same thing cuz I wasn't sure I could
count on our nginx config to actually serve asset requests correctly…
I've figured it out now!
I think I cleared this from the outfits/new template a while ago, but
never cleaned up this file, because I was too anxious that I was
correctly identifying all its call sites. But now I'm more confident!
At least, they seem unused to me on a quick audit! The scriptaculous
stuff has long been replaced by jQuery UI equivalents. (Wow, so many
generations of libraries! lol)
Mostly this is just me testing out what it would look like to
modularize the app more… I've noticed that some concerns, like
fundraising, are just not relevant to most of the app, and being able
to lock them away inside subfolders feels like it'll help tidy up
long folder lists.
Notably, I haven't touched the models case yet, because I worry that
might be a bit more complex, whereas everything else seems pretty
well-isolated? We'll try it out!
Tbh I'm not sure `special_color` is actually used anywhere? It used to
be how we decide what to show in the previewer on the item page, but
that's been replaced with the 2020 logic, so idk…
But in any case, I noticed that the description doesn't match the
pattern we have, so here's the fix!
I looked at this and was like. "ok literally what is
`nonstandard_colors` trying to do"
reading it again now, I'm realizing the idea is that it probably runs
two queries: one to get nonstandard colors, then depends on
ActiveRecord to implicitly convert the relation to an array and then to
IDs for the second query? Instead of doing a join??
Idk, it's unused, so trash it!
This used to be the behavior, and the site has plenty of graceful
fallbacks for it, I just forgot this one when doing Rails upgrades!
Note that the impress-2020 stuff is *not* as graceful about this, so
the wardrobe page won't show the pet until the color is in the DB. Ah
well, still an improvement!
Today I learned that nginx requires a special invocation to listen to
IPv6 addresses as well as IPv4. On some of my other projects, this was
causing Let's Encrypt certificate renewal to fail, because Let's
Encrypt prefers to connect over IPv6 when an AAAA record is present, so
its challenges were always returning 404, because nginx wasn't
listening on IPv6.
This shouldn't be affecting impress in production, because we don't
have an AAAA record right now. But I'm just making this change in all
my projects, to make sure this doesn't bite me in the future!
I'm not sure where these duplicate records have been coming from over
the years (I checked the timestamps and it's been happening
occasionally since 2013 up to late last year, there were ~1,600
instances), but for now let's just get rid of them!
This is related to the issues we've been addressing lately where some
biology assets have manifests but no PNG specified in them: the older
copies of the assets would have our generated PNG as a fallback, but
the newer copies would get served as part of the pet appearance *in
addition to* the older copies, and the newer copies would be marked as
having no DTI-generated image, which our system wasn't always able to
handle.
We've primarily been addressing this by leaning into more graceful
failure modes of skipping certain layers, but… these layers *shouldn't
be here*, and are cluttering up support tools and such; let's be rid of
them!
I ran this today seemingly without issue, but I kept a backup of the
`yarn db:export:public-data` task in `impress-2020` to be able to check
and rollback if we discover a mistake.
One last note: the `ORDER BY` clause in the `GROUP_CONCAT` call was a
late addition, *after* I ran this in production. Scanning the console
output, it seems like ordering by ID was MySQL's default behavior here
anyway (makes sense!), so I'm not gonna bother to rollback and re-run,
but I think specifying this is helpful to ensure we're not depending on
unspecified behavior and to be really clear about our intentions of
which record to keep (the one with the smallest DTI ID number).
I thought this refactor of this change was working, but actually it was
just failing to build the JS lmao. Here's a version with correct syntax!
😅
Is there a syntax for this kind of thing that I'm just forgetting? Idk,
oh well!
