Today I learned that nginx requires a special invocation to listen to
IPv6 addresses as well as IPv4. On some of my other projects, this was
causing Let's Encrypt certificate renewal to fail, because Let's
Encrypt prefers to connect over IPv6 when an AAAA record is present, so
its challenges were always returning 404, because nginx wasn't
listening on IPv6.
This shouldn't be affecting impress-2020 in production, because we
don't have an AAAA record right now. But I'm just making this change in
all my projects, to make sure this doesn't bite me in the future!
We ran the new `rails swf_assets:remove_duplicates` task in the
`impress` repository, saving our work before (previous commit) and
after (this commit), in case we need to study the results or roll back!
Sometimes we don't have an image for an SWF asset. This can happen when
assets were converted to HTML5 without a PNG specified in the manifest.
(One example is bio/363, the Hind Biology for the Mutant Blumaroo.)
I'm noticing there's a second issue here on DTI's end, where it looks
like *we* have two copies of certain layers? I wonder if this was a bug
in like, the impress-2020 modeling code we tested at one point? Or if
this is in our main modeling code? e.g. the Mutant Blumaroo's bio/363
is in our database with DTI ID 192 and DTI ID 606955, and only the
former is registered as having an image (which we made ourselves and
host on S3).
So, I'm gonna start by just having the more graceful failure mode of
skipping the missing layer. But I'm also concerned about the root here,
I'll investigate!
I don't think this actually affects any clients in practice, but now
that `PetAppearance` can be influenced not just by pet state ID but
also alt style ID, it's no longer correct to just use pet state ID as
the `id` (the default cache key). This could theoretically cause an alt
style to be locally cached as the default appearance of a
species/color/pose.
Instead, we now append an extra string in the case that there's also an
alt style influencing the appearance. This is a bit of a mess, because
I know there's some tooling that's expecting this field to be strictly
the pet state ID, mainly for support/debugging purposes I think? But I
imagine that's not a big deal in practice, and this change should
narrow the scope of a bug like that to just be like, "error: pet state
123-with-alt-style-456 not found" when trying to save a change to it
or something. Good enough!
Oops, before this change, outfits with alt styles would still show the
outfit as if no alt style were applied!
Now, we have the `Outfit` GraphQL type be internally aware of alt
styles, and set its `petAppearance` and `body` and `itemAppearances`
fields accordingly. No change was required to the actual
`/api/outfitImage` endpoint, once the GraphQL started returning the
right thing!
…because of that, I'm honestly kinda surprised that there's no obvious
issues arising with the Impress 2020 outfit interface itself? But it
seems to be correctly just, not showing alt styles at all, in the way I
intended because I never added support to it. So, okay, cool!
We're in the process of migrating away from translating these records,
because Neopets hasn't supported non-English languages in many years,
and it'll simplify our code and database lookups.
In Main DTI, we already wrote code to copy these fields onto the main
records and keep them in sync for now; now, once DTI 2020 isn't
referencing them anymore, it should be safe for the main app to drop
the tables altogether.
Note that some Prettier changes got mixed in here and that's fine!
I also wasn't suuuper careful testing these, most of them seem to be
trivially testable by just loading the homepage or doing a few basic
wardrobe actions, and the others are in Discord support log actions
that aren't enabled in development mode, so I'm just like… ehh I'll do
a couple support actions after deploy and see that they don't crash!
I'm not planning to port full Alt Style support over to the 2020
frontend, I really am winding that down, but adding a couple lil API
parameters are *by far* the easiest way to get Alt Styles working in
the main app because of how it calls the 2020 API.
So here we are, adding new API calls but not the frontend changes!
Note also the change to the public-data-constants, because we've
started (but not finished) moving name fields into the rows themselves
and deprecating the translations.
I did a pretty thorough reset of my desktop machine, and rather than go
spelunking for the same private key, I just rolled it over to a new
one. Let's set it up!
Self-hosted Plausible instance! I have need of usage numbers again,
after a good few years of just not using it; but I don't want to send
the data to Google, and I enjoy self-hosting things, so here we have it!
I actually forgot we were still sending the DTI 2020 numbers to
Plausible's cloud instance, oops! I stopped paying for it 😅 They do a
pretty good job anonymizing, so I'm not too worried about the practical
effects of that, but I'll be pleased to keep it more in-house going
forward.
I'm not sure why `async` isn't in the recommended tag anymore, but fine
by me!
Weird! Well! This caused two issues.
One is that we used to filter down to only assets whose urls end in
`.swf`, because I never added support for the sound ones :p
The other is that we were using the SWF URL to infer the potential
manifest URLs, which we can't do when the SWF URL is a weird
placeholder!
Thankfully, just yesterday (wow!) we happened to have Classic DTI keep
track of `manifest_url` during the modeling process, and we backfilled
everything. So the most recent `temp` assets have their manifest! But
slightly older ones (not that old tho!!) didn't, so I manually inferred
the manifest URL from the asset's `remote_id` field instead (which
worked cuz there was no hash component to the manifest URL, unlike some
manifests).
The item "Flowing Black Cloak" (86668) on the Zafara is the one we were
looking at and testing with!
Lol I guess we've probably just been having intermittent CORS issues
forever, oops. I hope the cache has been mostly warmed on the right
thing! But today I checked and the entire species/color picker on the
Rails app wasn't working for CORS reasons, so like. Yeah oof.
I didn't really want to do this exactly, but I think installing a global
Typescript binary caused some problems here, where running the app had
started giving me an error like "you need to install `@types/react`!"
and I'm like. I did? years ago?
I searched for the error and people said "oh just upgrade Next" so I did
and now it's fixed I guess? I only bumped up to the latest version of
v12, rather than up to v14 where things are now, to avoid breakages. I
poked around the site ever so slightly and it seems fine, so I'm not
worried about it!
If you delete a closet list without deleting the items in it, it'll dump them into your default own/want lists. This isn't necessarily the behavior we really want, but oh well, that's the truth right now!
The GraphQL endpoint was assuming that all hanger `list_id` values were valid, oops! Now, we ignore list IDs that don't match a list.
Oh beans, I made an env variable change that I thought would switch us over to db.impress.openneo.net, and I was just plain wrong, darn. impress-2020 has been fully failing since then, oops!!!
Here, we change it to be an environment variable, so that in the future it will work how I want it to lol
Oops, we added behavior that varies the CORS response headers according to the incoming `Origin` header, but we forgot to add `Vary: Origin`!
This doesn't cause an issue for the app when you make requests to the server directly, but since it's behind a Fastly cache layer, we ended up caching responses that didn't include CORS headers but should have.
Now, this will instruct the Fastly cache to treat requests with different `Origin` headers as being entirely different. (This means we won't be sharing caches between requests from impress-2020 and the Rails app anymore, but that should be okay in practice!)
This was Dice's idea ty!! Now, instead of crashing when looking up any pet whose name starts with a number, we do a clever lil workaround instead! We don't get *all* the data back (we're missing metadata), but that's fine for the main use case of typing your pet name in at the homepage.
Sigh, I guess xmlrpc was deleted? Back to the method that doesn't work for pets with leading digits in their names, sobbe
Dice has a neat idea for how to work around that, but I'm not sure how to fit it in our architecture, let's take a look!
One problem I run into with the archive task is that sometimes the queries time out? My hunch is that maybe some of the assets have like, weirdly big manifest files that are being transferred as surprisingly big text files?
Anyway, I'm increasing the timeout to 20s, which is big, but big feels good for a script that doesn't run often and where failing is not great news!
I'm also idly considering whether I wanna finally put in the work to do a bulk S3 uploader sometime, because the current version that iterates over multiple `aws s3 cp` calls is just real slow, I think because it establishes a new connection each time, and that operation is maybe surprisingly expensive? And the CLI doesn't have a way to do multiple uploads any more precisely than "sync up this whole folder", which is slow when the folder contains a lot of stuff you _know_ you don't want to head up there.
Tested this out and compared to Dice's other work in Neobot and I think the condition should be the other way around, as it is here? (I found myself starting to write the explanatory comment, and realizing it wasn't making sense, then going heyyy wait a minute lol)
I tried it in my lil Ubuntu WSL box and hey, it worked great! Neat!!
Pretty neat to have just sat down and fixed up the dev environment for other people?? I'll see about what it would take to actually invite people in…
Now, someone with production DB access can run `yarn db:export:public-data` to create `public-data-constants.sql` and `public-data-from-modeling.sql`.
Then, someone setting up their dev database can run `yarn db:setup-dev:full` and get all the wearables data imported right into their dev database!
I'm noticing just how poorly I'm keeping up with my own goals for finishing up DTI, and wondering if now is a good time to circle back to some old offers for code contributions I got last year… I also just figure that making this app Possible To Run with a backup of the basic public database is like. a pretty handy thing to have for archival's sake imo
Note that, for this change, we also set up Git LFS (Large File Storage). Github should be automatically compatible with this! It's a way to not write the whole 30MB database dump into the repository history, and instead keep it in a secondary filestore, because Git's core algorithms aren't really built to handle large blobs of data very well. Users setting up their dev environment will therefore also need to have Git LFS installed for this script to work! (Otherwise, they'll see a "pointer" file in `public-data-from-modeling.sql.gz` that contains some metadata about the file state but not the data itself.)
