The modeling code is slow! I think in production it's being cached, and
tbh I though I had development mode caching turned on over here, but
it's quite evidently _not_ doing it if so, so. Okay! Skip for now.
Oh right, we don't have Rails UJS going on anymore, which is what
handled the confirmation prompts for deleting lists. Turbo is the more
standard modern solution to that, and should speed up certain
pageloads, so let's do it!
Here I install the `turbo-rails` gem, then run `rails turbo:install` to
install the `@hotwired/turbo-rails` npm package. Then I move
`application.js` that's run all on pages but the outfit editor into our
section of JS that gets run through the bundler, and add Turbo to it.
I had to fix a couple tricky things:
1. The outfit editor page doesn't play nice with being swapped into the
document, so I make it require a full page reload instead.
2. Prefetching the Sign In link can cause the wrong `return_to` address
to be written to the `session`. (It's a GET request that does, ever
so slightly, take its own actions, oops!) As a simple hacky answer,
we disallow prefetching on that link.
Haven't fixed up the UJS stuff for confirm prompts to use Turbo yet,
that's next!
This `.gif` format is used in the items list "export to petpage"
feature, as the image URL for items whose URLs are known to contain
blocked words that prevent them from being used in petpages.
But when doing some Rails upgrade long ago, we didn't notice the new
security feature that blocks redirects to other sites without a special
flag being set. It was triggering 500 errors, oops.
Now, we set the flag!
I think this is the more canonical place for stuff like this these days!
It's nice to be able to just say the short name when calling `render`.
Here's the answer I looked up about it: https://stackoverflow.com/a/9892081/107415
My immediate motivation is that I'm looking at creating more About
pages, and thinking about where to put them; I think maybe we trash the
`StaticController`, move these partials out to here, and move terms
into a new `AboutController`?
When we moved more logic into the main app, we made some assumptions
about manifest art that were different than Impress 2020's, in hopes
that they would be More Correct for potential future edge cases.
Turns out, they were actually *less* correct for *current* edge cases!
Chips linked us to a few examples, including this Reddit post:
https://www.reddit.com/r/neopets/comments/1b8fd72/i_dont_think_thats_the_correct_image/
Fixed now!
According to our GlitchTip error tracker, every time we deploy, a
couple instances of `Async::Stop` and `Async::Container::Terminate`
come in, presumably because:
1. systemd sends a STOP signal to the `falcon host` process.
2. `falcon host` gives the in-progress requests some time to finish up
3. Sometimes some requests take too long, and so something happens.
(either a timer in Falcon or a KILL signal from systemd, not sure!)
that leads the ongoing requests to finally be terminated by raising
an `Async::Stop` or `Async::Container::Terminate`. (I'm not sure
when each happens, and maybe they happen at different points in the
process? Maybe one happens for the actual long-running ones, vs the
other happens if more requests come in during the meantime but get
caught in the spin-down process?)
4. Rails bubbles up the errors, our Sentry library notices them and
sends them to GlitchTip, the user presumably receives the generic
500 error, and the app can finally close down gracefully.
It's hard for me to validate that this is *exactly* what's happening
here or that my mitigation makes sense, but my logic here is basically,
if these exceptions are bubbling up as "uncaught exceptions" and
spamming up our error log, then the best solution would be to catch
them!
So in this change, we add an error handler for these two error classes,
which hopefully will 1) give users a better experience when this
happens, and 2) no longer send these errors to our logging 🤞❗️
That strange phenomenon where the best way to get a noisy bug out of
your logs is to fix it lmao
Oh rough, when moving an item list entry from one list to another, our
logic to merge their quantities if it's already in that list was just
fully crashing!
That is, moves without anything to merge were working, but moves that
required a merge were raising Internal Server Error 500, because the
`list_id` attribute wasn't present.
I'm not sure why this ever worked, I'm assuming using `list_id` in the
`where` condition would include it in the `select` implicitly in a
previous version of Rails? Or maybe Rails used to have fallback
behavior to run a second query, instead of raising
`MissingAttributeError` like it does now?
Well, in any case, this seems to fix it! Whew!
I don't think people see this very often visually, but it's showing up
in our error logging! The Rails API changed here long ago and we didn't
notice: to render public files, we should use the `file` argument
instead of `template`.
I previously added a warning to the item page, and thought about doing
one here but was sicky and misjudged the complexity and forgot you
don't need to hook into the `knownGlitches` API field to do it! Easy
peasy for a hacky little bug message!
I *think* what I'm observing is that:
1. The zone restrictions are different between these items.
2. The zone restrictions *change* when reloading the page sometimes. (I
assume from remodeling?)
3. The items look very buggy on many pets, because many appearances
seem to expect different zone restrictions than the item actually
has.
I think what this means is:
1. TNT has finally unbound restricted zones from the item level, and
allowed different appearances to have different restrictions. Neat!
2. The API still serves it the same way, as a field on the item.
So I think this means we need to update our schema to reflect the fact
that an item's `zones_restrict` field isn't *really* a property of the
item; it's a property of the combination of the item and the current
body ID.
My gut take here is that maybe this means it's time for the Large
Refactor that I've kinda been interested in for a while, but been
avoiding because of Impress 2020 compatibility issues: instead of a
`body_id` field on assets, and having them directly belong to items,
make an `ItemAppearance` record (closer to how 2020's GQL API modeled
it, I was looking ahead to this possibility!) that's keyed on item and
body ID, and assets belong to *that*.
Then, we could move the zones restriction field onto the
`ItemAppearance` record instead. And then it doesn't really matter to
us how TNT models it internally; whatever we saw is what we use.
(Again, I looked ahead to this in the 2020 app, and tried to use the
`restrictedZones` field on `ItemAppearance` when possible—even though
it secretly just reads directly from the `Item`!)
…but that's a pretty big departure from how things are modeled now, and
isn't something we can just throw together—especially coordinating it
across both apps. I was getting close to being able to shut off 2020
from a *front-facing* perspective (but still keeping a lot of the GQL
endpoints open for the wardrobe-2020 frontend), but I don't think we're
very close to being able to try to target turning off 2020's *backend*
as a prereq to this; or at least, if we do, we should expect that to
take a while. (Counting now, there's still 9 GQL queries—not as many as
I expected tbh, but still quite a few.)
So idk how to sequence this! But for now, let's put out a warning, and
start setting expectations.
The main *intended* user-facing effect of this is that "Items you own"
and "Items you want" filters should work in wardrobe-2020 now!
It is also possible that I messed something up and that this might
break unrelated searches somehow! We'll find out!! 😅
Did this before we had the ability to trigger searches from the app
itself, to allow me to open up the browser JS console and call this
function directly. Now we can just do it in app, goodbye!
Yay, we finally added it, the part where we include the appearance data
for the items based on both the species/color and the alt style! Now,
switching to Faerie Acara correctly filters the search only to items
that would fit (I think literally just only body_id=0 items right now,
but we're not banking on that!)
This only *really* shows up right now in the case where you construct
an Advanced Search form query (which only the wardrobe-2020 app does
now, and in limited form), and we return the query back (which only
gets used by the HTML view for item search, which doesn't have any way
to build one of these requests against it).
This is because, if you just type in `fits:alt-style-87305`, we always
keep your search string the same when outputting it back to you, to
avoid the weirdness of canonicalizing it and changing it up on you in
surprising ways!
But idk, this is just looking forward a bit, and keeping the system's
semantics in place. I hope someday we can bring robust text filter
and Advanced Search stuff back into the main app again, maybe!
I considered this at first, but decided to keep it simple until it
turned out to matter. Oops, it already matters, lol!
I want the item search code to be able to easily tell if the series
name is real or a placeholder, so we can decide whether to build the
filter text in `fits:$series-$color-$species` form or
`fits:alt-style-$id` form.
So in this change, we keep it that `AltStyle#series_name` returns the
placeholder string if none is set, but callers can explicitly ask
whether it's a real series name or not. Will use this in our next
change!
Previously we did this hackily by comparing the ID to a hardcoded list
of IDs, but I think putting this in the database is clearer and more
robust, and it should also help with our upcoming item search stuff
that will filter by it!
Previously, passing in `fits:blue` would cause a crash, because
`species_name` part of the split would be `nil`, oops!
In this change, we use a regex for more explicitness about the pattern
we're trying to match. We'll also add more cases next! (You'll note the
error message mentions `fits:nostalgic-faerie-draik`, which isn't
actually possible yet, but will be!)
Previously, the query wouldn't fill into the search box or page title
if e.g. parsing had failed. Now it does!
I'm not sure why the rescue strategy we previously had here doesn't
work anymore (I'm sure it must've in the past sometime?), but this is
simpler anyway, let's go!
I think this is a bit clearer and lets us clean up some of the syntax a
bit (don't need to always say `filters <<`), and also it will let us
use `return`, which I'm interested in for my next change!
Right, fitting isn't just body_id = this one, it's also body_id=0!
Anyway, doing this query on its own is still deathly slow, I wonder if
the idea I had about left joins (back when I was still working in a
Rails version that didn't support it lol) could help! Might poke at
that a smidge.
Oh right, `imageUrl` is the name of the field relative to what the app
expects, but under the hood `useOutfitAppearance` actually makes that
an alias for `imageUrlV2(idealSize: SIZE_600)`.
So we need to cache it as the same field with the same params, rather
than as just plain `imageUrl`!
This fixes the bug where wearing an item from search would require a
network round-trip and visually remove all items in the meantime.
(Also, none of this issue was visible to most users, because item
search is still feature-flagged onto the old GQL one for most people!)
This makes clicking on search results in the new mode actually work! It
correctly adds it to the outfit, and removes other items.
The thing that's behaving strangely is that, when you add the item, we
visually remove all items until we can finish a fresh network request
for what they should all look like. This probably means that the cache
lookup for `useOutfitAppearance` is not as satisfied with what we cache
here as `findItemConflicts` is? Something to investigate!
It'd be nice to customize the message a bit, but this should be rare
and I'd prefer the simplicity of just going with the default text.
I ran into this when I made a mistake in how I process the return value
of search results, so React Query caught and raised the error via
React, as intended! And I was annoyed that it wasn't logged anywhere,
so that's my motivation for this change—but also, the old message is
pretty meh and has some layout problems anyway.
I feel like this was part of `will_paginate` back before the Rails
community had itself figured out about what belongs in a model?
But yeah, a default per-page value for search results does not belong
here. And I don't think anything references it anymore, because we pass
`per_page` to the `paginate` call in `ItemsController` explicitly! So,
goodbye!
First off, I think our code has converged on a convention of gracefully
returning `nil` for manifest-less situations, so we can do that instead
of raise! And then that lets us just simplify this check to whether
`manifest` is present, instead of `manifest_url`, so we stop crashing
in cases where we get to this point in the code and there's a manifest
URL but not a manifest.
This was a bit tricky! When I initially turned it on, running
`rails swf_assets:manifests:load` would trigger database errors of "oh
no we can't get a connection from the pool!", because too many records
were trying to concurrently save at once.
So now, we give ourselves the ability to say `save_changes: false`, and
then save them all in one batch after! That way, we're still saving by
default in the edge cases where we're downloading and saving a manifest
on the fly, but batching them in cases where we're likely to be dealing
with a lot of them!
Now we're *really* duplicating with Impress 2020's system lol, but I
need a way to not keep trying to load manifests that are actually 404,
which are surprisingly plentiful!
This doesn't actually stop us from loading anything yet, it just tracks
the timestamps and the HTTP status! But next I'll add logic to skip
when it was 4xx recently.
This is both unnecessary now, but also caused a bug in the new search
stuff where searching by zone would pass an extra `locale` argument to
a filter that doesn't need it!
Idk when this regressed exactly, but probably people didn't super
notice because I don't think it's a very common thing to type directly
into the Infinite Closet search box! (It used to be crucial to the old
wardrobe app.)
But I'm using it in the wardrobe app again now, so, fixed!
For now, I'm doing it with a secret feature flag, since I want to be
committing but it isn't all quite working yet!
Search works right, and the appearance data is getting returned, but I
don't have the Apollo Cache integrations yet, which we rely on more
than I remembered!
Also, alt styles will crash it for now!
`is:np` now means "is not NC and is not PB".
Note that it might be good to make NC and PB explicitly mutually
exclusive too? It would complicate queries though, and not matter in
most cases… the Burlap Usul Bow is the only item that we currently
return for `is:pb is:nc`, which is probably because of a rarity issue?
Adding new functionality to the item search JSON endpoint, and adding
an adapter layer to match the GQL format!
Hopefully this will be pretty drop-in-able, we'll see!
Clearing the way to be able to delete the announcement banner, which is
currently the only link!
I feel like there's room to redo the site layout to find a place to
more properly link to this from, but I don't have one yet! And this is
enough of a niche reference that I think this is good enough?
A bit of a hack, because the thing triggering it was also a bit of a
hack? I feel like there's something we gotta do with refactoring how
our multiple concepts of state are managed… but in any case! This seems
to keep basic outfit-loading working, while no longer getting us
trapped in autosave loops!
Here's how I reproduced the bug:
1. Open a saved outfit.
2. Set the browser devtools to apply a latency of 5sec to all requests.
3. Add an item to the outfit, and wait for the autosave to start.
4. While it still says "Saving", remove the item again.
5. Watch how, when the first autosave request comes in, the item is
re-applied to the outfit, then autosave gets stuck looping forever.
The issue was that, when an outfit finishes saving, the change in
outfit data was triggering this effect in `useOutfitState` that was
*meant* to *initialize* local state from the saved outfit, not to keep
them in sync all the time. (In general, when saved outfit data comes
back from the server, we don't want to use it to "fix" local outfit
state in the case of discrepancies, because the most common source of
discrepancy will be the user having made further changes!)
But anyway, one thing I didn't realize is that we *were* depending on
this hacky hook to do more than I thought: it was responsible for
syncing `id` and `appearanceId` to the local state after saving the
outfit. So, I replaced the `rename` action dispatch here with a new
action that explicitly sets all fields the server is responsible for!
Ah yeah, if you're not on the wardrobe page (so we don't need the Alt
Styles UI), and the outfit's `altStyleId` is null (as is the case for
the item preview page), then there's no need to load the alt styles for
that species.
So before this change, going to `/items/123` would include an XHR
request to `/species/<id>/alt-styles.json`, which would not be used for
anything. After this change, that request is no longer sent. Hooray!
The alt styles controller is the one place we use this right now, but
I'm planning to generalize this to loading appearances during item
search, too!
I also add more `only` fields to the alt styles `as_json` call, because
idk it feels like good practice to both 1) say what we need in this
endpoint, rather than rely on default behavior upstream, and 2) to
avoid leaking fields we didn't realize were on there. (And also to
preserve bandwidth, too!)
I think there's no call sites for these anymore, so now I can start
repurposing these methods for the new API endpoints I'm planning! :3
Now, `SwfAsset#image_url` approximately matches Impress 2020 logic: use
the thumbnail PNG from the manifest if one exists, or the Impress 2020
converter for canvas movies, or the old AWS copy generated by gnash if
necessary, or return nil.
I built this API endpoint in anticipation of a change I never actually
made! I'll just remove it for now, leaning toward cleanuppery over
holding onto something I'm not sure about.
I think this used to be used in an API endpoint we've now deleted? I'm
just cleaning up call sites because I intend to refactor the `urls`
method and stuff, so I'm removing cruft that would complicate it!
I'm not certain-certain this is unused, but I did a global search for
`\bimages\b` in the codebase, and didn't find anything that looked like
a match to me!
Doing that sweet, sweet backfill!! It's not exactly *fast*, since
there's about 570k records to work through, but it's pretty good all
things considered! Thanks, surprisingly-reusable async code!
I'm gonna also use this for a task to try to warm up *all* the
manifests in the database! But to start, just a simple one, to prepare
the alt styles page quickly on first run. (This doesn't really matter
in production now that I've already visited the page once, but it helps
when resetting things in dev, and I think more it's about establishing
the pattern!)
The Neopets Media Archive is a service that mirrors `images.neopets.com`
over time! Right now we're starting by just loading manifests, and
using them to replace the hacks we used for determining the Alt Style
PNG and SVG URLs; but with time, I want to load *all* customization
media files, to have our own secondary file source that isn't dependent
on Neopets to always be up.
Impress 2020 already caches manifest files, but this strategy is
different in two ways:
1. We're using the filesystem rather than a database column. (That is,
manifest data is kinda duplicated in the system right now!) This is
because I intend to go in a more file-y way long-term anyway, to
load more than just the manifests.
2. Impress 2020 guesses at the manifest URLs by pattern, and reloads
them on a regular basis. Instead, we use the modeling system: when
TNT changes the URL of a manifest by appending a new `?v=` query
string to it, this system will consider it a new URL, and will load
the new copy accordingly.
Fun fact, I actually have been prototyping some of this stuff in a side
project I'd named `impress-media-server`! It's a little Sinatra app
that indeed *does* save all the files needed for customization, and can
generate lightweight lil preview iframes and images pretty easily. I
had initially been planning this as a separate service, but after
thinking over the arch a bit, I think it'll go smoother to just give
the main app all the same access and awareness—and I wrote it all in
Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over
bit-by-bit!
Anyway, only Alt Styles use this for now, but my motivation is to be
able to use more-correct asset URL logic to be able to finally swap
over wardrobe-2020's item search to impress.openneo.net's item search
API endpoint—which will get "Items You Own" searches working again, and
whittle down one of the last big things Impress 2020 can do that the
main app can't. Let's see how it goes!
Preparing to finally move wardrobe-2020's item search to use the main
app's API endpoints instead!
One blocker I forgot about here: Impress 2020 has actual support for
knowing an item's true appearance, like by reading the manifest and
stuff, that we haven't really ported over. I feel like maybe I should
pause and work on the changes to manifest-archiving that I'd been
planning anyway? I'll think about it.
I changed the type of this tag without realizing the JS references it
by both class and `div`!
I think at the time this was a perf suggestion for jQuery, because the
best way to query by class name was to query by tag first then filter?
It's possible our jQuery still does this, but I don't imagine it's very
relevant today, so I'll just remove that for better guarding against
similar bugs in the future instead.
I've moved the support secret into the encrypted credentials file, and
moved the origin into a top-level custom config value in the
environment files, with different defaults per environment but still
the ability to override it. (I don't use this, but it feels polite to
not actually *demand* that people use port 4000, y'know?)
Okay, so I still don't know why rendering is just so slow (though
migrating away from item translations did help!), but I can at least
cache entire closet lists as a basic measure.
That way, the first user to see the latest version of a closet list
will still need just as much time to load it… but *only* the ones that
have changed since last time (rather than always the full page), and
then subsequent users get to reuse it too!
Should help a lot for high-traffic lists, which incidentally are likely
to be the big ones belonging to highly active traders!
One big change we needed to make was to extract the `user-owns` and
`user-wants` classes (which we use for trade matches for *the user
viewing the list right now*) out of the cached HTML, and apply them
after with Javascript instead. I always dislike moving stuff to JS, but
the wins here seem. truly very very good, all things considered!
From an era when we didn't have that! Now we do!
(My motivation is that I'm trying to add new JS to this page and errors
in stickUp are crashing the page early, womp womp!)
This one is important, I didn't notice that this is a way of setting
attributes that won't be written to both tables! `name` will only be
written to the translation table (which crashes the save), and the
other fields would only be written to the main table. Fixed! (I don't
like the super-dynamic this code was written before, anyway.)
Missed this at first - now that the `name` field is just a normal field
and is always English, it's now an error to provide the locale to it as
a parameter, like we used to for the translated version of the field!
Like with Species, Color, and Zone, we're moving the translation data
directly onto the model, and just using English. This will simplify
some of our queries a lot (way fewer joins!), and it's what Neopets
does now anyway, and I have a secret hope that removing the complexity
along the codepath for `item.name` might help speed up large item lists
if we're lucky?? 🤞
Anyway, this is the first step, performing the migration to copy the
data onto the `items` table, making sure to keep them in sync for the
2020 app for now!
I think this was to explain why `order` wasn't part of this query, and
we probably used to sort in the controller? But now the item search
module takes care of all that, this is just confusing to say now imo!
Impress 2020 has had this for a while, I've wanted it for reference on
occasion, let's bring it in!
Very similar logic, and Ruby & Rails's date affordances are super
helpful for simplifying how to express it!
The homepage used to point to old projects that don't work anymore
anyway! This is the only project that stuck, so just redirect here!
We also remove the openneo.net link from the footer, because there's
nothing useful to say there anymore!
It hasn't been updated in a long time, let's just be rid of it!
It's possible I'll replace it with another blog sometime if we get the
chance to do more development work, it could be a useful way to improve
communication—but not yet!
I think I cleared this from the outfits/new template a while ago, but
never cleaned up this file, because I was too anxious that I was
correctly identifying all its call sites. But now I'm more confident!
