Commit graph

91 commits

Author SHA1 Message Date
fa202af26d Time out if manifest loading takes too long
This hasn't been causing issues as far as I know, I just noticed
*months ago* that I forgot to do this, and have had a sticky note about
it on my desk since then lol.

I tested this by temporarily setting the timeout to `0.5`, and watching
it fail!
2024-04-16 16:18:51 -07:00
cf6921329d Oops, choose the *first* PNG from the manifest, to avoid reference art
When we moved more logic into the main app, we made some assumptions
about manifest art that were different than Impress 2020's, in hopes
that they would be More Correct for potential future edge cases.

Turns out, they were actually *less* correct for *current* edge cases!
Chips linked us to a few examples, including this Reddit post:

https://www.reddit.com/r/neopets/comments/1b8fd72/i_dont_think_thats_the_correct_image/

Fixed now!
2024-03-07 14:16:05 -08:00
fb2bdd6ea5 Fix crash when dealing with 404'd manifests
First off, I think our code has converged on a convention of gracefully
returning `nil` for manifest-less situations, so we can do that instead
of raise! And then that lets us just simplify this check to whether
`manifest` is present, instead of `manifest_url`, so we stop crashing
in cases where we get to this point in the code and there's a manifest
URL but not a manifest.
2024-02-25 16:05:43 -08:00
2cac048158 Save manifest load info when preloading them, too
This was a bit tricky! When I initially turned it on, running
`rails swf_assets:manifests:load` would trigger database errors of "oh
no we can't get a connection from the pool!", because too many records
were trying to concurrently save at once.

So now, we give ourselves the ability to say `save_changes: false`, and
then save them all in one batch after! That way, we're still saving by
default in the edge cases where we're downloading and saving a manifest
on the fly, but batching them in cases where we're likely to be dealing
with a lot of them!
2024-02-25 16:02:36 -08:00
cc33ce1d6e Skip loading manifest if we recently failed
This helps speed up some item search result pages a lot in the new API
endpoint I'm building!
2024-02-25 15:46:50 -08:00
a684c915a9 Track when manifest was last loaded, and what status it returned
Now we're *really* duplicating with Impress 2020's system lol, but I
need a way to not keep trying to load manifests that are actually 404,
which are surprisingly plentiful!

This doesn't actually stop us from loading anything yet, it just tracks
the timestamps and the HTTP status! But next I'll add logic to skip
when it was 4xx recently.
2024-02-25 15:35:04 -08:00
067cee2d41 Oops, fix bug reading manifest assets when .svg is not present
Ah right, I make this mistake a lot when doing `group_by` stuff: if
there's no SVG, then `assets_by_ext[:svg]` is `nil`, not `[]`. Oh well!
2024-02-25 15:02:35 -08:00
c5cd1f2f3d Remove SwfAsset#html5_*_url in favor of SwfAsset#urls
The alt styles controller is the one place we use this right now, but
I'm planning to generalize this to loading appearances during item
search, too!

I also add more `only` fields to the alt styles `as_json` call, because
idk it feels like good practice to both 1) say what we need in this
endpoint, rather than rely on default behavior upstream, and 2) to
avoid leaking fields we didn't realize were on there. (And also to
preserve bandwidth, too!)
2024-02-24 16:29:47 -08:00
e0b5382c3f Update SwfAsset#image_url to use all the latest tech
I think there's no call sites for these anymore, so now I can start
repurposing these methods for the new API endpoints I'm planning! :3

Now, `SwfAsset#image_url` approximately matches Impress 2020 logic: use
the thumbnail PNG from the manifest if one exists, or the Impress 2020
converter for canvas movies, or the old AWS copy generated by gnash if
necessary, or return nil.
2024-02-24 16:12:02 -08:00
829d960d70 Move SwfAsset#override_body_id to the bottom of the file
This is a console-y method, it's not important enough to be up here!
2024-02-24 15:24:24 -08:00
cf1cdf7e7d Reorder the higher-up lines of SwfAsset
A lot of stuff was mixed together, including methods interrupting the
scopes and relationships, oops! Fixed!
2024-02-24 15:22:14 -08:00
e8737a520e Remove more unused SwfAsset scopes
Wowzers, a lot of cruft in here! I hope I'm not deleting anything
important, but I think my Ctrl-F audits are checking out?
2024-02-24 15:19:28 -08:00
7fee4e6e41 Remove unused SwfAsset.fitting_standard_body_ids and support methods 2024-02-24 15:16:25 -08:00
c25e015b70 Delete some more unused SwfAsset methods 2024-02-24 15:15:21 -08:00
f8cf8165df Remove unused SwfAsset#images method
I think this used to be used in an API endpoint we've now deleted? I'm
just cleaning up call sites because I intend to refactor the `urls`
method and stuff, so I'm removing cruft that would complicate it!

I'm not certain-certain this is unused, but I did a global search for
`\bimages\b` in the codebase, and didn't find anything that looked like
a match to me!
2024-02-24 15:15:21 -08:00
b7f4c5b6ac Remove asset_hosts initializer
It just contains one constant, used in one place. Inlined and deleted!
2024-02-24 15:05:37 -08:00
992954ce89 Create swf_assets:manifests:load task to save all manifest files
Doing that sweet, sweet backfill!! It's not exactly *fast*, since
there's about 570k records to work through, but it's pretty good all
things considered! Thanks, surprisingly-reusable async code!
2024-02-23 14:06:49 -08:00
9a3b33ea2f Preload many manifests concurrently for the Alt Styles page
I'm gonna also use this for a task to try to warm up *all* the
manifests in the database! But to start, just a simple one, to prepare
the alt styles page quickly on first run. (This doesn't really matter
in production now that I've already visited the page once, but it helps
when resetting things in dev, and I think more it's about establishing
the pattern!)
2024-02-23 13:45:12 -08:00
2cc46703b9 Create NeopetsMediaArchive, read the actual manifests for Alt Styles
The Neopets Media Archive is a service that mirrors `images.neopets.com`
over time! Right now we're starting by just loading manifests, and
using them to replace the hacks we used for determining the Alt Style
PNG and SVG URLs; but with time, I want to load *all* customization
media files, to have our own secondary file source that isn't dependent
on Neopets to always be up.

Impress 2020 already caches manifest files, but this strategy is
different in two ways:

1. We're using the filesystem rather than a database column. (That is,
   manifest data is kinda duplicated in the system right now!) This is
   because I intend to go in a more file-y way long-term anyway, to
   load more than just the manifests.
2. Impress 2020 guesses at the manifest URLs by pattern, and reloads
   them on a regular basis. Instead, we use the modeling system: when
   TNT changes the URL of a manifest by appending a new `?v=` query
   string to it, this system will consider it a new URL, and will load
   the new copy accordingly.

Fun fact, I actually have been prototyping some of this stuff in a side
project I'd named `impress-media-server`! It's a little Sinatra app
that indeed *does* save all the files needed for customization, and can
generate lightweight lil preview iframes and images pretty easily. I
had initially been planning this as a separate service, but after
thinking over the arch a bit, I think it'll go smoother to just give
the main app all the same access and awareness—and I wrote it all in
Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over
bit-by-bit!

Anyway, only Alt Styles use this for now, but my motivation is to be
able to use more-correct asset URL logic to be able to finally swap
over wardrobe-2020's item search to impress.openneo.net's item search
API endpoint—which will get "Items You Own" searches working again, and
whittle down one of the last big things Impress 2020 can do that the
main app can't. Let's see how it goes!
2024-02-23 12:02:39 -08:00
3f449310d6 Refactor item search JSON, add appearances
Preparing to finally move wardrobe-2020's item search to use the main
app's API endpoints instead!

One blocker I forgot about here: Impress 2020 has actual support for
knowing an item's true appearance, like by reading the manifest and
stuff, that we haven't really ported over. I feel like maybe I should
pause and work on the changes to manifest-archiving that I'd been
planning anyway? I'll think about it.
2024-02-23 10:44:50 -08:00
b50bf6ceb8 Add SVG support to alt styles
Just moving our hacks around and adapting them to the SVG case lol!
2024-01-31 03:02:19 -08:00
c2de6f7167 Display alt styles in outfit editor when selected
Yay it works(*)! But two major missing pieces:

- Outfit saving doesn't persist it at all
- Item compatibility is unaffected: items will still appear in search
  and in the preview, even when they don't fit anymore.
2024-01-30 07:01:03 -08:00
5004142dfb Add alt style support to modeling
Nothing to show them yet, but I think this works for loading it all in
the first place?

Still needs contributions tho!
2024-01-24 03:25:23 -08:00
c725a11920 Remove unused NC Mall spider code & fields
We haven't used the mall spider in this app in forever (I guess we even
deleted the code at some point?), but there was some vestigial stuff
left. Goodbye!
2023-11-11 15:45:38 -08:00
6c31094c2d Add known_glitches to appearances JSON 2023-11-11 07:21:13 -08:00
010f64264f Replace swf_assets#index with item_appearances#index
Preparing a better endpoint for wardrobe-2020 to use! I deleted the
now-unused swf_assets#index endpoint, and replaced it with an
"appearances" concept that isn't exactly reflected in the database
models but is a _lot_ easier for clients to work with imo.

Note that this was a big part of the motivation for the recent
`manifest_url` work—in this draft, I'm probably gonna have the client
request the manifest, rather than use impress-2020's trick of caching
it in the database! There's a bit of a perf penalty, but I think that's
a simpler starting point, and I have a hunch I'll be able to make up
the perf difference once we have the impress-media-server managing more
of these responsibilities.
2023-11-11 07:14:48 -08:00
d35b10c1b8 Oops, fix validation bug for SwfAsset
Oh right, saving an asset that doesn't have a manifest will crash this
validation step!
2023-11-10 16:16:32 -08:00
96998643b5 Add manifest_url to swf_assets table
Ok so, impress-2020 guesses the manifest URL every time based on common
URL patterns. But the right way to do this is to read it from the
modeling data! But also, we don't have a great way to get the modeling
data directly. (Though as I write this, I guess we do have that
auto-modeling trick we use in the DTI 2020 codebase, I wonder if that
could work for this too?)

So anyway, in this change, we update the modeling code to save the
manifest URL, and also the migration includes a big block that attempts
to run impress-2020's manifest-guessing logic for every asset and save
the result!

It's uhh. Not fast. It runs at about 1 asset per second (a lot of these
aren't cache hits), and sometimes stalls out. And we have >600k assets,
so the estimated wall time is uhh. Seven days?

I think there's something we could do here around like, concurrent
execution? Though tbqh with the nature of the slowness being seemingly
about hitting the slow underlying images.neopets.com server, I don't
actually have a lot of faith that concurrency would actually be faster?

I also think it could be sensible to like… extract this from the
migration, and run it as a script to infer missing manifest URLs. That
would be easier to run in chunks and resume if something goes wrong.
Cuz like, I think my reasoning here was that backfilling this data was
part of the migration process… but the thing is, this migration can't
reliably get a manifest for everything (both cuz it depends on an
external service and cuz not everything has one), so it's a perfectly
valid migration to just leave the column as null for all the rows to
start, and fill this in later. I wish I'd written it like that!

But anyway, I'm just running this for now, and taking a break for the
night. Maybe later I'll come around and extract this into a separate
task to just try this on all assets missing manifests instead!
2023-11-09 21:42:51 -08:00
d8b0bf3174 Skip item translations when modeling
Okay, this is a process that idk if it's even been working for a while anyway, I don't think Neopets translates item names anymore?

And it's crashing when I try to model stuff now, so like. yeah ok I'm fine with just skipping this, it's a shame to lose out on potential data going forward but *I think there just isn't data to get anyway*
2023-10-23 19:05:10 -07:00
5f3ce1210a Stop saving local copies of SWFs
I think we used this for both conversion to image, and also for CORS stuff when rendering Flash-based previews… let's trash it, I don't want to be growing our hard drive with files I don't think we use anymore!

If I'm wrong and it turns out we do use them for something, then like. hey I'm sure we'll find out soon enough, and it's very recoverable operation.
2023-10-23 19:05:10 -07:00
966a0025e5 Fix SWF downloads in modeling
A lot simpler with latest Ruby! We can drop the whole SSL workaround yaay
2023-10-23 19:05:10 -07:00
d97c32b5da Upgrade to Rails 5.2.8.1
Some important little upgrades but mostly straightforward!

Note that there's still a known issue where item searches crash, I was hoping that this was a bug in Rails 4.2 that would be fixed on upgading to 5, but nope, oh well!

Also uhh I just got a bit silly and didn't actually mean to go all the way to 5.2 in one go, I had meant to start at 5.0… but tbh the 5.1 and 5.2 changes seem small, and this seems to be working, so. Yeah ok let's roll!
2023-10-23 19:05:05 -07:00
f80d220a62 Upgrade to Rails 4.2.11.3 and Ruby 2.4.10
Some tricks required here to get the dependencies to work out, but we got it!!

Oh also, we move away from the rbenv in Ubuntu's package manager, because it doesn't support more recent Rubies like 2.4.10.
2023-10-23 19:05:05 -07:00
0d4c6ca077 Delete utf8 backfill for old Ruby
Idk if the replacement `require` is necessary exactly, but it's the one-to-one replacement for this lib, so let's start there for now!
2023-10-23 19:05:05 -07:00
c4f3a472ff Remove RightAWS gem
We're not connecting to AWS directly anymore, now that we deleted the SWF conversion stuff, so we can finally clear this out!
2023-10-23 19:05:05 -07:00
Matchu
93c596007d Clarify image path methods for SwfAsset
Using `s3_path` and stuff made it sound like we were still referencing the original Amazon S3 images - but actually our new asset proxy just uses the same path structure, and we didn't change anything about it.

Oh also I deleted an after_conversion method that isn't used anymore, forgot about that!
2023-10-23 19:05:05 -07:00
Matchu
e121d8bba2 Remove SWF conversion
We've already swapped out the backend for this stuff to Impress 2020, so the resque task and the broken image report UI aren't actually relevant anymore. Delete them!

This helps us delete Resque soon too.
2023-10-23 19:05:04 -07:00
Matchu
c581b063c4 Oops, fix not_occupies logic error
Right, previously we were querying "has *at least one asset* that is not in zone X" instead of "has NO assets that are in zone X".

I don't know a fast way to query for that, this will have to do for now!
2023-10-23 19:05:03 -07:00
Matchu
a29e016555 Update to new scope syntax
Ohh ok, without this change all of our `scope`s were just immediately evaluating the argument and fetching _all_ such matching records immediately, instead of waiting to actually be called. This led to bugs like `pet_type.as_json` returning ALL pet states in the whole db, because the `PetState.emotion_order` scope was being treated as a single predefined query, rather than a query fragment to merge into the current context.

This also explains what happened in 724ed83: that's why things before the scope in the query were being ignored.
2023-10-23 19:05:03 -07:00
Matchu
aa3d28c641 Update syntax for set_inheritance_column 2023-10-23 19:05:02 -07:00
4f357c2f9c
Use images without awaiting conversion anymore
We set up `impress-asset-images.openneo.net` to redirect to the right asset, without needing to depend on AWS anymore for HTML5-converted items!

Our quick fix for this: always serve `has_image: true` to the frontend, so it always tries to use the image, regardless of whether we've marked it as converted in the database. (We've turned off the converters too!)
2022-10-15 16:26:12 -07:00
Emi Dunn-Rankin
fe9adb5766
Oops, fix mall spider bug, added by our HTTPS fix
Oh, yeah, shit, okay, when we set `self.url` like that, it's supposed to be the _canonical_ URL for the SWF, not our proxied one—this is the URL that's gonna go in the database.

We do proxying late in the process, like when we're actually setting up to download something, but for just referencing where the asset lives, we use `images.neopets.com`.

In this change, we revert the use of `NEOPETS_IMAGES_URL_ORIGIN`, but we _do_ update this to `https` for good measure. (We currently have both HTTP and HTTPS urls in the database, I guess neopets.com started serving different URLs at some point, this is probably the future! And anything interpreting these URLs will need to handle both cases anyway, unless we do some kind of migration update situation thing.)

We're migrating the incorrect assets with the following query (with the limit changed to match the number we currently see in the DB, just as a safety check):
```
UPDATE swf_assets SET url = REPLACE(url, 'http://images.neopets-asset-proxy.openneo.net', 'https://images.neopets.com') WHERE url LIKE 'http://images.neopets-asset-proxy.openneo.net%' ORDER BY id LIMIT 2000;
```
2022-08-23 03:04:54 -07:00
Emi Dunn-Rankin
bc64164d69
Sigh, fix HTTPS for images AGAIN with a proxy now
Okay, like in the previous commit, we're dealing with forced HTTPS, on a server that isn't going to cooperate with our dependencies' HTTPS version. And this time, I don't think there's a secret origin server that will accept `http://` requests for us.

Thankfully, we have the perfect hack in our back pocket: our own pre-existing images.neopets.com proxy server! I set the following in our secret `.env` file, and now we're good:

```
NEOPETS_IMAGES_URL_ORIGIN=http://images.neopets-asset-proxy.openneo.net
```
2022-08-02 21:17:52 -07:00
9540e2122b
Disable HTTPS for SWF asset downloads
Whoops, everything broke because the world is far ahead of us on security! Oh well.
2022-07-23 21:44:17 -07:00
aceffc56ab
Fix SWF downloading for HTTPS URLs
TNT started using HTTPS URLs! And our old Ruby version (lol 😬) still requires explicit invocation to perform SSL during a request, so requests were failing!

Now, we explicitly build the `Net::HTTPS` object, and turn on `use_ssl` if it's an HTTPS URL! (The shorthand invocation didn't seem to have an option for this, that I could find!)
2021-06-12 02:23:18 -07:00
238a458131
Fix body ID bug saving SwfAsset outside modeling
Oops, if you saved `SwfAsset` outside of modeling code, the `item` field would be empty, and so `item.body_specific?` wouldn't happen.

This would trigger when you even just report a broken image!

Now, we always run the SQL query to check for that flag.
2021-03-16 10:40:01 -07:00
Matt Dunn-Rankin
5cae876c46 ugh, one more bug fix? 2017-10-29 15:18:53 -07:00
Matchu
4a539f32db hope I did this swfimages fix right xD 2015-08-05 20:22:23 -04:00
b0cc4c2396 swf links 2015-05-03 16:57:42 -05:00
6984201990 dev util method to manually change SWF asset body ID 2013-06-26 20:08:19 -07:00