Commit graph

69 commits

Author SHA1 Message Date
5004142dfb Add alt style support to modeling
Nothing to show them yet, but I think this works for loading it all in
the first place?

Still needs contributions tho!
2024-01-24 03:25:23 -08:00
c725a11920 Remove unused NC Mall spider code & fields
We haven't used the mall spider in this app in forever (I guess we even
deleted the code at some point?), but there was some vestigial stuff
left. Goodbye!
2023-11-11 15:45:38 -08:00
6c31094c2d Add known_glitches to appearances JSON 2023-11-11 07:21:13 -08:00
010f64264f Replace swf_assets#index with item_appearances#index
Preparing a better endpoint for wardrobe-2020 to use! I deleted the
now-unused swf_assets#index endpoint, and replaced it with an
"appearances" concept that isn't exactly reflected in the database
models but is a _lot_ easier for clients to work with imo.

Note that this was a big part of the motivation for the recent
`manifest_url` work—in this draft, I'm probably gonna have the client
request the manifest, rather than use impress-2020's trick of caching
it in the database! There's a bit of a perf penalty, but I think that's
a simpler starting point, and I have a hunch I'll be able to make up
the perf difference once we have the impress-media-server managing more
of these responsibilities.
2023-11-11 07:14:48 -08:00
d35b10c1b8 Oops, fix validation bug for SwfAsset
Oh right, saving an asset that doesn't have a manifest will crash this
validation step!
2023-11-10 16:16:32 -08:00
96998643b5 Add manifest_url to swf_assets table
Ok so, impress-2020 guesses the manifest URL every time based on common
URL patterns. But the right way to do this is to read it from the
modeling data! But also, we don't have a great way to get the modeling
data directly. (Though as I write this, I guess we do have that
auto-modeling trick we use in the DTI 2020 codebase, I wonder if that
could work for this too?)

So anyway, in this change, we update the modeling code to save the
manifest URL, and also the migration includes a big block that attempts
to run impress-2020's manifest-guessing logic for every asset and save
the result!

It's uhh. Not fast. It runs at about 1 asset per second (a lot of these
aren't cache hits), and sometimes stalls out. And we have >600k assets,
so the estimated wall time is uhh. Seven days?

I think there's something we could do here around like, concurrent
execution? Though tbqh with the nature of the slowness being seemingly
about hitting the slow underlying images.neopets.com server, I don't
actually have a lot of faith that concurrency would actually be faster?

I also think it could be sensible to like… extract this from the
migration, and run it as a script to infer missing manifest URLs. That
would be easier to run in chunks and resume if something goes wrong.
Cuz like, I think my reasoning here was that backfilling this data was
part of the migration process… but the thing is, this migration can't
reliably get a manifest for everything (both cuz it depends on an
external service and cuz not everything has one), so it's a perfectly
valid migration to just leave the column as null for all the rows to
start, and fill this in later. I wish I'd written it like that!

But anyway, I'm just running this for now, and taking a break for the
night. Maybe later I'll come around and extract this into a separate
task to just try this on all assets missing manifests instead!
2023-11-09 21:42:51 -08:00
d8b0bf3174 Skip item translations when modeling
Okay, this is a process that idk if it's even been working for a while anyway, I don't think Neopets translates item names anymore?

And it's crashing when I try to model stuff now, so like. yeah ok I'm fine with just skipping this, it's a shame to lose out on potential data going forward but *I think there just isn't data to get anyway*
2023-10-23 19:05:10 -07:00
5f3ce1210a Stop saving local copies of SWFs
I think we used this for both conversion to image, and also for CORS stuff when rendering Flash-based previews… let's trash it, I don't want to be growing our hard drive with files I don't think we use anymore!

If I'm wrong and it turns out we do use them for something, then like. hey I'm sure we'll find out soon enough, and it's very recoverable operation.
2023-10-23 19:05:10 -07:00
966a0025e5 Fix SWF downloads in modeling
A lot simpler with latest Ruby! We can drop the whole SSL workaround yaay
2023-10-23 19:05:10 -07:00
d97c32b5da Upgrade to Rails 5.2.8.1
Some important little upgrades but mostly straightforward!

Note that there's still a known issue where item searches crash, I was hoping that this was a bug in Rails 4.2 that would be fixed on upgading to 5, but nope, oh well!

Also uhh I just got a bit silly and didn't actually mean to go all the way to 5.2 in one go, I had meant to start at 5.0… but tbh the 5.1 and 5.2 changes seem small, and this seems to be working, so. Yeah ok let's roll!
2023-10-23 19:05:05 -07:00
f80d220a62 Upgrade to Rails 4.2.11.3 and Ruby 2.4.10
Some tricks required here to get the dependencies to work out, but we got it!!

Oh also, we move away from the rbenv in Ubuntu's package manager, because it doesn't support more recent Rubies like 2.4.10.
2023-10-23 19:05:05 -07:00
0d4c6ca077 Delete utf8 backfill for old Ruby
Idk if the replacement `require` is necessary exactly, but it's the one-to-one replacement for this lib, so let's start there for now!
2023-10-23 19:05:05 -07:00
c4f3a472ff Remove RightAWS gem
We're not connecting to AWS directly anymore, now that we deleted the SWF conversion stuff, so we can finally clear this out!
2023-10-23 19:05:05 -07:00
Matchu
93c596007d Clarify image path methods for SwfAsset
Using `s3_path` and stuff made it sound like we were still referencing the original Amazon S3 images - but actually our new asset proxy just uses the same path structure, and we didn't change anything about it.

Oh also I deleted an after_conversion method that isn't used anymore, forgot about that!
2023-10-23 19:05:05 -07:00
Matchu
e121d8bba2 Remove SWF conversion
We've already swapped out the backend for this stuff to Impress 2020, so the resque task and the broken image report UI aren't actually relevant anymore. Delete them!

This helps us delete Resque soon too.
2023-10-23 19:05:04 -07:00
Matchu
c581b063c4 Oops, fix not_occupies logic error
Right, previously we were querying "has *at least one asset* that is not in zone X" instead of "has NO assets that are in zone X".

I don't know a fast way to query for that, this will have to do for now!
2023-10-23 19:05:03 -07:00
Matchu
a29e016555 Update to new scope syntax
Ohh ok, without this change all of our `scope`s were just immediately evaluating the argument and fetching _all_ such matching records immediately, instead of waiting to actually be called. This led to bugs like `pet_type.as_json` returning ALL pet states in the whole db, because the `PetState.emotion_order` scope was being treated as a single predefined query, rather than a query fragment to merge into the current context.

This also explains what happened in 724ed83: that's why things before the scope in the query were being ignored.
2023-10-23 19:05:03 -07:00
Matchu
aa3d28c641 Update syntax for set_inheritance_column 2023-10-23 19:05:02 -07:00
4f357c2f9c
Use images without awaiting conversion anymore
We set up `impress-asset-images.openneo.net` to redirect to the right asset, without needing to depend on AWS anymore for HTML5-converted items!

Our quick fix for this: always serve `has_image: true` to the frontend, so it always tries to use the image, regardless of whether we've marked it as converted in the database. (We've turned off the converters too!)
2022-10-15 16:26:12 -07:00
Emi Dunn-Rankin
fe9adb5766
Oops, fix mall spider bug, added by our HTTPS fix
Oh, yeah, shit, okay, when we set `self.url` like that, it's supposed to be the _canonical_ URL for the SWF, not our proxied one—this is the URL that's gonna go in the database.

We do proxying late in the process, like when we're actually setting up to download something, but for just referencing where the asset lives, we use `images.neopets.com`.

In this change, we revert the use of `NEOPETS_IMAGES_URL_ORIGIN`, but we _do_ update this to `https` for good measure. (We currently have both HTTP and HTTPS urls in the database, I guess neopets.com started serving different URLs at some point, this is probably the future! And anything interpreting these URLs will need to handle both cases anyway, unless we do some kind of migration update situation thing.)

We're migrating the incorrect assets with the following query (with the limit changed to match the number we currently see in the DB, just as a safety check):
```
UPDATE swf_assets SET url = REPLACE(url, 'http://images.neopets-asset-proxy.openneo.net', 'https://images.neopets.com') WHERE url LIKE 'http://images.neopets-asset-proxy.openneo.net%' ORDER BY id LIMIT 2000;
```
2022-08-23 03:04:54 -07:00
Emi Dunn-Rankin
bc64164d69
Sigh, fix HTTPS for images AGAIN with a proxy now
Okay, like in the previous commit, we're dealing with forced HTTPS, on a server that isn't going to cooperate with our dependencies' HTTPS version. And this time, I don't think there's a secret origin server that will accept `http://` requests for us.

Thankfully, we have the perfect hack in our back pocket: our own pre-existing images.neopets.com proxy server! I set the following in our secret `.env` file, and now we're good:

```
NEOPETS_IMAGES_URL_ORIGIN=http://images.neopets-asset-proxy.openneo.net
```
2022-08-02 21:17:52 -07:00
9540e2122b
Disable HTTPS for SWF asset downloads
Whoops, everything broke because the world is far ahead of us on security! Oh well.
2022-07-23 21:44:17 -07:00
aceffc56ab
Fix SWF downloading for HTTPS URLs
TNT started using HTTPS URLs! And our old Ruby version (lol 😬) still requires explicit invocation to perform SSL during a request, so requests were failing!

Now, we explicitly build the `Net::HTTPS` object, and turn on `use_ssl` if it's an HTTPS URL! (The shorthand invocation didn't seem to have an option for this, that I could find!)
2021-06-12 02:23:18 -07:00
238a458131
Fix body ID bug saving SwfAsset outside modeling
Oops, if you saved `SwfAsset` outside of modeling code, the `item` field would be empty, and so `item.body_specific?` wouldn't happen.

This would trigger when you even just report a broken image!

Now, we always run the SQL query to check for that flag.
2021-03-16 10:40:01 -07:00
Matt Dunn-Rankin
5cae876c46 ugh, one more bug fix? 2017-10-29 15:18:53 -07:00
Matchu
4a539f32db hope I did this swfimages fix right xD 2015-08-05 20:22:23 -04:00
b0cc4c2396 swf links 2015-05-03 16:57:42 -05:00
6984201990 dev util method to manually change SWF asset body ID 2013-06-26 20:08:19 -07:00
ae9b690536 assets that fit a special color can also have body_id == 0
The Baby Floor Gym, in particular, seems to have body_id == 0, but
is only for Baby pets. This commit unbreaks that item
2013-03-07 19:31:49 -06:00
a80f70bb88 phew, fixed some issues with contribution saving in rails 3.2 2013-03-05 20:51:24 -06:00
250f425509 rails 3.1 upgrade - still buggy 2013-03-05 15:10:25 -06:00
26ac3782ec move zones to database 2013-01-24 18:26:00 -06:00
bfd825d98e attempt to fix new species_support_ids format. sigh. 2013-01-23 00:25:09 -06:00
9701221035 wardrobe now considers item.species_support_ids when deciding compatibility
For example, the Meerca Maid Tray is a foreground item, so the SWF is marked
as compatible with all body types, but the item itself is clearly marked as
Meercas-only. items#show reflected this properly, but the swf_assets#index
call that the wardrobe uses ignored item.species_support_ids.

So, /bodies/:body_id/swf_assets.json?item_ids[]=... was deprecated in favor
of /pet_types/:pet_type_id/items/swf_assets.json?item_ids=[]..., which is
much like the former route but, before loading assets, also loads the pet
type and items, then filters the items by compatibility, then only loads
assets for the compatible items.
2013-01-02 23:15:32 -05:00
775ef7fa51 finally fix encased in ice - woo! 2012-10-05 20:56:52 -05:00
7c015e2d88 carrierwave for asset swfs 2012-07-16 16:45:26 -04:00
696b2aedaf give SWFs real, unique ID numbers
Lots of scary bugs were being caused by the fact that the possibly-duplicate Neopets ID
was being treated as an SWF's real primary key, meaning that a save meant for object swf
number 123 could be saved to biology swf number 123. Which is awful.

This update gives SWFs their own unique internal ID numbers. All external lookups still use
the remote ID and the type, meaning that the client side remains totally unchanged (phew).
However, all database relationships with SWFs use the new ID numbers, making everything
cleaner. Yay.

There are probably a few places where it would be appropriate to optimize certain lookups
that still depend on remote ID and type. Whatever. Today's goal was to remove crazy
glitches that have been floating around like mad. And I think that goal has been met.
2012-01-12 17:17:59 -06:00
285c7858c0 app can now load environment even if schema not yet loaded 2011-09-06 11:15:09 -05:00
b50b9d237d allow broken image resubmits after 1hr 2011-08-07 20:43:42 -04:00
7358aae680 report broken images 2011-08-07 18:23:44 -04:00
9eaeadaa8c delete tmp dir after converting swfs 2011-06-10 14:45:33 -04:00
cf94c7ef59 utf-8 support in both ruby 1.9 and 1.8 2011-06-04 18:40:15 -04:00
6c9ddac8dd totally pro wardrobe image adapter, via konami 2011-05-22 16:30:02 -04:00
926f2a5350 improve swf image request enqueue on creation 2011-05-22 13:10:29 -04:00
b13fd7ae99 moving toward s3 image storage 2011-05-20 19:19:14 -04:00
883ecde836 convert_swf! method runs 2011-05-14 10:25:11 -04:00
648649f5cc support special colors in the infinite closet 2011-05-02 18:07:56 -04:00
2a7fea11e6 handle timeout errors for pet data, swfs, pet images 2011-02-19 22:49:13 -05:00
ea4ec0f99d gracefully handle connection errors with neopets servers 2011-02-19 22:09:12 -05:00
b45faa6753 Merge branch 'mall_spider' 2010-11-27 18:51:08 -05:00