Commit graph

49 commits

Author SHA1 Message Date
ef9958c11e Add asset URLs to archive:create:list-urls script
Here, we read URLs out from the swf_assets table, including SWFs, manfests, and everything referenced by the manifests.

There are a few data-polishing tricks we needed to do to get this to work! Most notably, newer manfests reference themselves, but older ones don't; so we try to infer the manifest URL from the other URLs. (Our database caches the manifest content, but not the manifest URL it came from.)
2022-09-12 17:26:11 -07:00
3ce895d52f Start building archive:create:list-urls script
Just working on making an images.neopets.com mirror, just in case! To start, I'm extracting all the URLs we need to back up; and then I'll make a separate script whose job is to mirror all of the URLs in the list.
2022-09-12 15:53:22 -07:00
4c9dbf91fb Use latest ~owls NC trade values API
They're moving away from the bulk endpoint to individual item data lookups, so we're updating to match!
2022-09-04 01:35:05 -07:00
25092b865a Add Delete button for outfits
Hey finally! I got in the mood and did it, after… a year? idk lol

The button should only appear for outfits that are already saved, that are owned by you. And the server enforces it!

I also added a new util function to give actually useful error messages when the GraphQL server throws an error. Might be wise to use this in more places where we're currently just using `error.message`!
2022-08-15 20:23:17 -07:00
c1eef6222b Oops, fix db pooling for scripts
Right, ok, `db.close()` needs to be `db.end()` now.

This probably didn't break the user-syncing cron job though, because that doesn't automatically update I think? so it should still be comfortably running older version of the code that should still work just fine
2022-02-03 16:14:40 -08:00
19f1ec092e Turn on Honeycomb instrumentation again
Well, instrumentation seems to be working fine again! The bug we ran into during commit e5081dab7e is gone. Cool!

I want to be able to see what's making the new box slow. My hypothesis was (and it seems to be right) that communication with the database on the Classic DTI server is slow.

But now that they're on the same Linode account and region, I think I can set up a private VLAN to make them muuuch faster. We'll try it out!
2021-11-26 23:41:22 -08:00
eb2eb241ba Use the first Auth0 connection instead of prompt
Oh right, we have another prompt we need to not prompt for, lol :p

Here, I just assume that there's one database connection on the Auth0 account. If that's not true, the script will error—not because this is a fundamentally unresolvable problem, but because I don't want to write code for configuring a situation that doesn't exist yet :p
2021-10-02 01:01:29 -07:00
357061221f Use env vars for MySQL in export-users-to-auth0
Huh, oops, there are a _few_ reasons the user sync cron job hasn't been running correctly.

I fixed some of the config in prod, but then discovered one more issue: the script prompts for an admin database password, so of _course_ it can't auto-run, lol.

Instead, I've now created a `impress2020-util` account, with just a few permissions (but specifically the `openneo_id.users` permission that I _don't_ give the app!), and added the username and password to the secret .env file, both locally and in prod. (In this case, prod means the Linode VPS, not Vercel, because that's where our cron runs.)
2021-10-02 00:57:39 -07:00
ba8e4d8aa7 Trickier disabling honeycomb instrumentation
Hm, okay, so the documented way to not instrument anything doesn't actually stop them from patching Module._load. But this undocumented option sure does! So, woo, let's try it! lol
2021-08-08 00:23:57 -07:00
e5081dab7e Disable honeycomb auto instrumentation
Huh, well, I can't figure out what in our production env stopped working with Honeycomb's automatic instrumentation… so, oh well! Let's try disabling it for now and see if it works.

This means our Honeycomb logs will no longer include _super helpful_ visualizations of how HTTP requests and MySQL queries create a request dependency waterfall… but I haven't opened Honeycomb in a while, and this bug is blocking all of prod, so if this fixes the site then I'm okay with that as a stopgap!

Btw the error message was:
```
Unhandled rejection: TypeError: Cannot read property 'id' of undefined    at exports.instrumentLoad (/var/task/node_modules/honeycomb-beeline/lib/instrumentation.js:80:14)    at Function._load (/var/task/node_modules/honeycomb-beeline/lib/instrumentation.js:164:16)    at ModuleWrap.<anonymous> (internal/modules/esm/translators.js:199:29)    at ModuleJob.run (internal/modules/esm/module_job.js:169:25)    at Loader.import (internal/modules/esm/loader.js:177:24)
```

Oh also, this is the first time eslint has looked at scripts/build-cached-data.js I guess, so I fixed some lint errors in there.
2021-08-08 00:14:55 -07:00
617ffd9a38 Add --upsert option to auth0 script 2021-06-16 08:24:59 -07:00
1d97988383 Finish outfit auto-saving!
Hope it actually work-works lol

Did some refactors in useOutfitState to support the new reset action we do after auto-saving, in case the server tweaked things like the name.
2021-05-04 12:33:13 -07:00
02228f533a Add idempotency comment to auth0 script 2021-03-31 16:47:39 -07:00
62952b80dd Can edit closet list text, incl as Support
I wanted the ability to clear out closet list text for Support users, and figured I should just build the UI for end users too, and grant Support users the same access!
2021-03-23 17:48:11 -07:00
0e8e50b054 Simpler, faster modeling query
I narrowed down the problem to the fact that we were joining in pet types against assets, and *then* running GROUP and DISTINCT and everything. Assets x compatible species/color pairs is a LOT of rows!

Here, we instead get all the relevant body IDs first, and *then* match them against pet types—which we fetch in one batch to match body to canonical species/color.

I'm also trashing the weird caching mechanism we did here, because in practice it doesn't seem reliable anyway. If anything, I'd want to look at stronger CDN caching. (I made a small improvement to the caching annotation, but ultimately it still doesn't matter, because this query uses logged-in stuff and always comes out max-age=0 anyway.)
2021-03-18 13:02:06 -07:00
cfbb23d9ff Keep track of when manifest was last cached
Can use this later to make sure we re-run them on a regular interval or something!
2021-03-11 02:23:40 -08:00
197c623426 delete-user.js script
I already had a script for this lying around, and adapted it a tiny bit to the repository!

Part of me thought about building it in as a support tool. I might've if:
- this CLI didn't already exist
- we already had tighter permissioning, this is pretty high stakes!!
2021-03-10 05:19:51 -08:00
2f36f8a0e8 Export individual user to auth0 2021-03-10 05:18:31 -08:00
2164f06021 Add --resync-existing to cache-asset-manifests
Wowie, looks like all the SVG asset manifests changed format lol! Running this now to update them all. There's a lot of them!
2021-01-21 15:14:23 -08:00
79258e11b9 Add --skip-unchanged flag to cache-asset-manifests
I'm not getting cron success _or_ cron failure emails for running this script on our Linode box. I was getting failures back when I had the command wrong, though.

My hypothesis is that the script output is too long to email, because of some limit somewhere along the way. I'll update the cron job to use `--skip-unchanged`, in hopes that it helps me get the emails! (I'm not suuure it's running, is the thing... though hey, here's a way to check: as of now, 512,624 of 521,896 assets are converted. If that changes eventually without a manual script run, then the cron is working!)
2021-01-21 10:38:47 -08:00
6da2ddb453 Perf & feedback upgrades to cache-asset-manifests
I want to start running this on a regular cron, and making the script faster (stop sending redundant queries) and clearer (# actually updated) is super useful for that!
2021-01-20 09:49:05 -08:00
24f29173bb only sync recent users to auth0 2021-01-16 11:08:12 -08:00
6168f98b8e include previous misses in cache-asset-manifests
Originally, this was sorta a cache warmup script: we wanted to fill in manifests that we hadn't checked for yet.

But now, I want to _also_ check previous cache misses, that we stored in the db as an empty string. Maybe it's been converted now!
2020-12-28 14:04:28 -08:00
fd864ab8ec fix cache-asset-manifests script
Just wanted to run it and see if much has been converted since we last checked!
2020-12-28 14:00:11 -08:00
7c9313f4a6 support tool to edit usernames 2020-11-18 07:42:40 -08:00
53d399f46b add neomail username to user GQL 2020-10-23 22:55:13 -07:00
3a20deba09 can remove owned/wanted items from item page 2020-10-23 22:43:56 -07:00
57889a3a88 can add own/wanted items from item page
the buttons work now! but only when adding 😅 remove comes next!
2020-10-22 21:20:49 -07:00
6c97c15979 add closet_lists to dev schema too 2020-10-22 20:32:02 -07:00
0dcbe6fc2d Add users and closet_hangers to local db schema 2020-10-22 19:53:32 -07:00
99e6480486 add logging to modeling
my hope is that, if we fuck things up, this will make it clear 😅
2020-10-06 07:06:19 -07:00
df2d814c13 enable running against a local dev database
had to add some missing tables, but it seems to work! (some known errors though, from assumptions we make e.g. blue acaras existing)
2020-10-06 06:18:19 -07:00
41e70ba8d0 finish modeling full pet appearance 2020-09-25 03:29:02 -07:00
50537758c5 start test/dev db IDs at 1, not wherever prod is
We download the schema from prod, and omit real data, but I didn't notice that we were still pulling the metadata of the auto increment counter for IDs! Now, we scrub that from the schema file we save.
2020-09-25 03:29:02 -07:00
5332c9e265 save biology assets on model
and start in comments on pet states :)
2020-09-25 03:29:01 -07:00
ff3fc943d7 modeling saves pet type 2020-09-25 03:29:01 -07:00
9111dfddd3 save item swf assets during modeling 2020-09-25 03:29:01 -07:00
dfeeb9fe0d modeling will save new item data (but not assets)
just a first step!
2020-09-18 07:34:41 -07:00
07691a4e6b add basic test db infra
Boom, now we can also run a clean MySQL test db on each test that wants it :)

the test I wrote as a sample is currently marked `it.skip` because it's not passing yet!
2020-09-18 05:50:17 -07:00
68b7beae1b start setting up a local dev database 2020-09-18 05:24:03 -07:00
12b87ee7d1 set up auth on the server + test utils 2020-09-02 23:00:16 -07:00
4796d213aa oh right, remove LIMIT 1 from export script! 2020-09-02 15:50:33 -07:00
56332ec8c0 redo auth0 export script to use APIs, not JSON 2020-09-02 15:26:33 -07:00
6982f00729 script to export users to auth0 2020-09-02 03:49:58 -07:00
3a6e3fac8e add isCommonlyUsedByItems to Zone
This is in preparation for hiding bio zone restrictions but showing item zone restrictions!

I also refactor the build-cached-data script substantially, to run GraphQL against the server instead of a custom query.
2020-09-01 01:16:30 -07:00
47d22ad25c Build cached zones, stop querying on server
In this change, we cache the zones table as part of the JS build process. This keeps the database as our source of truth, while aggressively caching the data at deploy time.

See the new README for some rationale!

I tested this by pulling up dev Honeycomb, and observing that we no longer run db queries to `zones` in the new traces for the wardrobe page. (It's a good thing we did it this way, because I noticed some code in the server that was still loading the zone anyway, and fixed it here!)
2020-08-19 17:50:05 -07:00
20523a9562 add dump mode to cache-asset-manifests
I was finding the script too slow running on my local machine, because the SQL RTTs were too slow - and with one connection, they were essentially a serial bottleneck, not taking much advantage of our concurrency.

Here, I instead add a `--dump` option, which outputs SQL to stdout. I then uploaded the resulting SQL to the DTI box, and ran it up there. Doing the network part fast on my machine, and the SQL part fast on the cloud machine!

I first considered uploading this script to the cloud machine, but it's an old Ubuntu and I couldn't figure out how to install a recent NodeJS onto it 🙃
2020-08-19 17:19:15 -07:00
4977f1ee54 Revert "cache zone data"
This reverts commit 0f7ab9d10e.

The Production Vercel deploys don't seem to like how I did this build trick, even though the Preview deploys seem fine with it 🤔 Reverting for now, sent a message to Vercel support.
2020-08-17 18:49:37 -07:00
e20364cc0a create cache-asset-manifests script
Just gonna bulk load all those manifests into the db, and then that should make most loads notably faster by removing the net request! 🤞

We'll still load manifests inline sometimes, but only the first time anyone pulls up the layer in impress-2020. After that, it should be cached forever!
2020-08-17 18:23:39 -07:00