Commit graph

137 commits

Author SHA1 Message Date
bf2745f9f1 Oops, remove leftover debug code 2022-12-22 18:44:47 -08:00
d37b7fab2f Add installation guide to README
I tried it in my lil Ubuntu WSL box and hey, it worked great! Neat!!

Pretty neat to have just sat down and fixed up the dev environment for other people?? I'll see about what it would take to actually invite people in…
2022-11-13 09:09:36 -08:00
b84c8ba34e Script to set up dev db with public DTI data
Now, someone with production DB access can run `yarn db:export:public-data` to create `public-data-constants.sql` and `public-data-from-modeling.sql`.

Then, someone setting up their dev database can run `yarn db:setup-dev:full` and get all the wearables data imported right into their dev database!

I'm noticing just how poorly I'm keeping up with my own goals for finishing up DTI, and wondering if now is a good time to circle back to some old offers for code contributions I got last year… I also just figure that making this app Possible To Run with a backup of the basic public database is like. a pretty handy thing to have for archival's sake imo

Note that, for this change, we also set up Git LFS (Large File Storage). Github should be automatically compatible with this! It's a way to not write the whole 30MB database dump into the repository history, and instead keep it in a secondary filestore, because Git's core algorithms aren't really built to handle large blobs of data very well. Users setting up their dev environment will therefore also need to have Git LFS installed for this script to work! (Otherwise, they'll see a "pointer" file in `public-data-from-modeling.sql.gz` that contains some metadata about the file state but not the data itself.)
2022-11-13 07:03:44 -08:00
8dee9ddbed Refactor archive scripts to prepare/create/upload
Sat down and thought about the structure here and how to make the full/delta stuff make more sense together! Here's what I came up with!

In both full and delta archiving, we prepare the manifest, we create the local archive, then we upload it to remote.
2022-10-13 16:07:12 -07:00
35713069fa Delta version of archive scripts
I like running the full `archive:create` to help us be _confident_ we've got the whole darn thing, but it takes multiple days to run on my machine and its slow HDD, which… I'm willing to do _sometimes_, but not frequently.

But if we had a version of the script that ran faster, and only on URLs we still _need_, we could run that more regularly and keep our live archive relatively up-to-date. This would enable us to build reliable fallback infra for when images.neopets.com isn't responding (like today lol)!

Anyway, I stopped early in this process because images.neopets.com is bad today, which means I can't really run updates today, lol :p but the delta-ing stuff seems to work, and takes closer to 30min to get the full state from the live archive, which is, y'know, still slow, but will make for a MUCH faster process than multiple days, lol
2022-10-13 15:08:29 -07:00
e8d7f6678d Auto-modeling script??
It seems to be working!! How exciting!! I'm just letting it run on stuff now :3

One important issue is that Classic DTI doesn't show images for items modeled this way, because we don't download the SWFs for it. But I wanna update it to stop using AWS anyway and do the same stuff 2020 does, I think we can do that pretty sneakily!
2022-10-11 11:13:10 -07:00
b9b0db8b3a Some archive:create tweaks
I'm looking into what it would take to update the archive on a regular basis. The commands right now *are* pretty good at avoiding duplicate work… but the S3 upload still seems like it's taking very long even to just validate what's in the archive already. We might have to build our own little cache rather than using `aws s3 sync`, if we want faster incremental updates?

Here, I make a few quality-of-life changes to add a `archive:create` command that runs everything in a straight line. That way, I can let it run and see how much wall-time it takes, to be able to decide whether speeding it up feels necessary. (vs whether it's a few-hours task I can just set a reminder to manually run every week or something)
2022-10-02 07:08:40 -07:00
ae568072f8 Use AWS CLI for archive upload instead of s3cmd
Okay there we go, I was following Linode's guidance and ended up using a non-Amazon S3 client, but it turns out you can get the official Amazon AWS client to play with private services too, and it doesn't do the thing s3cmd does of trying to list every single file before doing anything 😅

This command _is_ doing weird stall-outs here and there, but is mostly just chugging along. It's not exactly fast, I imagine it'll take some time, but the fact that it's like. working. is huge lmao
2022-09-24 13:18:40 -07:00
6d86e3e2a9 /api/readFromArchive to serve a backed up image
Okay so the funny thing is that my upload script is clearly like *super* not working lol, it's been running more than an hour now and still hasn't finished listening the files. So there's only actually a handful of files to test with here, from the `archive:create:upload-test` script!

But anyway, uhh once the archive is actually uploaded, this is a way to read it back! Mainly as a way to assure me that it's all saved correctly, but also as a potential backup for images.neopets.com if it goes down again sometime.
2022-09-24 12:44:13 -07:00
38cb22980e Add archive:create:upload script
This will upload to our remote storage! We're using `s3cmd`, but our storage isn't actually Amazon S3, it's Linode Object Storage, which has an S3-compatible API. (And it's where our VPSes already are, and its pricing model is very generous for our relatively small scale of data.)

I haven't _really_ tested this exactly yet, because while `archive:create:upload-test` works great and uploads the 3 targeted files successfully… running the big one takes a very long time to even _enumerate_ all the files on my machine. (This makes sense, because I'm keeping the ~100GB archive on my HDD, which is not a fast disk!)

So I'm pushing ahead even though the script is untested, because I wanna work on other stuff too!
2022-09-24 11:01:48 -07:00
d2db7e94a3 Refactor package.json scripts a tiny bit
We extract a `run-script` command that contains the big `ts-node` incantation!
2022-09-24 10:33:16 -07:00
681eb5cdc5 Add dotenv to archive:create:download-urls
That way, I can specify `ARCHIVE_DIR` in my .env file, instead of having to remember to specify it every time!
2022-09-22 22:08:16 -07:00
1163d41d32 Uninstall react-router-dom
Begone!!
2022-09-15 00:44:33 -07:00
26df1e921a Remove Cypress integration tests
We haven't attended to them in a while, idk if they still work. They were helpful for automatic TDD when building certain complex features like outfit saving, but now it's just noise in the repo I think. We can get them back sometime by reverting this if we really want them!
2022-09-14 17:44:19 -07:00
80038ed9c4 Add openneo_id to the dev database
Specifically, we rename the dev database to `openneo_impress` for consistency with the main app, and create a second schema file for `openneo_id`, so we can do local account creation.
2022-09-13 21:06:35 -07:00
fa50d92ad4 Use dev db in yarn dev
If you want prod db, now you use `yarn local-prod`. Just to encourage myself to stick to the dev db a bit more now that I've got it really set up properly on my machine!
2022-09-13 19:51:19 -07:00
69d96c0b72 Update dev mysql schema
I took away the column-statistics option because the mysqldump version I'm using from mariadb doesn't seem to support it. Whatever!
2022-09-13 00:00:54 -07:00
d9ca07c9b0 Use wget for archive:create:download-urls
Hey this is an exciting development! A list of URLs, that we want to clone onto our hard drive, turns out to be something `wget` is already very good at!

Originally I used `wget`'s `--input-file` option to process the `urls-cache.txt` file, but then I learned how to parallelize it from this StackOverflow answer: https://stackoverflow.com/a/11850469/107415. (Following the guidance in the comments, I removed `-n 1`, to avoid the overhead of extra processes and allow `wget` instances to keep using shared connections over time. Idk why it was in there, maybe the author didn't know `wget` accepts multiple args?)

Anyway yeah, it's working great, except for the weird images.neopets.com downtime! 😅 Specifically I'm noticing that all the item thumbnail images came back really fast, but the customization images are taking for-EV-er. I wonder if that's just caching properties, or if there's a different backing server for it and it's responding much more slowly? Who's to say!

In any case, I'm keeping the timeout in this script pretty low (10 seconds), and just letting failures fail. We can try re-running it again sometime when the downtime is resolved or the cache is warmed up.
2022-09-12 21:47:28 -07:00
3ce895d52f Start building archive:create:list-urls script
Just working on making an images.neopets.com mirror, just in case! To start, I'm extracting all the URLs we need to back up; and then I'll make a separate script whose job is to mirror all of the URLs in the list.
2022-09-12 15:53:22 -07:00
4c9dbf91fb Use latest ~owls NC trade values API
They're moving away from the bulk endpoint to individual item data lookups, so we're updating to match!
2022-09-04 01:35:05 -07:00
8239116e4f Upgrade @apollo/client to 3.6.9
Mainly because I want the `reset` function from `useMutation` for the login form work 😅 but also it's good to keep pace!
2022-08-16 23:27:30 -07:00
ca812784a6 Reinstall yargs
Uhh hmm, I don't remember when we removed it from package.json, I guess
maybe I thought it was unused and didn't look carefully enough?

Anyway, this fixes the export-users-to-auth0 script, which was crashing
because yargs wasn't installed, oops!
2022-01-07 18:08:04 -08:00
dependabot[bot]
6f3ea6311e
Bump aws-sdk from 2.726.0 to 2.814.0
Bumps [aws-sdk](https://github.com/aws/aws-sdk-js) from 2.726.0 to 2.814.0.
- [Release notes](https://github.com/aws/aws-sdk-js/releases)
- [Changelog](https://github.com/aws/aws-sdk-js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js/compare/v2.726.0...v2.814.0)

---
updated-dependencies:
- dependency-name: aws-sdk
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-26 22:47:07 +00:00
dependabot[bot]
8413d601ad
Bump apollo-server from 2.19.2 to 2.25.3
Bumps [apollo-server](https://github.com/apollographql/apollo-server/tree/HEAD/packages/apollo-server) from 2.19.2 to 2.25.3.
- [Release notes](https://github.com/apollographql/apollo-server/releases)
- [Changelog](https://github.com/apollographql/apollo-server/blob/main/CHANGELOG.md)
- [Commits](https://github.com/apollographql/apollo-server/commits/apollo-server@2.25.3/packages/apollo-server)

---
updated-dependencies:
- dependency-name: apollo-server
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-26 22:45:08 +00:00
0c2939dfe4 Use Puppeteer instead of Playwright
We used Playwright in the first place to try to work around a Vercel deploy issue, and I'm not sure it really ended up mattering lol :p

But yeah, I'm putting the new Puppeteer code through the same prod stress test, and it just doesn't seem to be getting into the same broken state that Playwright was. I'm guessing it's just that Puppeteer has more investment in edge-case handling? (There's also the fact that we're no longer running things as root, which could have been a fucky problem, too?)
2021-11-13 02:16:58 -08:00
18bc3df6f4 Use browser pooling for /api/assetImage
I tried running a pressure test against assetImage on prod with the open-source tool `wrk`:

```
wrk -t12 -c20 -d20s --timeout 20s 'https://impress-2020-box.openneo.net/api/assetImage?libraryUrl=https%3A%2F%2Fimages.neopets.com%2Fcp%2Fitems%2Fdata%2F000%2F000%2F522%2F522756_2bde0443ae%2F522756.js&size=600'
```

I found that, unsurprisingly, we run a lot of concurrent requests, which fill up memory with a lot of Chromium instances!

In this change, we declare a small pool of 2 browser contexts, to allow a bit of concurrency but still very strictly limit how many browser instances can actually get created. We might tune this number depending on the actual performance characteristics!
2021-11-12 23:35:30 -08:00
9753cbe173 /api/assetImage fixes in production
Now that we're not on Vercel's AWS Lambda deployment, we can switch to something a bit more standard!

I also tweaked up our version of Playwright, because, hey, why not?

Getting the package list was a bit tricky, but we got there! Left a comment to explain where it's from.
2021-11-12 21:39:35 -08:00
9310a250d6 Fix some bugs running deploy-setup from scratch
As an exercise, I've wiped the box clean, and I'm reinstalling from the scripts! :3

I added the SSH hardening rules to the playbook instead of doing them by hand this time.

I made a mistake with creating `/srv/impress-2020`, right, you need to *say* what it should be created *as* for the creation step to work!

I also guess my recent pm2 changes made it not actually be willing to start the app anymore, because `/srv/impress-2020/current` doesn't exist or have `node_modules` yet. I'm doing a cute thing where I create a placeholder app during setup, so there's always something to run, without introducing the complexities of a real deploy to the setup process.

And right, of course, we need to install nginx before running certbot! But we need to add certbot config *after* running certbot!

And then just some misc cleanups for consistency and correctness!
2021-11-03 23:11:50 -07:00
d17263139e Fix pm2 monitoring
Okay huh, while digging a bit into another issue, I found what was wrong with our config and pm2's built-in monitoring! You can't use `yarn start`, because the wrapper script breaks its ability to look inside and see what's happening.

I also removed the compiler flag thing from the `start` script in `package.json`, because I think it's redundant? There's no compilation to be done in a live server.

I think I might remove monit after this? It's nice extra resilience in a sense, but it feels like extra complexity when it's doing the job `pm2` is supposed to do. (And tbh I've almost never heard of nginx crashing, and if it does it's probably a scenario worth investigating by hand.)
2021-11-03 16:46:35 -07:00
edd983c97a Refactor deploy to build locally, not remotely 2021-11-02 18:47:13 -07:00
dde8cee1e3 Add deploy playbook: pulls git and installs deps 2021-11-02 16:36:39 -07:00
537aeb4118 Ansible tasks to set up web user, install Node 2021-11-02 16:01:30 -07:00
a915bc4b49 Start of an Ansible playbook
Yep yep, we're getting deploy tasks set up! :3
2021-11-02 14:45:05 -07:00
36c32cdd70 Install Sharp for production
Oh neat, when trying `yarn build && yarn start` locally, I got a message about installing Sharp for better image optimization performance in production.

It mentions that this isn't relevant for Vercel, where it's auto-added. But it's good to get on it now anyway!
2021-11-02 01:00:52 -07:00
589c48beda [WIP] Fix eslint for Next.js
Tweaked some of the default Next.js rules, fixed lint-staged for `next lint`, made a few small easy lint fixes. Feels good!

Note that using the `dirs` option in `next.config.js` was causing `lint-staged` to lint _everything_. That's why I edited `yarn lint` to specify the dirs instead: that way, that command will lint all those dirs, but they won't get included in invocations with `--file`.

There are still a few lint errors left after this commit, because our <img> tags aren't working (@next/next/no-img-element). I'll fix those when we figure out what's wrong with images!
2021-11-01 22:07:46 -07:00
567e333f9e [WIP] Run cra-to-next codemod to be on Nextjs
I'm interested in ejecting from Vercel, so I'm trying to get off their proprietary-ish create-react-app + Vercel API thing, and onto Nextjs, which is very similar in shape, but more portable.

I had to disable `craCompat` in `next.config.js` to stop us from crashing on their webpack config, see https://github.com/vercel/next.js/discussions/25858#discussioncomment-1573822

The frontend seems to work at a basic level, but network requests fail, and images don't seem to be working. I'll work on those next!

Note that this commit was forced through despite failing lint checks. We'll need to fix that up too!

Also, after the codemod, I moved `src/pages` to the more canonical location `pages`. Lint tooling seemed surprised to not find a `pages` directory, and I didn't see a config that was making it work correctly in the other location, so I figured it's that Next is willing to check `pages` or `src/pages`? But this is more canonical so yeah!
2021-11-01 21:49:23 -07:00
b6b99d899f Update @apollo/client
I was hoping this would fix a cache merge ambiguity, but nope :p but hey, good to be on latest!
2021-09-30 18:37:09 -07:00
dependabot[bot]
f5b159b808
Bump immer from 8.0.1 to 9.0.6
Bumps [immer](https://github.com/immerjs/immer) from 8.0.1 to 9.0.6.
- [Release notes](https://github.com/immerjs/immer/releases)
- [Commits](https://github.com/immerjs/immer/compare/v8.0.1...v9.0.6)

---
updated-dependencies:
- dependency-name: immer
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-03 01:34:24 +00:00
adf70dc25f Use chrome-aws-lambda for assetImage
Okay cool, this one worked! We use this special Chrome package with AWS Lambda support, and then we use normal Playright in dev, and then we exclude `playwright` from the deployment (even though it got auto-detected by `require("playwright")`) to just barely sneak in under the 50MB limit for this function. Phew!

The preview deploys for this seem to be, actually working? So that's exciting!
2021-08-19 16:27:22 -07:00
a1243ad08f Merge branch 'main' of https://github.com/matchu/impress-2020 into main 2021-08-17 01:22:08 -07:00
9ec917e4d3 Use playwright-aws-lambda
So, just using normal playwright was crashing with this error: https://github.com/microsoft/playwright/issues/5862

I didn't understand why everyone was using playwright-core until I read the comments more carefully, and saw that it was because folks were using playwright-aws-lambda, because that's where Vercel functions run. (It has some special compat stuff.)

So I'm figuring that maybe the special case in Vercel's builder that fixes this for playwright-core maybe doesn't apply to normal playwright? But that people don't actually run into that issue in practice, because they're all using playwright-core for playwright-aws-lambda instead?

Idk, let's see how it goes! My hope is that this both fixes the immediate crasher about browsers.json being missing, _and_ fixes a problem we were _gonna_ have down the line about normal playwright not working in an AWS Lambda setting.
2021-08-17 01:01:09 -07:00
0cf97ef612 Try updating honeycomb-beeline?
In production we're suddenly getting errors in module wrapping in honeycomb-beeline. I wonder if it's like, an incompatibility with Vercel's version of Node?

Well, this new version seems to still be playing nice on dev, so hopefully that's all it is and this fixes it! I give it like a 35% chance lol :p
2021-08-07 23:13:52 -07:00
4e31a4bec7 /internal/assetImage can render movies
This is a new page, that I'm gonna use a headless browser to navigate to and screenshot the asset!
2021-07-02 14:36:08 -07:00
be3a162a8a Virtualize item list scrolling
This helps the render time by a lot!
2021-06-19 12:36:19 -07:00
70a31c794e Fix dependencies for prod
Hmm, prod builds weren't happy with how I specified the version here

…do prod builds not listen to the lockfile? Suspicious :p
2021-06-16 18:28:40 -07:00
75ceeba6e2 Bundle CreateJS, instead of loading async
So I finally started looking into the race condition that makes item previews sometimes fail to load, and as expected, it was that we were trying to load the movie before CreateJS had necessarily loaded. Usually the timing worked out, esp after a reload, but not under certain circumstances!

Anyway, I've been wanting for a while to just bundle them instead. That'll help us more eagerly load them when we need them, and not depend on external CDNs, and remove a bunch of loading state!

So yeah, I had to learn how the `easeljs` and `tweenjs` NPM packages did their bundling, and how to use `imports-loader` to let them just register straight onto `window`! But we got there and it's pretty nice tbh!
2021-06-16 18:00:25 -07:00
cf30b25be0 First draft of UserItemListPage
A lot is missing! No descriptions, no support for the "Not in a list" case, no scroll performance windowing, no editing!

But it's a start :3
2021-06-12 04:45:23 -07:00
8f83ac412c Add sharing meta tags to outfit pages
Meta tags are a bit tricky in apps built with `create-react-app`! While some bots like Google are able to render the full page when crawling, not all bots are. Most will just see the empty-ish index.html that would normally load up the application.

But we want outfit sharing to work! And be cool! And use our new outfit thumbnails!

In this change, we add a new server-side rendering API route to handle `/outfits/:id`.

It's very weak server-side rendering: it just loads index.html, and makes a few small tweaks inside the `<head>` tag. But it should be enough for sharing to work in clients that support the basics of Open Graph, which I think most major providers respect! (I know Twitter has their own tags, but it also respects the basics of OG, so let's see whether there's anything we end up _wanting_ to tweak or not!)
2021-05-14 19:51:48 -07:00
fddeeecbdb More eslint fixes
Oh oops, my no-unused-vars changes affected more than I realized! It caused builds to break.

I've now fixed it to pass on our existing codebase!
2021-05-13 18:11:44 -07:00
8c8e845dc5 Fix eslint for /api
Oops, I didn't notice this when bulk-fixing eslint stuff in src!
2021-05-13 01:13:21 -07:00