Commit graph

124 commits

Author SHA1 Message Date
26df1e921a Remove Cypress integration tests
We haven't attended to them in a while, idk if they still work. They were helpful for automatic TDD when building certain complex features like outfit saving, but now it's just noise in the repo I think. We can get them back sometime by reverting this if we really want them!
2022-09-14 17:44:19 -07:00
80038ed9c4 Add openneo_id to the dev database
Specifically, we rename the dev database to `openneo_impress` for consistency with the main app, and create a second schema file for `openneo_id`, so we can do local account creation.
2022-09-13 21:06:35 -07:00
fa50d92ad4 Use dev db in yarn dev
If you want prod db, now you use `yarn local-prod`. Just to encourage myself to stick to the dev db a bit more now that I've got it really set up properly on my machine!
2022-09-13 19:51:19 -07:00
69d96c0b72 Update dev mysql schema
I took away the column-statistics option because the mysqldump version I'm using from mariadb doesn't seem to support it. Whatever!
2022-09-13 00:00:54 -07:00
d9ca07c9b0 Use wget for archive:create:download-urls
Hey this is an exciting development! A list of URLs, that we want to clone onto our hard drive, turns out to be something `wget` is already very good at!

Originally I used `wget`'s `--input-file` option to process the `urls-cache.txt` file, but then I learned how to parallelize it from this StackOverflow answer: https://stackoverflow.com/a/11850469/107415. (Following the guidance in the comments, I removed `-n 1`, to avoid the overhead of extra processes and allow `wget` instances to keep using shared connections over time. Idk why it was in there, maybe the author didn't know `wget` accepts multiple args?)

Anyway yeah, it's working great, except for the weird images.neopets.com downtime! 😅 Specifically I'm noticing that all the item thumbnail images came back really fast, but the customization images are taking for-EV-er. I wonder if that's just caching properties, or if there's a different backing server for it and it's responding much more slowly? Who's to say!

In any case, I'm keeping the timeout in this script pretty low (10 seconds), and just letting failures fail. We can try re-running it again sometime when the downtime is resolved or the cache is warmed up.
2022-09-12 21:47:28 -07:00
3ce895d52f Start building archive:create:list-urls script
Just working on making an images.neopets.com mirror, just in case! To start, I'm extracting all the URLs we need to back up; and then I'll make a separate script whose job is to mirror all of the URLs in the list.
2022-09-12 15:53:22 -07:00
4c9dbf91fb Use latest ~owls NC trade values API
They're moving away from the bulk endpoint to individual item data lookups, so we're updating to match!
2022-09-04 01:35:05 -07:00
8239116e4f Upgrade @apollo/client to 3.6.9
Mainly because I want the `reset` function from `useMutation` for the login form work 😅 but also it's good to keep pace!
2022-08-16 23:27:30 -07:00
ca812784a6 Reinstall yargs
Uhh hmm, I don't remember when we removed it from package.json, I guess
maybe I thought it was unused and didn't look carefully enough?

Anyway, this fixes the export-users-to-auth0 script, which was crashing
because yargs wasn't installed, oops!
2022-01-07 18:08:04 -08:00
dependabot[bot]
6f3ea6311e
Bump aws-sdk from 2.726.0 to 2.814.0
Bumps [aws-sdk](https://github.com/aws/aws-sdk-js) from 2.726.0 to 2.814.0.
- [Release notes](https://github.com/aws/aws-sdk-js/releases)
- [Changelog](https://github.com/aws/aws-sdk-js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-js/compare/v2.726.0...v2.814.0)

---
updated-dependencies:
- dependency-name: aws-sdk
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-26 22:47:07 +00:00
dependabot[bot]
8413d601ad
Bump apollo-server from 2.19.2 to 2.25.3
Bumps [apollo-server](https://github.com/apollographql/apollo-server/tree/HEAD/packages/apollo-server) from 2.19.2 to 2.25.3.
- [Release notes](https://github.com/apollographql/apollo-server/releases)
- [Changelog](https://github.com/apollographql/apollo-server/blob/main/CHANGELOG.md)
- [Commits](https://github.com/apollographql/apollo-server/commits/apollo-server@2.25.3/packages/apollo-server)

---
updated-dependencies:
- dependency-name: apollo-server
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-11-26 22:45:08 +00:00
0c2939dfe4 Use Puppeteer instead of Playwright
We used Playwright in the first place to try to work around a Vercel deploy issue, and I'm not sure it really ended up mattering lol :p

But yeah, I'm putting the new Puppeteer code through the same prod stress test, and it just doesn't seem to be getting into the same broken state that Playwright was. I'm guessing it's just that Puppeteer has more investment in edge-case handling? (There's also the fact that we're no longer running things as root, which could have been a fucky problem, too?)
2021-11-13 02:16:58 -08:00
18bc3df6f4 Use browser pooling for /api/assetImage
I tried running a pressure test against assetImage on prod with the open-source tool `wrk`:

```
wrk -t12 -c20 -d20s --timeout 20s 'https://impress-2020-box.openneo.net/api/assetImage?libraryUrl=https%3A%2F%2Fimages.neopets.com%2Fcp%2Fitems%2Fdata%2F000%2F000%2F522%2F522756_2bde0443ae%2F522756.js&size=600'
```

I found that, unsurprisingly, we run a lot of concurrent requests, which fill up memory with a lot of Chromium instances!

In this change, we declare a small pool of 2 browser contexts, to allow a bit of concurrency but still very strictly limit how many browser instances can actually get created. We might tune this number depending on the actual performance characteristics!
2021-11-12 23:35:30 -08:00
9753cbe173 /api/assetImage fixes in production
Now that we're not on Vercel's AWS Lambda deployment, we can switch to something a bit more standard!

I also tweaked up our version of Playwright, because, hey, why not?

Getting the package list was a bit tricky, but we got there! Left a comment to explain where it's from.
2021-11-12 21:39:35 -08:00
9310a250d6 Fix some bugs running deploy-setup from scratch
As an exercise, I've wiped the box clean, and I'm reinstalling from the scripts! :3

I added the SSH hardening rules to the playbook instead of doing them by hand this time.

I made a mistake with creating `/srv/impress-2020`, right, you need to *say* what it should be created *as* for the creation step to work!

I also guess my recent pm2 changes made it not actually be willing to start the app anymore, because `/srv/impress-2020/current` doesn't exist or have `node_modules` yet. I'm doing a cute thing where I create a placeholder app during setup, so there's always something to run, without introducing the complexities of a real deploy to the setup process.

And right, of course, we need to install nginx before running certbot! But we need to add certbot config *after* running certbot!

And then just some misc cleanups for consistency and correctness!
2021-11-03 23:11:50 -07:00
d17263139e Fix pm2 monitoring
Okay huh, while digging a bit into another issue, I found what was wrong with our config and pm2's built-in monitoring! You can't use `yarn start`, because the wrapper script breaks its ability to look inside and see what's happening.

I also removed the compiler flag thing from the `start` script in `package.json`, because I think it's redundant? There's no compilation to be done in a live server.

I think I might remove monit after this? It's nice extra resilience in a sense, but it feels like extra complexity when it's doing the job `pm2` is supposed to do. (And tbh I've almost never heard of nginx crashing, and if it does it's probably a scenario worth investigating by hand.)
2021-11-03 16:46:35 -07:00
edd983c97a Refactor deploy to build locally, not remotely 2021-11-02 18:47:13 -07:00
dde8cee1e3 Add deploy playbook: pulls git and installs deps 2021-11-02 16:36:39 -07:00
537aeb4118 Ansible tasks to set up web user, install Node 2021-11-02 16:01:30 -07:00
a915bc4b49 Start of an Ansible playbook
Yep yep, we're getting deploy tasks set up! :3
2021-11-02 14:45:05 -07:00
36c32cdd70 Install Sharp for production
Oh neat, when trying `yarn build && yarn start` locally, I got a message about installing Sharp for better image optimization performance in production.

It mentions that this isn't relevant for Vercel, where it's auto-added. But it's good to get on it now anyway!
2021-11-02 01:00:52 -07:00
589c48beda [WIP] Fix eslint for Next.js
Tweaked some of the default Next.js rules, fixed lint-staged for `next lint`, made a few small easy lint fixes. Feels good!

Note that using the `dirs` option in `next.config.js` was causing `lint-staged` to lint _everything_. That's why I edited `yarn lint` to specify the dirs instead: that way, that command will lint all those dirs, but they won't get included in invocations with `--file`.

There are still a few lint errors left after this commit, because our <img> tags aren't working (@next/next/no-img-element). I'll fix those when we figure out what's wrong with images!
2021-11-01 22:07:46 -07:00
567e333f9e [WIP] Run cra-to-next codemod to be on Nextjs
I'm interested in ejecting from Vercel, so I'm trying to get off their proprietary-ish create-react-app + Vercel API thing, and onto Nextjs, which is very similar in shape, but more portable.

I had to disable `craCompat` in `next.config.js` to stop us from crashing on their webpack config, see https://github.com/vercel/next.js/discussions/25858#discussioncomment-1573822

The frontend seems to work at a basic level, but network requests fail, and images don't seem to be working. I'll work on those next!

Note that this commit was forced through despite failing lint checks. We'll need to fix that up too!

Also, after the codemod, I moved `src/pages` to the more canonical location `pages`. Lint tooling seemed surprised to not find a `pages` directory, and I didn't see a config that was making it work correctly in the other location, so I figured it's that Next is willing to check `pages` or `src/pages`? But this is more canonical so yeah!
2021-11-01 21:49:23 -07:00
b6b99d899f Update @apollo/client
I was hoping this would fix a cache merge ambiguity, but nope :p but hey, good to be on latest!
2021-09-30 18:37:09 -07:00
dependabot[bot]
f5b159b808
Bump immer from 8.0.1 to 9.0.6
Bumps [immer](https://github.com/immerjs/immer) from 8.0.1 to 9.0.6.
- [Release notes](https://github.com/immerjs/immer/releases)
- [Commits](https://github.com/immerjs/immer/compare/v8.0.1...v9.0.6)

---
updated-dependencies:
- dependency-name: immer
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-03 01:34:24 +00:00
adf70dc25f Use chrome-aws-lambda for assetImage
Okay cool, this one worked! We use this special Chrome package with AWS Lambda support, and then we use normal Playright in dev, and then we exclude `playwright` from the deployment (even though it got auto-detected by `require("playwright")`) to just barely sneak in under the 50MB limit for this function. Phew!

The preview deploys for this seem to be, actually working? So that's exciting!
2021-08-19 16:27:22 -07:00
a1243ad08f Merge branch 'main' of https://github.com/matchu/impress-2020 into main 2021-08-17 01:22:08 -07:00
9ec917e4d3 Use playwright-aws-lambda
So, just using normal playwright was crashing with this error: https://github.com/microsoft/playwright/issues/5862

I didn't understand why everyone was using playwright-core until I read the comments more carefully, and saw that it was because folks were using playwright-aws-lambda, because that's where Vercel functions run. (It has some special compat stuff.)

So I'm figuring that maybe the special case in Vercel's builder that fixes this for playwright-core maybe doesn't apply to normal playwright? But that people don't actually run into that issue in practice, because they're all using playwright-core for playwright-aws-lambda instead?

Idk, let's see how it goes! My hope is that this both fixes the immediate crasher about browsers.json being missing, _and_ fixes a problem we were _gonna_ have down the line about normal playwright not working in an AWS Lambda setting.
2021-08-17 01:01:09 -07:00
0cf97ef612 Try updating honeycomb-beeline?
In production we're suddenly getting errors in module wrapping in honeycomb-beeline. I wonder if it's like, an incompatibility with Vercel's version of Node?

Well, this new version seems to still be playing nice on dev, so hopefully that's all it is and this fixes it! I give it like a 35% chance lol :p
2021-08-07 23:13:52 -07:00
4e31a4bec7 /internal/assetImage can render movies
This is a new page, that I'm gonna use a headless browser to navigate to and screenshot the asset!
2021-07-02 14:36:08 -07:00
be3a162a8a Virtualize item list scrolling
This helps the render time by a lot!
2021-06-19 12:36:19 -07:00
70a31c794e Fix dependencies for prod
Hmm, prod builds weren't happy with how I specified the version here

…do prod builds not listen to the lockfile? Suspicious :p
2021-06-16 18:28:40 -07:00
75ceeba6e2 Bundle CreateJS, instead of loading async
So I finally started looking into the race condition that makes item previews sometimes fail to load, and as expected, it was that we were trying to load the movie before CreateJS had necessarily loaded. Usually the timing worked out, esp after a reload, but not under certain circumstances!

Anyway, I've been wanting for a while to just bundle them instead. That'll help us more eagerly load them when we need them, and not depend on external CDNs, and remove a bunch of loading state!

So yeah, I had to learn how the `easeljs` and `tweenjs` NPM packages did their bundling, and how to use `imports-loader` to let them just register straight onto `window`! But we got there and it's pretty nice tbh!
2021-06-16 18:00:25 -07:00
cf30b25be0 First draft of UserItemListPage
A lot is missing! No descriptions, no support for the "Not in a list" case, no scroll performance windowing, no editing!

But it's a start :3
2021-06-12 04:45:23 -07:00
8f83ac412c Add sharing meta tags to outfit pages
Meta tags are a bit tricky in apps built with `create-react-app`! While some bots like Google are able to render the full page when crawling, not all bots are. Most will just see the empty-ish index.html that would normally load up the application.

But we want outfit sharing to work! And be cool! And use our new outfit thumbnails!

In this change, we add a new server-side rendering API route to handle `/outfits/:id`.

It's very weak server-side rendering: it just loads index.html, and makes a few small tweaks inside the `<head>` tag. But it should be enough for sharing to work in clients that support the basics of Open Graph, which I think most major providers respect! (I know Twitter has their own tags, but it also respects the basics of OG, so let's see whether there's anything we end up _wanting_ to tweak or not!)
2021-05-14 19:51:48 -07:00
fddeeecbdb More eslint fixes
Oh oops, my no-unused-vars changes affected more than I realized! It caused builds to break.

I've now fixed it to pass on our existing codebase!
2021-05-13 18:11:44 -07:00
8c8e845dc5 Fix eslint for /api
Oops, I didn't notice this when bulk-fixing eslint stuff in src!
2021-05-13 01:13:21 -07:00
8da664227f Add a pre-commit lint hook
I've got it in my IDE too, but I want more safeguards lol, I'm tired of goofing up the pushes :p

I don't allow warnings here, but still have them as warnings not errors; because I don't want them to block build, but I _do_ want them to block commit.
2021-05-04 18:50:47 -07:00
a69bc8184a Upgrade Cypress
I was hoping it would fix a bug I'm having in a new test, and it didn't. But it didn't seem to break anything either so!
2021-05-04 16:07:00 -07:00
7d9e83a0c1 Switch console.log to a lint warning, not error
Didn't realize lint errors blocked compilation lol

I just want it to be annoyed enough at me to draw my attention back to it lol
2021-05-04 14:26:44 -07:00
03acbb9757 Add Cypress env to eslint 2021-05-04 12:33:13 -07:00
ec3aa1747d Lint against console.log
and replace `console.log` instances in the codebase with better alternatives!
2021-05-03 15:01:49 -07:00
ee0185a846 Upgrade Chakra to 1.6.0
Need this for an upcoming bug fix!
2021-04-30 12:48:03 -07:00
71215fe599 LRU caches to speed up outfit layer preload
The layer preloader already takes advantage of, and primes, the HTTP cache.

But we still do duplicate work, on every OutfitPreview render, to re-execute movie clip libraries, and create a movie clip to test for animations. The former is nontrivial cost, and the latter is often large cost. This can make even basic outfit changes slow, when there's no change to the movie clip layers and the player is paused!

Here, we add an LRU cache for movie clip libraries, and for the question of "is it animated?". This should speed up a number of places where we would reload the movie (including between toggling the item), and various changes that were triggering full movie clip rebuilds unnecessarily.

We _aren't_ solving here for the fact that toggling an animated item requires rebuilding the movie clip, which could conceivably be cached—but with some state management trickiness, because ideally it should be a separate clip for each context where it's being shown. Imo not yet worth the effort! (esp because I think users understand that toggling an animated item can be slow, whereas this was affecting _other_ actions way too much)
2021-04-16 20:16:56 -07:00
405e35a546 Basic outfit state Cypress tests 2021-04-16 04:27:19 -07:00
06ec0b0b52 Add disabled save UI 2021-03-29 19:46:21 -07:00
197c623426 delete-user.js script
I already had a script for this lying around, and adapted it a tiny bit to the repository!

Part of me thought about building it in as a support tool. I might've if:
- this CLI didn't already exist
- we already had tighter permissioning, this is pretty high stakes!!
2021-03-10 05:19:51 -08:00
281fdccb89 Upgrade Chakra, remove tooltip flicker workaround 2021-02-22 19:00:47 -08:00
d0e759d12a Oh, actually add TS_NODE_COMPILER to yarn start!
I wrote about this in the commit message, but forgot to do it!
2021-02-07 00:23:19 -08:00
1bb52e9f49 Use typescript-cached-transpile
I have this in the scripts, but the dev command is the big deal: it saves me ~2sec locally per GraphQL request :3

But the dev command requires an upstream change, I have a local fork working with it, but I'm gonna try to get a PR upstream!
2021-02-07 00:08:21 -08:00