OpenNeo/impress-2020
OpenNeo/impress-2020
1
0
Fork 0
Code Pull requests Activity

Add archive:create:upload script

Browse source 
This will upload to our remote storage! We're using `s3cmd`, but our storage isn't actually Amazon S3, it's Linode Object Storage, which has an S3-compatible API. (And it's where our VPSes already are, and its pricing model is very generous for our relatively small scale of data.)

I haven't _really_ tested this exactly yet, because while `archive:create:upload-test` works great and uploads the 3 targeted files successfully… running the big one takes a very long time to even _enumerate_ all the files on my machine. (This makes sense, because I'm keeping the ~100GB archive on my HDD, which is not a fast disk!)

So I'm pushing ahead even though the script is untested, because I wanna work on other stuff too!
This commit is contained in:
Emi Matchu 2022-09-24 11:01:48 -07:00
parent d2db7e94a3
commit 38cb22980e
1 changed files with 4 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
package.json
View file

@ -84,7 +84,10 @@
"export-users-to-auth0": "yarn run-script scripts/export-users-to-auth0.js",
"validate-owls-data": "yarn run-script scripts/validate-owls-data.js",
"archive:create:list-urls": "yarn run-script scripts/archive/create/list-urls.js",
"archive:create:download-urls": "dotenv -- ./scripts/archive/create/download-urls.sh"
"archive:create:download-urls": "dotenv -- ./scripts/archive/create/download-urls.sh",
"s3cmd": "s3cmd --host=$(dotenv -p ARCHIVE_STORAGE_HOST) --host-bucket='%(bucket)s.'$(dotenv -p ARCHIVE_STORAGE_HOST) --access_key=$(dotenv -p ARCHIVE_STORAGE_READWRITE_ACCESS_KEY) --secret_key=$(dotenv -p ARCHIVE_STORAGE_READWRITE_SECRET_KEY)",
"archive:create:upload": "yarn s3cmd sync $(dotenv -p ARCHIVE_DIR)/ s3://$(dotenv -p ARCHIVE_STORAGE_BUCKET) --verbose",
"archive:create:upload-test": "yarn s3cmd sync $(dotenv -p ARCHIVE_DIR)/images.neopets.com/cp/items/data/000/000/000/1_8422bedbf9/ s3://$(dotenv -p ARCHIVE_STORAGE_BUCKET)/images.neopets.com/cp/items/data/000/000/000/1_8422bedbf9/ --verbose"
},
"browserslist": {
"production": [