OpenNeo/impress-2020
OpenNeo/impress-2020
1
0
Fork 0
Code Pull requests Activity

Increase timeout in archive SQL queries

Browse source 
One problem I run into with the archive task is that sometimes the queries time out? My hunch is that maybe some of the assets have like, weirdly big manifest files that are being transferred as surprisingly big text files?

Anyway, I'm increasing the timeout to 20s, which is big, but big feels good for a script that doesn't run often and where failing is not great news!

I'm also idly considering whether I wanna finally put in the work to do a bulk S3 uploader sometime, because the current version that iterates over multiple `aws s3 cp` calls is just real slow, I think because it establishes a new connection each time, and that operation is maybe surprisingly expensive? And the CLI doesn't have a way to do multiple uploads any more precisely than "sync up this whole folder", which is slow when the folder contains a lot of stuff you _know_ you don't want to head up there.
This commit is contained in:
Emi Matchu 2023-06-12 10:56:40 -07:00
parent f54ea61854
commit 571949b066
1 changed files with 2 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
scripts/archive/prepare/full.js
View file

@ -88,7 +88,7 @@ async function loadItems(offset, limit, db) {
const [
rows,
] = await db.query(
`SELECT id, thumbnail_url FROM items ORDER BY id LIMIT ? OFFSET ?;`,
{sql: `SELECT id, thumbnail_url FROM items ORDER BY id LIMIT ? OFFSET ?;`, timeout: 20000},
[limit, offset]
);
return rows.map(normalizeRow);
@ -103,7 +103,7 @@ async function loadSwfAssets(offset, limit, db) {
const [
rows,
] = await db.query(
`SELECT id, url, manifest FROM swf_assets ORDER BY id LIMIT ? OFFSET ?;`,
{sql: `SELECT id, url, manifest FROM swf_assets ORDER BY id LIMIT ? OFFSET ?;`, timeout: 20000},
[limit, offset]
);
return rows.map(normalizeRow);