Use wget for archive:create:download-urls
Hey this is an exciting development! A list of URLs, that we want to clone onto our hard drive, turns out to be something `wget` is already very good at!
Originally I used `wget`'s `--input-file` option to process the `urls-cache.txt` file, but then I learned how to parallelize it from this StackOverflow answer: https://stackoverflow.com/a/11850469/107415. (Following the guidance in the comments, I removed `-n 1`, to avoid the overhead of extra processes and allow `wget` instances to keep using shared connections over time. Idk why it was in there, maybe the author didn't know `wget` accepts multiple args?)
Anyway yeah, it's working great, except for the weird images.neopets.com downtime! 😅 Specifically I'm noticing that all the item thumbnail images came back really fast, but the customization images are taking for-EV-er. I wonder if that's just caching properties, or if there's a different backing server for it and it's responding much more slowly? Who's to say!
In any case, I'm keeping the timeout in this script pretty low (10 seconds), and just letting failures fail. We can try re-running it again sometime when the downtime is resolved or the cache is warmed up.
2022-09-12 21:47:28 -07:00
|
|
|
echo 'Starting! (Note: If many of the URLs are already downloaded, it will take some time for wget to quietly check them all and find the new ones.)'
|
2022-10-02 07:08:40 -07:00
|
|
|
xargs --arg-file=$(dirname $0)/urls-cache.txt -P 8 wget --directory-prefix=${ARCHIVE_DIR=$(dirname $0)} --force-directories --no-clobber --timeout=10 --retry-connrefused --retry-on-host-error --no-cookies --compression=auto --https-only --no-verbose
|
|
|
|
|
|
|
|
|
|
# It's expected that xargs will exit with code 123 if wget failed to load some
|
|
|
|
|
# of the URLs. So, if it exited with 123, exit this script with 0 (success).
|
|
|
|
|
# Otherwise, exit with the code that xargs exited with.
|
|
|
|
|
# (It would be nice if we could tell wget or xargs that a 404 isn't a failure?
|
|
|
|
|
# And have them succeed instead? But I couldn't find a way to do that!)
|
|
|
|
|
XARGS_EXIT_CODE=$?
|
|
|
|
|
if [ $XARGS_EXIT_CODE -eq 123 ]
|
|
|
|
|
then
|
|
|
|
|
exit 0
|
|
|
|
|
else
|
|
|
|
|
exit $XARGS_EXIT_CODE
|
|
|
|
|
fi
|