impress/app/services/neopets_media_archive.rb

121 lines
4.3 KiB
Ruby
Raw Normal View History

Create NeopetsMediaArchive, read the actual manifests for Alt Styles The Neopets Media Archive is a service that mirrors `images.neopets.com` over time! Right now we're starting by just loading manifests, and using them to replace the hacks we used for determining the Alt Style PNG and SVG URLs; but with time, I want to load *all* customization media files, to have our own secondary file source that isn't dependent on Neopets to always be up. Impress 2020 already caches manifest files, but this strategy is different in two ways: 1. We're using the filesystem rather than a database column. (That is, manifest data is kinda duplicated in the system right now!) This is because I intend to go in a more file-y way long-term anyway, to load more than just the manifests. 2. Impress 2020 guesses at the manifest URLs by pattern, and reloads them on a regular basis. Instead, we use the modeling system: when TNT changes the URL of a manifest by appending a new `?v=` query string to it, this system will consider it a new URL, and will load the new copy accordingly. Fun fact, I actually have been prototyping some of this stuff in a side project I'd named `impress-media-server`! It's a little Sinatra app that indeed *does* save all the files needed for customization, and can generate lightweight lil preview iframes and images pretty easily. I had initially been planning this as a separate service, but after thinking over the arch a bit, I think it'll go smoother to just give the main app all the same access and awareness—and I wrote it all in Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over bit-by-bit! Anyway, only Alt Styles use this for now, but my motivation is to be able to use more-correct asset URL logic to be able to finally swap over wardrobe-2020's item search to impress.openneo.net's item search API endpoint—which will get "Items You Own" searches working again, and whittle down one of the last big things Impress 2020 can do that the main app can't. Let's see how it goes!
2024-02-23 12:02:39 -08:00
require "addressable/uri"
require "async/http/internet/instance"
Create NeopetsMediaArchive, read the actual manifests for Alt Styles The Neopets Media Archive is a service that mirrors `images.neopets.com` over time! Right now we're starting by just loading manifests, and using them to replace the hacks we used for determining the Alt Style PNG and SVG URLs; but with time, I want to load *all* customization media files, to have our own secondary file source that isn't dependent on Neopets to always be up. Impress 2020 already caches manifest files, but this strategy is different in two ways: 1. We're using the filesystem rather than a database column. (That is, manifest data is kinda duplicated in the system right now!) This is because I intend to go in a more file-y way long-term anyway, to load more than just the manifests. 2. Impress 2020 guesses at the manifest URLs by pattern, and reloads them on a regular basis. Instead, we use the modeling system: when TNT changes the URL of a manifest by appending a new `?v=` query string to it, this system will consider it a new URL, and will load the new copy accordingly. Fun fact, I actually have been prototyping some of this stuff in a side project I'd named `impress-media-server`! It's a little Sinatra app that indeed *does* save all the files needed for customization, and can generate lightweight lil preview iframes and images pretty easily. I had initially been planning this as a separate service, but after thinking over the arch a bit, I think it'll go smoother to just give the main app all the same access and awareness—and I wrote it all in Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over bit-by-bit! Anyway, only Alt Styles use this for now, but my motivation is to be able to use more-correct asset URL logic to be able to finally swap over wardrobe-2020's item search to impress.openneo.net's item search API endpoint—which will get "Items You Own" searches working again, and whittle down one of the last big things Impress 2020 can do that the main app can't. Let's see how it goes!
2024-02-23 12:02:39 -08:00
require "json"
# The Neopets Media Archive is a service that mirrors images.neopets.com files
# locally. You can request a file from it, and we'll serve it from disk if we
# have it, or request and save it if not.
#
# This is a bit different than a cache, because the intention is not just
# optimization but that we *want* to be saving images.neopets.com as a
# long-term archive, not dependent on their services having 100% uptime in
# order for us to operate. We never discard old files, we just keep going!
module NeopetsMediaArchive
# Share a pool of persistent connections, rather than reconnecting on
# each request. (This library does that automatically!)
INTERNET = Async::HTTP::Internet.instance
Create NeopetsMediaArchive, read the actual manifests for Alt Styles The Neopets Media Archive is a service that mirrors `images.neopets.com` over time! Right now we're starting by just loading manifests, and using them to replace the hacks we used for determining the Alt Style PNG and SVG URLs; but with time, I want to load *all* customization media files, to have our own secondary file source that isn't dependent on Neopets to always be up. Impress 2020 already caches manifest files, but this strategy is different in two ways: 1. We're using the filesystem rather than a database column. (That is, manifest data is kinda duplicated in the system right now!) This is because I intend to go in a more file-y way long-term anyway, to load more than just the manifests. 2. Impress 2020 guesses at the manifest URLs by pattern, and reloads them on a regular basis. Instead, we use the modeling system: when TNT changes the URL of a manifest by appending a new `?v=` query string to it, this system will consider it a new URL, and will load the new copy accordingly. Fun fact, I actually have been prototyping some of this stuff in a side project I'd named `impress-media-server`! It's a little Sinatra app that indeed *does* save all the files needed for customization, and can generate lightweight lil preview iframes and images pretty easily. I had initially been planning this as a separate service, but after thinking over the arch a bit, I think it'll go smoother to just give the main app all the same access and awareness—and I wrote it all in Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over bit-by-bit! Anyway, only Alt Styles use this for now, but my motivation is to be able to use more-correct asset URL logic to be able to finally swap over wardrobe-2020's item search to impress.openneo.net's item search API endpoint—which will get "Items You Own" searches working again, and whittle down one of the last big things Impress 2020 can do that the main app can't. Let's see how it goes!
2024-02-23 12:02:39 -08:00
ROOT_PATH = Pathname.new(Rails.configuration.neopets_media_archive_root)
Create NeopetsMediaArchive, read the actual manifests for Alt Styles The Neopets Media Archive is a service that mirrors `images.neopets.com` over time! Right now we're starting by just loading manifests, and using them to replace the hacks we used for determining the Alt Style PNG and SVG URLs; but with time, I want to load *all* customization media files, to have our own secondary file source that isn't dependent on Neopets to always be up. Impress 2020 already caches manifest files, but this strategy is different in two ways: 1. We're using the filesystem rather than a database column. (That is, manifest data is kinda duplicated in the system right now!) This is because I intend to go in a more file-y way long-term anyway, to load more than just the manifests. 2. Impress 2020 guesses at the manifest URLs by pattern, and reloads them on a regular basis. Instead, we use the modeling system: when TNT changes the URL of a manifest by appending a new `?v=` query string to it, this system will consider it a new URL, and will load the new copy accordingly. Fun fact, I actually have been prototyping some of this stuff in a side project I'd named `impress-media-server`! It's a little Sinatra app that indeed *does* save all the files needed for customization, and can generate lightweight lil preview iframes and images pretty easily. I had initially been planning this as a separate service, but after thinking over the arch a bit, I think it'll go smoother to just give the main app all the same access and awareness—and I wrote it all in Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over bit-by-bit! Anyway, only Alt Styles use this for now, but my motivation is to be able to use more-correct asset URL logic to be able to finally swap over wardrobe-2020's item search to impress.openneo.net's item search API endpoint—which will get "Items You Own" searches working again, and whittle down one of the last big things Impress 2020 can do that the main app can't. Let's see how it goes!
2024-02-23 12:02:39 -08:00
# Load the file from the given `images.neopets.com` URI, as JSON.
def self.load_json(uri)
JSON.parse(load_file(uri))
end
# Load the file from the given `images.neopets.com` URI.
def self.load_file(uri, return_content: true)
local_path = local_file_path(uri)
# Read the file locally if we have it.
if return_content
begin
content = File.read(local_path)
debug "Loaded source file from filesystem: #{local_path}"
return content
rescue Errno::ENOENT
# If it doesn't exist, that's fine: just move on and download it.
end
else
# When we don't need the content, "loading" the file is just ensuring
# it exists. If it doesn't, we'll move on and load it from source.
# (We use this when preloading files, to save the cost of reading files
# we're not ready to use yet.)
if File.exist?(local_path)
debug "Source file is already loaded, skipping: #{local_path}"
return
end
end
# Download the file from the origin, then save a copy for next time.
content = load_file_from_origin(uri)
Create NeopetsMediaArchive, read the actual manifests for Alt Styles The Neopets Media Archive is a service that mirrors `images.neopets.com` over time! Right now we're starting by just loading manifests, and using them to replace the hacks we used for determining the Alt Style PNG and SVG URLs; but with time, I want to load *all* customization media files, to have our own secondary file source that isn't dependent on Neopets to always be up. Impress 2020 already caches manifest files, but this strategy is different in two ways: 1. We're using the filesystem rather than a database column. (That is, manifest data is kinda duplicated in the system right now!) This is because I intend to go in a more file-y way long-term anyway, to load more than just the manifests. 2. Impress 2020 guesses at the manifest URLs by pattern, and reloads them on a regular basis. Instead, we use the modeling system: when TNT changes the URL of a manifest by appending a new `?v=` query string to it, this system will consider it a new URL, and will load the new copy accordingly. Fun fact, I actually have been prototyping some of this stuff in a side project I'd named `impress-media-server`! It's a little Sinatra app that indeed *does* save all the files needed for customization, and can generate lightweight lil preview iframes and images pretty easily. I had initially been planning this as a separate service, but after thinking over the arch a bit, I think it'll go smoother to just give the main app all the same access and awareness—and I wrote it all in Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over bit-by-bit! Anyway, only Alt Styles use this for now, but my motivation is to be able to use more-correct asset URL logic to be able to finally swap over wardrobe-2020's item search to impress.openneo.net's item search API endpoint—which will get "Items You Own" searches working again, and whittle down one of the last big things Impress 2020 can do that the main app can't. Let's see how it goes!
2024-02-23 12:02:39 -08:00
info "Loaded source file from origin: #{uri}"
local_path.dirname.mkpath
File.write(local_path, content)
info "Wrote source file to filesystem: #{local_path}"
return_content ? content : nil
end
# Load the file from the given `images.neopets.com` URI, but don't return its
# content. This can be faster in cases where the file's content isn't
# relevant to us, and we just want to ensure it exists.
def self.preload_file(uri)
load_file(uri, return_content: false)
end
# Load the file from the given `images.neopets.com` URI, directly from the
# source, without checking the local filesystem.
def self.load_file_from_origin(uri)
unless Addressable::URI.parse(uri).origin == "https://images.neopets.com"
raise ArgumentError, "NeopetsMediaArchive can only load from " +
"https://images.neopets.com, but got #{uri}"
end
# By running this request in a `Sync` block, we make this method look
# synchronous to the caller—but if run in the context of an async task, it
# will pause execution and move onto other work until the request is done.
# We use this in the `swf_assets:manifests:load` task to perform many
# requests in parallel!
Sync do
response = INTERNET.get(uri)
if response.status == 404
raise NotFound, "origin server returned 404: #{uri}"
elsif response.status != 200
raise "expected status 200 but got #{response.status} (#{uri})"
end
response.body.read
Create NeopetsMediaArchive, read the actual manifests for Alt Styles The Neopets Media Archive is a service that mirrors `images.neopets.com` over time! Right now we're starting by just loading manifests, and using them to replace the hacks we used for determining the Alt Style PNG and SVG URLs; but with time, I want to load *all* customization media files, to have our own secondary file source that isn't dependent on Neopets to always be up. Impress 2020 already caches manifest files, but this strategy is different in two ways: 1. We're using the filesystem rather than a database column. (That is, manifest data is kinda duplicated in the system right now!) This is because I intend to go in a more file-y way long-term anyway, to load more than just the manifests. 2. Impress 2020 guesses at the manifest URLs by pattern, and reloads them on a regular basis. Instead, we use the modeling system: when TNT changes the URL of a manifest by appending a new `?v=` query string to it, this system will consider it a new URL, and will load the new copy accordingly. Fun fact, I actually have been prototyping some of this stuff in a side project I'd named `impress-media-server`! It's a little Sinatra app that indeed *does* save all the files needed for customization, and can generate lightweight lil preview iframes and images pretty easily. I had initially been planning this as a separate service, but after thinking over the arch a bit, I think it'll go smoother to just give the main app all the same access and awareness—and I wrote it all in Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over bit-by-bit! Anyway, only Alt Styles use this for now, but my motivation is to be able to use more-correct asset URL logic to be able to finally swap over wardrobe-2020's item search to impress.openneo.net's item search API endpoint—which will get "Items You Own" searches working again, and whittle down one of the last big things Impress 2020 can do that the main app can't. Let's see how it goes!
2024-02-23 12:02:39 -08:00
end
end
def self.path_within_archive(uri)
uri = Addressable::URI.parse(uri)
path = uri.host + uri.path
# We include the query string as part of the file path, which is a bit odd!
# But Neopets often uses this for cache-busting, so we do need a mechanism
# for knowing whether we're holding the right version of the file. We could
# also consider storing the file by just its normal path, but with some
# metadata to track versioning information (e.g. a sqlite db, or a metadata
# file in the same directory).
path += "?" + uri.query if !uri.query.nil? && !uri.query.empty?
path
end
def self.local_file_path(uri)
ROOT_PATH + path_within_archive(uri)
Create NeopetsMediaArchive, read the actual manifests for Alt Styles The Neopets Media Archive is a service that mirrors `images.neopets.com` over time! Right now we're starting by just loading manifests, and using them to replace the hacks we used for determining the Alt Style PNG and SVG URLs; but with time, I want to load *all* customization media files, to have our own secondary file source that isn't dependent on Neopets to always be up. Impress 2020 already caches manifest files, but this strategy is different in two ways: 1. We're using the filesystem rather than a database column. (That is, manifest data is kinda duplicated in the system right now!) This is because I intend to go in a more file-y way long-term anyway, to load more than just the manifests. 2. Impress 2020 guesses at the manifest URLs by pattern, and reloads them on a regular basis. Instead, we use the modeling system: when TNT changes the URL of a manifest by appending a new `?v=` query string to it, this system will consider it a new URL, and will load the new copy accordingly. Fun fact, I actually have been prototyping some of this stuff in a side project I'd named `impress-media-server`! It's a little Sinatra app that indeed *does* save all the files needed for customization, and can generate lightweight lil preview iframes and images pretty easily. I had initially been planning this as a separate service, but after thinking over the arch a bit, I think it'll go smoother to just give the main app all the same access and awareness—and I wrote it all in Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over bit-by-bit! Anyway, only Alt Styles use this for now, but my motivation is to be able to use more-correct asset URL logic to be able to finally swap over wardrobe-2020's item search to impress.openneo.net's item search API endpoint—which will get "Items You Own" searches working again, and whittle down one of the last big things Impress 2020 can do that the main app can't. Let's see how it goes!
2024-02-23 12:02:39 -08:00
end
class NotFound < StandardError; end
private
def self.info(message)
Rails.logger.info "[NeopetsMediaArchive] #{message}"
end
def self.debug(message)
Rails.logger.debug "[NeopetsMediaArchive] #{message}"
end
end