Create NeopetsMediaArchive, read the actual manifests for Alt Styles

The Neopets Media Archive is a service that mirrors `images.neopets.com`
over time! Right now we're starting by just loading manifests, and
using them to replace the hacks we used for determining the Alt Style
PNG and SVG URLs; but with time, I want to load *all* customization
media files, to have our own secondary file source that isn't dependent
on Neopets to always be up.

Impress 2020 already caches manifest files, but this strategy is
different in two ways:

1. We're using the filesystem rather than a database column. (That is,
   manifest data is kinda duplicated in the system right now!) This is
   because I intend to go in a more file-y way long-term anyway, to
   load more than just the manifests.
2. Impress 2020 guesses at the manifest URLs by pattern, and reloads
   them on a regular basis. Instead, we use the modeling system: when
   TNT changes the URL of a manifest by appending a new `?v=` query
   string to it, this system will consider it a new URL, and will load
   the new copy accordingly.

Fun fact, I actually have been prototyping some of this stuff in a side
project I'd named `impress-media-server`! It's a little Sinatra app
that indeed *does* save all the files needed for customization, and can
generate lightweight lil preview iframes and images pretty easily. I
had initially been planning this as a separate service, but after
thinking over the arch a bit, I think it'll go smoother to just give
the main app all the same access and awareness—and I wrote it all in
Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over
bit-by-bit!

Anyway, only Alt Styles use this for now, but my motivation is to be
able to use more-correct asset URL logic to be able to finally swap
over wardrobe-2020's item search to impress.openneo.net's item search
API endpoint—which will get "Items You Own" searches working again, and
whittle down one of the last big things Impress 2020 can do that the
main app can't. Let's see how it goes!
This commit is contained in:
Emi Matchu 2024-02-23 12:02:39 -08:00
parent 3f449310d6
commit 2cc46703b9
6 changed files with 164 additions and 21 deletions

View file

@ -113,31 +113,33 @@ class SwfAsset < ApplicationRecord
}
end
MANIFEST_PATTERN = %r{^https://images.neopets.com/(?<prefix>.+)/(?<id>[0-9]+)(?<hash_part>_[^/]+)?/manifest\.json}
def manifest
NeopetsMediaArchive.load_json(manifest_url)
end
MANIFEST_BASE_URL = Addressable::URI.parse("https://images.neopets.com")
def manifest_asset_urls
return {} if manifest_url.nil?
begin
# Organize the asset URLs by file extension, grab the ones we want, and
# convert them from paths to full URLs.
manifest["cpmanifest"]["assets"][0]["asset_data"].
to_h { |a| [a["file_ext"].to_sym, a] }.
slice(:png, :svg, :js)
.transform_values { |a| (MANIFEST_BASE_URL + a["url"]).to_s }
rescue StandardError => error
Rails.logger.error "Could not read URLs from manifest: #{error.full_message}"
return {}
end
end
def html5_image_url
return nil if manifest_url.nil?
# HACK: Just assuming all of these were well-formed by the same process,
# and infer the image URL from the manifest URL! But strictly speaking we
# should be reading the manifest to check!
match = manifest_url.match(MANIFEST_PATTERN)
return nil if match.nil?
"https://images.neopets.com/#{match[:prefix]}/" +
"#{match[:id]}#{match[:hash_part]}/#{match[:id]}.png"
manifest_asset_urls[:png]
end
def html5_svg_url
return nil if manifest_url.nil?
# HACK: Just assuming all of these were well-formed by the same process,
# and infer the image URL from the manifest URL! But strictly speaking we
# should be reading the manifest to check!
match = manifest_url.match(MANIFEST_PATTERN)
return nil if match.nil?
"https://images.neopets.com/#{match[:prefix]}/" +
"#{match[:id]}#{match[:hash_part]}/#{match[:id]}.svg"
manifest_asset_urls[:svg]
end
def known_glitches

View file

@ -0,0 +1,119 @@
require "addressable/template"
require "addressable/uri"
require "httparty"
require "json"
# The Neopets Media Archive is a service that mirrors images.neopets.com files
# locally. You can request a file from it, and we'll serve it from disk if we
# have it, or request and save it if not.
#
# This is a bit different than a cache, because the intention is not just
# optimization but that we *want* to be saving images.neopets.com as a
# long-term archive, not dependent on their services having 100% uptime in
# order for us to operate. We never discard old files, we just keep going!
module NeopetsMediaArchive
include HTTParty
base_uri "https://images.neopets.com/"
OLD_MANIFEST_PATH_TEMPLATE = Addressable::Template.new(
"https://images.neopets.com/cp/{short_type}/data/{id1}/{id2}/{id3}/{id}_{hash}/manifest.json"
)
NEW_MANIFEST_PATH_TEMPLATE = Addressable::Template.new(
"https://images.neopets.com/cp/{short_type}/data/{id1}/{id2}/{id3}/{id}/manifest.json"
)
# Load the file from the given `images.neopets.com` URI, as JSON.
def self.load_json(uri)
JSON.parse(load_file(uri))
end
# Load the file from the given `images.neopets.com` URI.
def self.load_file(uri, return_content: true)
local_path = local_file_path(uri)
# Read the file locally if we have it.
if return_content
begin
content = File.read(local_path)
debug "Loaded source file from filesystem: #{local_path}"
return content
rescue Errno::ENOENT
# If it doesn't exist, that's fine: just move on and download it.
end
else
# When we don't need the content, "loading" the file is just ensuring
# it exists. If it doesn't, we'll move on and load it from source.
# (We use this when preloading files, to save the cost of reading files
# we're not ready to use yet.)
if File.exist?(local_path)
debug "Source file is already loaded, skipping: #{local_path}"
return
end
end
# Download the file from the origin, then save a copy for next time.
response = load_file_from_origin(uri)
info "Loaded source file from origin: #{uri}"
content = response.body
local_path.dirname.mkpath
File.write(local_path, content)
info "Wrote source file to filesystem: #{local_path}"
return_content ? content : nil
end
# Load the file from the given `images.neopets.com` URI, but don't return its
# content. This can be faster in cases where the file's content isn't
# relevant to us, and we just want to ensure it exists.
def self.preload_file(uri)
load_file(uri, return_content: false)
end
# Load the file from the given `images.neopets.com` URI, directly from the
# source, without checking the local filesystem.
def self.load_file_from_origin(uri)
unless Addressable::URI.parse(uri).origin == "https://images.neopets.com"
raise ArgumentError, "NeopetsMediaArchive can only load from " +
"https://images.neopets.com, but got #{uri}"
end
response = get(uri)
if response.code == 404
raise NotFound, "origin server returned 404: #{uri}"
elsif response.code != 200
raise "expected status 200 but got #{response.code} (#{uri})"
end
response
end
def self.path_within_archive(uri)
uri = Addressable::URI.parse(uri)
path = uri.host + uri.path
# We include the query string as part of the file path, which is a bit odd!
# But Neopets often uses this for cache-busting, so we do need a mechanism
# for knowing whether we're holding the right version of the file. We could
# also consider storing the file by just its normal path, but with some
# metadata to track versioning information (e.g. a sqlite db, or a metadata
# file in the same directory).
path += "?" + uri.query if !uri.query.nil? && !uri.query.empty?
path
end
def self.local_file_path(uri)
Rails.configuration.neopets_media_archive_root + path_within_archive(uri)
end
class NotFound < StandardError; end
private
def self.info(message)
Rails.logger.info "[NeopetsMediaArchive] #{message}"
end
def self.debug(message)
Rails.logger.debug "[NeopetsMediaArchive] #{message}"
end
end

View file

@ -107,4 +107,9 @@ Rails.application.configure do
# override this with the IMPRESS_2020_ORIGIN environment variable!)
config.impress_2020_origin = ENV.fetch("IMPRESS_2020_ORIGIN",
"http://localhost:4000")
# Save the Neopets Media Archive in the local `tmp` folder. (In production,
# we keep this in a long-term location instead!)
config.neopets_media_archive_root = Rails.root / "tmp" /
"neopets_media_archive" / "development"
end

View file

@ -126,4 +126,8 @@ Rails.application.configure do
# IMPRESS_2020_ORIGIN environment variable!)
config.impress_2020_origin = ENV.fetch("IMPRESS_2020_ORIGIN",
"https://impress-2020.openneo.net")
# Save the Neopets Media Archive in `/var/lib/neopets-media-archive`, a
# long-term storage location.
config.neopets_media_archive_root = "/var/lib/neopets-media-archive"
end

View file

@ -66,4 +66,9 @@ Rails.application.configure do
# override this with the IMPRESS_2020_ORIGIN environment variable!)
config.impress_2020_origin = ENV.fetch("IMPRESS_2020_ORIGIN",
"http://localhost:4000")
# Save the Neopets Media Archive in the local `tmp` folder. (In production,
# we keep this in a long-term location instead!)
config.neopets_media_archive_root = Rails.root / "tmp" /
"neopets_media_archive" / "test"
end

View file

@ -402,6 +402,14 @@
password: "{{ mysql_user_password_2020 }}"
priv: "openneo_impress.*:ALL,openneo_id.*:ALL"
- name: Create the Neopets Media Archive data directory
file:
path: /var/lib/neopets-media-archive
owner: impress
group: impress
mode: "755"
state: directory
handlers:
- name: Reload nginx
systemd: