Create NeopetsMediaArchive, read the actual manifests for Alt Styles
The Neopets Media Archive is a service that mirrors `images.neopets.com` over time! Right now we're starting by just loading manifests, and using them to replace the hacks we used for determining the Alt Style PNG and SVG URLs; but with time, I want to load *all* customization media files, to have our own secondary file source that isn't dependent on Neopets to always be up. Impress 2020 already caches manifest files, but this strategy is different in two ways: 1. We're using the filesystem rather than a database column. (That is, manifest data is kinda duplicated in the system right now!) This is because I intend to go in a more file-y way long-term anyway, to load more than just the manifests. 2. Impress 2020 guesses at the manifest URLs by pattern, and reloads them on a regular basis. Instead, we use the modeling system: when TNT changes the URL of a manifest by appending a new `?v=` query string to it, this system will consider it a new URL, and will load the new copy accordingly. Fun fact, I actually have been prototyping some of this stuff in a side project I'd named `impress-media-server`! It's a little Sinatra app that indeed *does* save all the files needed for customization, and can generate lightweight lil preview iframes and images pretty easily. I had initially been planning this as a separate service, but after thinking over the arch a bit, I think it'll go smoother to just give the main app all the same access and awareness—and I wrote it all in Ruby and plain HTML/JS/CSS, so it should be pretty easy to port over bit-by-bit! Anyway, only Alt Styles use this for now, but my motivation is to be able to use more-correct asset URL logic to be able to finally swap over wardrobe-2020's item search to impress.openneo.net's item search API endpoint—which will get "Items You Own" searches working again, and whittle down one of the last big things Impress 2020 can do that the main app can't. Let's see how it goes!
This commit is contained in:
parent
3f449310d6
commit
2cc46703b9
6 changed files with 164 additions and 21 deletions
|
@ -113,31 +113,33 @@ class SwfAsset < ApplicationRecord
|
||||||
}
|
}
|
||||||
end
|
end
|
||||||
|
|
||||||
MANIFEST_PATTERN = %r{^https://images.neopets.com/(?<prefix>.+)/(?<id>[0-9]+)(?<hash_part>_[^/]+)?/manifest\.json}
|
def manifest
|
||||||
|
NeopetsMediaArchive.load_json(manifest_url)
|
||||||
|
end
|
||||||
|
|
||||||
|
MANIFEST_BASE_URL = Addressable::URI.parse("https://images.neopets.com")
|
||||||
|
def manifest_asset_urls
|
||||||
|
return {} if manifest_url.nil?
|
||||||
|
|
||||||
|
begin
|
||||||
|
# Organize the asset URLs by file extension, grab the ones we want, and
|
||||||
|
# convert them from paths to full URLs.
|
||||||
|
manifest["cpmanifest"]["assets"][0]["asset_data"].
|
||||||
|
to_h { |a| [a["file_ext"].to_sym, a] }.
|
||||||
|
slice(:png, :svg, :js)
|
||||||
|
.transform_values { |a| (MANIFEST_BASE_URL + a["url"]).to_s }
|
||||||
|
rescue StandardError => error
|
||||||
|
Rails.logger.error "Could not read URLs from manifest: #{error.full_message}"
|
||||||
|
return {}
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
def html5_image_url
|
def html5_image_url
|
||||||
return nil if manifest_url.nil?
|
manifest_asset_urls[:png]
|
||||||
|
|
||||||
# HACK: Just assuming all of these were well-formed by the same process,
|
|
||||||
# and infer the image URL from the manifest URL! But strictly speaking we
|
|
||||||
# should be reading the manifest to check!
|
|
||||||
match = manifest_url.match(MANIFEST_PATTERN)
|
|
||||||
return nil if match.nil?
|
|
||||||
|
|
||||||
"https://images.neopets.com/#{match[:prefix]}/" +
|
|
||||||
"#{match[:id]}#{match[:hash_part]}/#{match[:id]}.png"
|
|
||||||
end
|
end
|
||||||
|
|
||||||
def html5_svg_url
|
def html5_svg_url
|
||||||
return nil if manifest_url.nil?
|
manifest_asset_urls[:svg]
|
||||||
|
|
||||||
# HACK: Just assuming all of these were well-formed by the same process,
|
|
||||||
# and infer the image URL from the manifest URL! But strictly speaking we
|
|
||||||
# should be reading the manifest to check!
|
|
||||||
match = manifest_url.match(MANIFEST_PATTERN)
|
|
||||||
return nil if match.nil?
|
|
||||||
|
|
||||||
"https://images.neopets.com/#{match[:prefix]}/" +
|
|
||||||
"#{match[:id]}#{match[:hash_part]}/#{match[:id]}.svg"
|
|
||||||
end
|
end
|
||||||
|
|
||||||
def known_glitches
|
def known_glitches
|
||||||
|
|
119
app/services/neopets_media_archive.rb
Normal file
119
app/services/neopets_media_archive.rb
Normal file
|
@ -0,0 +1,119 @@
|
||||||
|
require "addressable/template"
|
||||||
|
require "addressable/uri"
|
||||||
|
require "httparty"
|
||||||
|
require "json"
|
||||||
|
|
||||||
|
# The Neopets Media Archive is a service that mirrors images.neopets.com files
|
||||||
|
# locally. You can request a file from it, and we'll serve it from disk if we
|
||||||
|
# have it, or request and save it if not.
|
||||||
|
#
|
||||||
|
# This is a bit different than a cache, because the intention is not just
|
||||||
|
# optimization but that we *want* to be saving images.neopets.com as a
|
||||||
|
# long-term archive, not dependent on their services having 100% uptime in
|
||||||
|
# order for us to operate. We never discard old files, we just keep going!
|
||||||
|
module NeopetsMediaArchive
|
||||||
|
include HTTParty
|
||||||
|
base_uri "https://images.neopets.com/"
|
||||||
|
|
||||||
|
OLD_MANIFEST_PATH_TEMPLATE = Addressable::Template.new(
|
||||||
|
"https://images.neopets.com/cp/{short_type}/data/{id1}/{id2}/{id3}/{id}_{hash}/manifest.json"
|
||||||
|
)
|
||||||
|
NEW_MANIFEST_PATH_TEMPLATE = Addressable::Template.new(
|
||||||
|
"https://images.neopets.com/cp/{short_type}/data/{id1}/{id2}/{id3}/{id}/manifest.json"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Load the file from the given `images.neopets.com` URI, as JSON.
|
||||||
|
def self.load_json(uri)
|
||||||
|
JSON.parse(load_file(uri))
|
||||||
|
end
|
||||||
|
|
||||||
|
# Load the file from the given `images.neopets.com` URI.
|
||||||
|
def self.load_file(uri, return_content: true)
|
||||||
|
local_path = local_file_path(uri)
|
||||||
|
|
||||||
|
# Read the file locally if we have it.
|
||||||
|
if return_content
|
||||||
|
begin
|
||||||
|
content = File.read(local_path)
|
||||||
|
debug "Loaded source file from filesystem: #{local_path}"
|
||||||
|
return content
|
||||||
|
rescue Errno::ENOENT
|
||||||
|
# If it doesn't exist, that's fine: just move on and download it.
|
||||||
|
end
|
||||||
|
else
|
||||||
|
# When we don't need the content, "loading" the file is just ensuring
|
||||||
|
# it exists. If it doesn't, we'll move on and load it from source.
|
||||||
|
# (We use this when preloading files, to save the cost of reading files
|
||||||
|
# we're not ready to use yet.)
|
||||||
|
if File.exist?(local_path)
|
||||||
|
debug "Source file is already loaded, skipping: #{local_path}"
|
||||||
|
return
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# Download the file from the origin, then save a copy for next time.
|
||||||
|
response = load_file_from_origin(uri)
|
||||||
|
info "Loaded source file from origin: #{uri}"
|
||||||
|
content = response.body
|
||||||
|
local_path.dirname.mkpath
|
||||||
|
File.write(local_path, content)
|
||||||
|
info "Wrote source file to filesystem: #{local_path}"
|
||||||
|
|
||||||
|
return_content ? content : nil
|
||||||
|
end
|
||||||
|
|
||||||
|
# Load the file from the given `images.neopets.com` URI, but don't return its
|
||||||
|
# content. This can be faster in cases where the file's content isn't
|
||||||
|
# relevant to us, and we just want to ensure it exists.
|
||||||
|
def self.preload_file(uri)
|
||||||
|
load_file(uri, return_content: false)
|
||||||
|
end
|
||||||
|
|
||||||
|
# Load the file from the given `images.neopets.com` URI, directly from the
|
||||||
|
# source, without checking the local filesystem.
|
||||||
|
def self.load_file_from_origin(uri)
|
||||||
|
unless Addressable::URI.parse(uri).origin == "https://images.neopets.com"
|
||||||
|
raise ArgumentError, "NeopetsMediaArchive can only load from " +
|
||||||
|
"https://images.neopets.com, but got #{uri}"
|
||||||
|
end
|
||||||
|
|
||||||
|
response = get(uri)
|
||||||
|
if response.code == 404
|
||||||
|
raise NotFound, "origin server returned 404: #{uri}"
|
||||||
|
elsif response.code != 200
|
||||||
|
raise "expected status 200 but got #{response.code} (#{uri})"
|
||||||
|
end
|
||||||
|
response
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.path_within_archive(uri)
|
||||||
|
uri = Addressable::URI.parse(uri)
|
||||||
|
path = uri.host + uri.path
|
||||||
|
|
||||||
|
# We include the query string as part of the file path, which is a bit odd!
|
||||||
|
# But Neopets often uses this for cache-busting, so we do need a mechanism
|
||||||
|
# for knowing whether we're holding the right version of the file. We could
|
||||||
|
# also consider storing the file by just its normal path, but with some
|
||||||
|
# metadata to track versioning information (e.g. a sqlite db, or a metadata
|
||||||
|
# file in the same directory).
|
||||||
|
path += "?" + uri.query if !uri.query.nil? && !uri.query.empty?
|
||||||
|
|
||||||
|
path
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.local_file_path(uri)
|
||||||
|
Rails.configuration.neopets_media_archive_root + path_within_archive(uri)
|
||||||
|
end
|
||||||
|
|
||||||
|
class NotFound < StandardError; end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def self.info(message)
|
||||||
|
Rails.logger.info "[NeopetsMediaArchive] #{message}"
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.debug(message)
|
||||||
|
Rails.logger.debug "[NeopetsMediaArchive] #{message}"
|
||||||
|
end
|
||||||
|
end
|
|
@ -107,4 +107,9 @@ Rails.application.configure do
|
||||||
# override this with the IMPRESS_2020_ORIGIN environment variable!)
|
# override this with the IMPRESS_2020_ORIGIN environment variable!)
|
||||||
config.impress_2020_origin = ENV.fetch("IMPRESS_2020_ORIGIN",
|
config.impress_2020_origin = ENV.fetch("IMPRESS_2020_ORIGIN",
|
||||||
"http://localhost:4000")
|
"http://localhost:4000")
|
||||||
|
|
||||||
|
# Save the Neopets Media Archive in the local `tmp` folder. (In production,
|
||||||
|
# we keep this in a long-term location instead!)
|
||||||
|
config.neopets_media_archive_root = Rails.root / "tmp" /
|
||||||
|
"neopets_media_archive" / "development"
|
||||||
end
|
end
|
||||||
|
|
|
@ -126,4 +126,8 @@ Rails.application.configure do
|
||||||
# IMPRESS_2020_ORIGIN environment variable!)
|
# IMPRESS_2020_ORIGIN environment variable!)
|
||||||
config.impress_2020_origin = ENV.fetch("IMPRESS_2020_ORIGIN",
|
config.impress_2020_origin = ENV.fetch("IMPRESS_2020_ORIGIN",
|
||||||
"https://impress-2020.openneo.net")
|
"https://impress-2020.openneo.net")
|
||||||
|
|
||||||
|
# Save the Neopets Media Archive in `/var/lib/neopets-media-archive`, a
|
||||||
|
# long-term storage location.
|
||||||
|
config.neopets_media_archive_root = "/var/lib/neopets-media-archive"
|
||||||
end
|
end
|
||||||
|
|
|
@ -66,4 +66,9 @@ Rails.application.configure do
|
||||||
# override this with the IMPRESS_2020_ORIGIN environment variable!)
|
# override this with the IMPRESS_2020_ORIGIN environment variable!)
|
||||||
config.impress_2020_origin = ENV.fetch("IMPRESS_2020_ORIGIN",
|
config.impress_2020_origin = ENV.fetch("IMPRESS_2020_ORIGIN",
|
||||||
"http://localhost:4000")
|
"http://localhost:4000")
|
||||||
|
|
||||||
|
# Save the Neopets Media Archive in the local `tmp` folder. (In production,
|
||||||
|
# we keep this in a long-term location instead!)
|
||||||
|
config.neopets_media_archive_root = Rails.root / "tmp" /
|
||||||
|
"neopets_media_archive" / "test"
|
||||||
end
|
end
|
||||||
|
|
|
@ -402,6 +402,14 @@
|
||||||
password: "{{ mysql_user_password_2020 }}"
|
password: "{{ mysql_user_password_2020 }}"
|
||||||
priv: "openneo_impress.*:ALL,openneo_id.*:ALL"
|
priv: "openneo_impress.*:ALL,openneo_id.*:ALL"
|
||||||
|
|
||||||
|
- name: Create the Neopets Media Archive data directory
|
||||||
|
file:
|
||||||
|
path: /var/lib/neopets-media-archive
|
||||||
|
owner: impress
|
||||||
|
group: impress
|
||||||
|
mode: "755"
|
||||||
|
state: directory
|
||||||
|
|
||||||
handlers:
|
handlers:
|
||||||
- name: Reload nginx
|
- name: Reload nginx
|
||||||
systemd:
|
systemd:
|
||||||
|
|
Loading…
Reference in a new issue