Do preloading manually on user list pages, to reduce memory usage

I used the new profiler tools on this page, and noticed a lot of
allocations in the Globalize library, which we use for translating
database records. I realized that we were loading all of the fields of
not just all of the items on the page, but all of their translation
records in all locales! We used to scrape data for lots of languages, so
that can be quite a lot!

Unfortunately, Rails's `includes` method to efficiently preload related
records always loads all fields, and simply can't be overridden.

So, in this change we write manual preloading code, to identify the
records we need, load them in big bulk queries, and assign them back to
the appropriate associations. Basically just what `includes` does, but
written out a bit more, to give us the chance to specify SELECT and
WHERE clauses!
This commit is contained in:
Emi Matchu 2023-10-27 19:42:02 -07:00
parent c496e33c37
commit 7948974949
3 changed files with 88 additions and 8 deletions

View file

@ -227,24 +227,47 @@ class ClosetHangersController < ApplicationController
end
end
def find_closet_lists_by_owned(closet_lists)
return {} if closet_lists == []
closet_lists.alphabetical.includes(:hangers => {:item => :translations}).
group_by(&:hangers_owned)
def find_closet_lists_by_owned(lists_scope)
return {} if lists_scope == []
lists = lists_scope.alphabetical
ClosetList.preload_items(
lists,
hangers_scope: hangers_scope,
items_scope: items_scope,
item_translations_scope: item_translations_scope,
)
lists.group_by(&:hangers_owned)
end
def find_unlisted_closet_hangers_by_owned(visible_groups)
unless visible_groups.empty?
@user.closet_hangers.unlisted.
hangers = @user.closet_hangers.unlisted.
owned_before_wanted.alphabetical_by_item_name.
includes(:item => :translations).
where(:owned => [visible_groups]).
group_by(&:owned)
where(:owned => [visible_groups])
ClosetHanger.preload_items(
hangers,
items_scope: items_scope,
item_translations_scope: item_translations_scope,
)
hangers.group_by(&:owned)
else
{}
end
end
def hangers_scope
ClosetHanger.select(:id, :item_id, :list_id, :quantity)
end
def items_scope
Item.select(:id, :thumbnail_url, :rarity_index)
end
def item_translations_scope
Item::Translation.select(:id, :item_id, :locale, :name, :description).
where(locale: I18n.locale)
end
def owned
owned = true
if closet_hanger_params

View file

@ -64,6 +64,33 @@ class ClosetHanger < ApplicationRecord
base
end
def self.preload_items(
hangers,
items_scope: Item.all,
item_translations_scope: Item::Translation.all
)
# Preload the records we need. (This is like `includes`, but `includes`
# always selects all fields for all records, and we give the caller the
# opportunity to specify which fields it actually wants via scope!)
items = items_scope.where(id: hangers.map(&:item_id))
translations = item_translations_scope.where(item_id: items.map(&:id))
# Group the records by relevant IDs.
translations_by_item_id = translations.group_by(&:item_id)
items_by_id = items.to_h { |i| [i.id, i] }
# Assign the preloaded records to the records they belong to. (This is like
# doing e.g. i.translations = ..., but that's a database write - we
# actually just want to set the `translations` field itself directly!
# Hacky, ripped from how `ActiveRecord::Associations::Preloader` does it!)
items.each do |item|
item.association(:translations).target = translations_by_item_id[item.id]
end
hangers.each do |hanger|
hanger.association(:item).target = items_by_id[hanger.item_id]
end
end
def self.set_quantity!(quantity, options)
quantity = quantity.to_i
conditions = {:user_id => options[:user_id].to_i,

View file

@ -31,6 +31,36 @@ class ClosetList < ApplicationRecord
send(method_name)
end
def self.preload_items(
lists,
hangers_scope: ClosetHanger.all,
items_scope: Item.all,
item_translations_scope: Item::Translation.all
)
# Preload the records we need. (This is like `includes`, but `includes`
# always selects all fields for all records, and we give the caller the
# opportunity to specify which fields it actually wants via scope!)
hangers = hangers_scope.where(list_id: lists.map(&:id))
# Group the records by relevant IDs.
hangers_by_list_id = hangers.group_by(&:list_id)
# Assign the preloaded records to the records they belong to. (This is like
# doing e.g. i.translations = ..., but that's a database write - we
# actually just want to set the `translations` field itself directly!
# Hacky, ripped from how `ActiveRecord::Associations::Preloader` does it!)
lists.each do |list|
list.association(:hangers).target = hangers_by_list_id[list.id]
end
# Then, do similar preloading for the hangers and their items.
ClosetHanger.preload_items(
hangers,
items_scope: items_scope,
item_translations_scope: item_translations_scope,
)
end
module VisibilityMethods
delegate :trading?, to: :visibility_level