Do preloading manually on user list pages, to reduce memory usage

I used the new profiler tools on this page, and noticed a lot of
allocations in the Globalize library, which we use for translating
database records. I realized that we were loading all of the fields of
not just all of the items on the page, but all of their translation
records in all locales! We used to scrape data for lots of languages, so
that can be quite a lot!

Unfortunately, Rails's `includes` method to efficiently preload related
records always loads all fields, and simply can't be overridden.

So, in this change we write manual preloading code, to identify the
records we need, load them in big bulk queries, and assign them back to
the appropriate associations. Basically just what `includes` does, but
written out a bit more, to give us the chance to specify SELECT and
WHERE clauses!

app/controllers/closet_hangers_controller.rb

@@ -227,24 +227,47 @@ class ClosetHangersController < ApplicationController
     end
   end
 
-  def find_closet_lists_by_owned(closet_lists)
+  def find_closet_lists_by_owned(lists_scope)
-    return {} if closet_lists == []
+    return {} if lists_scope == []
-    closet_lists.alphabetical.includes(:hangers => {:item => :translations}).
+    lists = lists_scope.alphabetical
-      group_by(&:hangers_owned)
+    ClosetList.preload_items(
+      lists,
+      hangers_scope: hangers_scope,
+      items_scope: items_scope,
+      item_translations_scope: item_translations_scope,
+    )
+    lists.group_by(&:hangers_owned)
   end
   
   def find_unlisted_closet_hangers_by_owned(visible_groups)
     unless visible_groups.empty?
-      @user.closet_hangers.unlisted.
+      hangers = @user.closet_hangers.unlisted.
         owned_before_wanted.alphabetical_by_item_name.
-        includes(:item => :translations).
+        where(:owned => [visible_groups])
-        where(:owned => [visible_groups]).
+      ClosetHanger.preload_items(
-        group_by(&:owned)
+        hangers,
+        items_scope: items_scope,
+        item_translations_scope: item_translations_scope,
+      )
+      hangers.group_by(&:owned)
     else
       {}
     end
   end
 
+  def hangers_scope
+    ClosetHanger.select(:id, :item_id, :list_id, :quantity)
+  end
+
+  def items_scope
+    Item.select(:id, :thumbnail_url, :rarity_index)
+  end
+
+  def item_translations_scope
+    Item::Translation.select(:id, :item_id, :locale, :name, :description).
+      where(locale: I18n.locale)
+  end
+
   def owned
     owned = true
     if closet_hanger_params

app/models/closet_hanger.rb

@@ -64,6 +64,33 @@ class ClosetHanger < ApplicationRecord
     base
   end
 
+  def self.preload_items(
+    hangers,
+    items_scope: Item.all,
+    item_translations_scope: Item::Translation.all
+  )
+    # Preload the records we need. (This is like `includes`, but `includes`
+    # always selects all fields for all records, and we give the caller the
+    # opportunity to specify which fields it actually wants via scope!)
+    items = items_scope.where(id: hangers.map(&:item_id))
+    translations = item_translations_scope.where(item_id: items.map(&:id))
+
+    # Group the records by relevant IDs.
+    translations_by_item_id = translations.group_by(&:item_id)
+    items_by_id = items.to_h { |i| [i.id, i] }
+
+    # Assign the preloaded records to the records they belong to. (This is like
+    # doing e.g. i.translations = ..., but that's a database write - we
+    # actually just want to set the `translations` field itself directly!
+    # Hacky, ripped from how `ActiveRecord::Associations::Preloader` does it!)
+    items.each do |item|
+      item.association(:translations).target = translations_by_item_id[item.id]
+    end
+    hangers.each do |hanger|
+      hanger.association(:item).target = items_by_id[hanger.item_id]
+    end
+  end
+  
   def self.set_quantity!(quantity, options)
     quantity = quantity.to_i
     conditions = {:user_id => options[:user_id].to_i,

app/models/closet_list.rb

@@ -31,6 +31,36 @@ class ClosetList < ApplicationRecord
     send(method_name)
   end
 
+  def self.preload_items(
+    lists,
+    hangers_scope: ClosetHanger.all,
+    items_scope: Item.all,
+    item_translations_scope: Item::Translation.all
+  )
+    # Preload the records we need. (This is like `includes`, but `includes`
+    # always selects all fields for all records, and we give the caller the
+    # opportunity to specify which fields it actually wants via scope!)
+    hangers = hangers_scope.where(list_id: lists.map(&:id))
+
+    # Group the records by relevant IDs.
+    hangers_by_list_id = hangers.group_by(&:list_id)
+
+    # Assign the preloaded records to the records they belong to. (This is like
+    # doing e.g. i.translations = ..., but that's a database write - we
+    # actually just want to set the `translations` field itself directly!
+    # Hacky, ripped from how `ActiveRecord::Associations::Preloader` does it!)
+    lists.each do |list|
+      list.association(:hangers).target = hangers_by_list_id[list.id]
+    end
+
+    # Then, do similar preloading for the hangers and their items.
+    ClosetHanger.preload_items(
+      hangers,
+      items_scope: items_scope,
+      item_translations_scope: item_translations_scope,
+    )
+  end
+
   module VisibilityMethods
     delegate :trading?, to: :visibility_level