OpenNeo/impress
OpenNeo/impress
1
0
Fork 1
Code Pull requests Activity
impress/vendor/cache
History
Emi Matchu 8dc11f9940 Create rails public_data:commit task, to share public data dumps 
I'm starting to port over the functionality that was previously just,
me running `yarn db:export:public-data` in `impress-2020` and
committing it to Git LFS every time.

My immediate motivation is that the `impress-2020` git repository is
getting weirdly large?? Idk how these 40MB files have blown up to a
solid 16GB of Git LFS data (we don't have THAT many!!!), but I guess
there's something about Git LFS's architecture and disk usage that I'm
not understanding.

So, let's move to a simpler system in which we don't bind the public
data to the codebase, but instead just regularly dump it in production
and make it available for download.

This change adds the `rails public_data:commit` task, which when run in
production will make the latest available at
`https://impress.openneo.net/public-data/latest.sql.gz`, and will also
store a running log of previous dumps, viewable at
`https://impress.openneo.net/public-data/`.

Things left to do:
1. Create a `rails public_data:pull` task, to download `latest.sql.gz`
   and import it into the local development database.
2. Set up a cron job to dump this out regularly, idk maybe weekly? That
   will grow, but not very fast (about 2GB per year), and we can add
   logic to rotate out old ones if it starts to grow too far. (If we
   wanted to get really intricate, we could do like, daily for the past
   week, then weekly for the past 3 months, then monthly for the past
   year, idk. There must be tools that do this!)
2024-02-29 14:30:33 -08:00
..
actioncable-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
actionmailbox-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
actionmailer-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
actionpack-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
actiontext-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
actionview-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
activejob-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
activemodel-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
activerecord-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
activestorage-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
activesupport-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
addressable-2.8.6.gem Run bundle update 2024-02-22 11:48:32 -08:00
async-2.8.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
async-container-0.16.13.gem Run bundle update 2024-02-22 11:48:32 -08:00
async-http-0.61.0.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
async-http-cache-0.4.3.gem Replace falcon server with puma 2024-01-23 21:55:26 -08:00
async-io-1.41.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
async-pool-0.4.0.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
babel-source-5.8.35.gem Upgrade to Ruby 2.2.4, Rails 4.0.13 2023-10-23 19:05:02 -07:00
babel-transpiler-0.7.0.gem Upgrade to Ruby 2.2.4, Rails 4.0.13 2023-10-23 19:05:02 -07:00
base64-0.2.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
bcrypt-3.1.20.gem Run bundle update 2024-02-22 11:48:32 -08:00
bigdecimal-3.1.6.gem Run bundle update 2024-02-22 11:48:32 -08:00
bindex-0.8.1.gem Upgrade web-console 2023-10-23 19:05:05 -07:00
bootsnap-1.18.3.gem Run bundle update 2024-02-22 11:48:32 -08:00
build-environment-1.13.0.gem Replace falcon server with puma 2024-01-23 21:55:26 -08:00
builder-3.2.4.gem Update gems to Rails 4.1 (crashes!) 2023-10-23 19:05:04 -07:00
concurrent-ruby-1.2.3.gem Run bundle update 2024-02-22 11:48:32 -08:00
connection_pool-2.4.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
console-1.23.4.gem Run bundle update 2024-02-22 11:48:32 -08:00
crass-1.0.6.gem Upgrade to Rails 4.2.11.3 and Ruby 2.4.10 2023-10-23 19:05:05 -07:00
date-3.3.4.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
devise-4.9.3.gem Upgrade to Rails 7.1.1 2023-10-25 15:05:31 -07:00
devise-encryptable-0.2.0.gem Can log into OpenNeo ID accounts directly! 2023-10-23 19:05:07 -07:00
dotenv-2.8.1.gem Remove dotenv-deployment, upgrade dotenv-rails 2023-10-23 19:05:05 -07:00
dotenv-rails-2.8.1.gem Remove dotenv-deployment, upgrade dotenv-rails 2023-10-23 19:05:05 -07:00
drb-2.2.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
e2mmap-0.1.0.gem Create rails public_data:commit task, to share public data dumps 2024-02-29 14:30:33 -08:00
erubi-1.12.0.gem Upgrade to Rails 5.2.8.1 2023-10-23 19:05:05 -07:00
execjs-2.9.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
falcon-0.43.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
ffi-1.16.3.gem Run bundle update 2024-02-22 11:48:32 -08:00
fiber-annotation-0.2.0.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
fiber-local-1.0.0.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
globalid-1.2.1.gem Security updates 2023-10-23 19:08:56 -07:00
haml-6.3.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
http_accept_language-2.1.1.gem Version the http_accept_languages gem 2023-10-23 19:05:05 -07:00
httparty-0.21.0.gem Upgrade to Rails 7.1.1 2023-10-25 15:05:31 -07:00
i18n-1.14.1.gem Upgrade to Rails 5.2.8.1 2023-10-23 19:05:05 -07:00
io-console-0.7.2.gem Run bundle update 2024-02-22 11:48:32 -08:00
io-event-1.4.4.gem Run bundle update 2024-02-22 11:48:32 -08:00
irb-1.11.2.gem Run bundle update 2024-02-22 11:48:32 -08:00
jsbundling-rails-1.3.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
json-2.7.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
launchy-2.5.2.gem Update all our gems as far as we can 2023-10-23 19:05:06 -07:00
letter_opener-1.9.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
localhost-1.2.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
loofah-2.22.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
mail-2.8.1.gem Fix gems we'd locked to old versions 2023-10-23 19:05:06 -07:00
mapping-1.1.1.gem Replace falcon server with puma 2024-01-23 21:55:26 -08:00
marcel-1.0.2.gem Upgrade to Rails 5.2.8.1 2023-10-23 19:05:05 -07:00
memory_profiler-1.0.1.gem Add mini profiler to each page 2023-10-27 19:38:49 -07:00
mini_mime-1.1.5.gem Security updates 2023-10-23 19:08:56 -07:00
minitest-5.22.2.gem Run bundle update 2024-02-22 11:48:32 -08:00
msgpack-1.7.2.gem Update all our gems as far as we can 2023-10-23 19:05:06 -07:00
multi_xml-0.6.0.gem Upgrade to Rails 7.1.1 2023-10-25 15:05:31 -07:00
mutex_m-0.2.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
mysql2-0.5.6.gem Run bundle update 2024-02-22 11:48:32 -08:00
net-imap-0.4.10.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
net-pop-0.1.2.gem Fix gems we'd locked to old versions 2023-10-23 19:05:06 -07:00
net-protocol-0.2.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
net-smtp-0.4.0.1.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
nio4r-2.7.0.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
nokogiri-1.16.2-x86_64-linux.gem Run bundle update 2024-02-22 11:48:32 -08:00
openssl-3.2.0.gem Replace falcon server with puma 2024-01-23 21:55:26 -08:00
orm_adapter-0.5.0.gem improve app infrastructure to make a dev environment easier to reproduce 2015-07-16 17:44:52 -04:00
parallel-1.24.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
process-metrics-0.2.1.gem Replace falcon server with puma 2024-01-23 21:55:26 -08:00
protocol-hpack-1.4.2.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
protocol-http-0.25.0.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
protocol-http1-0.16.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
protocol-http2-0.15.1.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
protocol-rack-0.4.1.gem Replace falcon server with puma 2024-01-23 21:55:26 -08:00
psych-5.1.2.gem Run bundle update 2024-02-22 11:48:32 -08:00
public_suffix-5.0.4.gem Run bundle update 2024-02-22 11:48:32 -08:00
racc-1.7.3.gem Run bundle update 2024-02-22 11:48:32 -08:00
rack-3.0.9.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
rack-attack-6.7.0.gem Update all our gems as far as we can 2023-10-23 19:05:06 -07:00
rack-mini-profiler-3.3.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
rack-session-2.0.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
rack-test-2.1.0.gem Upgrade to Rails 5.2.8.1 2023-10-23 19:05:05 -07:00
rackup-2.1.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
rails-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
rails-dom-testing-2.2.0.gem Upgrade to Rails 7.0.6 2023-10-23 19:05:07 -07:00
rails-html-sanitizer-1.6.0.gem Upgrade to Rails 7.0.6 2023-10-23 19:05:07 -07:00
rails-i18n-7.0.8.gem Upgrade to Rails 7.1.1 2023-10-25 15:05:31 -07:00
railties-7.1.3.2.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
rake-13.1.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
rdiscount-2.2.7.3.gem Run bundle update 2024-02-22 11:48:32 -08:00
rdoc-6.6.2.gem Run bundle update 2024-02-22 11:48:32 -08:00
react-rails-2.7.1.gem Upgrade to Ruby 2.2.4, Rails 4.0.13 2023-10-23 19:05:02 -07:00
record_tag_helper-1.0.1.gem Upgrade to Rails 5.2.8.1 2023-10-23 19:05:05 -07:00
reline-0.4.2.gem Run bundle update 2024-02-22 11:48:32 -08:00
responders-3.1.1.gem Upgrade to Rails 7.1.1 2023-10-25 15:05:31 -07:00
ruby2_keywords-0.0.5.gem Upgrade to Rails 7.1.1 2023-10-25 15:05:31 -07:00
samovar-2.2.0.gem Replace falcon server with puma 2024-01-23 21:55:26 -08:00
sanitize-6.1.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
sass-rails-6.0.0.gem Upgrade to latest Sass and Sprockets 2023-10-23 19:05:09 -07:00
sassc-2.4.0.gem Upgrade to latest Sass and Sprockets 2023-10-23 19:05:09 -07:00
sassc-rails-2.1.2.gem Upgrade to latest Sass and Sprockets 2023-10-23 19:05:09 -07:00
sentry-rails-5.16.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
sentry-ruby-5.16.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
shell-0.8.1.gem Create rails public_data:commit task, to share public data dumps 2024-02-29 14:30:33 -08:00
sprockets-4.2.1.gem Run bundle update 2024-02-22 11:48:32 -08:00
sprockets-rails-3.4.2.gem Upgrade to Rails 6.0.6.1 2023-10-23 19:05:06 -07:00
sqlite3-1.7.2-x86_64-linux.gem Run bundle update 2024-02-22 11:48:32 -08:00
stackprof-0.2.26.gem Run bundle update 2024-02-22 11:48:32 -08:00
stringio-3.1.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
sync-0.5.0.gem Create rails public_data:commit task, to share public data dumps 2024-02-29 14:30:33 -08:00
temple-0.10.3.gem Run bundle update 2024-02-22 11:48:32 -08:00
terser-1.2.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
thor-1.3.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
thread-local-1.1.0.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
tilt-2.3.0.gem Run bundle update 2024-02-22 11:48:32 -08:00
timeout-0.4.1.gem Upgrade to Rails 7.1.3.2 2024-02-22 11:25:52 -08:00
timers-4.3.5.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
traces-0.11.1.gem Move manifest backfill to swf_assets:manifests task 2023-11-10 16:52:50 -08:00
tzinfo-2.0.6.gem Upgrade to Rails 6.1.7.4 2023-10-23 19:05:07 -07:00
warden-1.2.9.gem Run bundle update 2024-02-22 11:48:32 -08:00
web-console-4.2.1.gem Security updates 2023-10-23 19:08:56 -07:00
webrick-1.8.1.gem Upgrade to Rails 7.1.1 2023-10-25 15:05:31 -07:00
websocket-driver-0.7.6.gem Upgrade to Rails 5.2.8.1 2023-10-23 19:05:05 -07:00
websocket-extensions-0.1.5.gem Upgrade to Rails 5.2.8.1 2023-10-23 19:05:05 -07:00
will_paginate-4.0.0.gem Upgrade to Rails 5.2.8.1 2023-10-23 19:05:05 -07:00
zeitwerk-2.6.13.gem Run bundle update 2024-02-22 11:48:32 -08:00