I'm starting to port over the functionality that was previously just,
me running `yarn db:export:public-data` in `impress-2020` and
committing it to Git LFS every time.
My immediate motivation is that the `impress-2020` git repository is
getting weirdly large?? Idk how these 40MB files have blown up to a
solid 16GB of Git LFS data (we don't have THAT many!!!), but I guess
there's something about Git LFS's architecture and disk usage that I'm
not understanding.
So, let's move to a simpler system in which we don't bind the public
data to the codebase, but instead just regularly dump it in production
and make it available for download.
This change adds the `rails public_data:commit` task, which when run in
production will make the latest available at
`https://impress.openneo.net/public-data/latest.sql.gz`, and will also
store a running log of previous dumps, viewable at
`https://impress.openneo.net/public-data/`.
Things left to do:
1. Create a `rails public_data:pull` task, to download `latest.sql.gz`
and import it into the local development database.
2. Set up a cron job to dump this out regularly, idk maybe weekly? That
will grow, but not very fast (about 2GB per year), and we can add
logic to rotate out old ones if it starts to grow too far. (If we
wanted to get really intricate, we could do like, daily for the past
week, then weekly for the past 3 months, then monthly for the past
year, idk. There must be tools that do this!)
This was necessary when we were running old Rubies that I couldn't build on macOS, but now we're on standard modern stuff, so I'm not gonna leave around a config that we no longer use and keep updated!
We add jsbuilding-rails to get esbuild running in the app, and then we copy-paste the files we need from impress-2020 into here!
I stopped at the point where it was building successfully, but it's not running correctly: it's not sure about `process.env` in `next`, and I think the right next step is to delete the NextJS deps altogether and use React Router instead.
The intent is to set up for an upgrade of Ruby and Rails to the modern versions, but I want to start by having a stable running copy that we can incrementally pull up to new versions of things!
And it turns out getting Ruby 1.9.3 to build on modern platforms is hard! I started by trying on macOS and just couldn't get there, the instructions I found for workarounds didn't seem to work anymore.
So the solution I landed on was to set up an Ubuntu VM, and follow some instructions from https://stackoverflow.com/q/51986932 to patch Ruby to work with the version of OpenSSL we have access to!
And it was enough of a challenge that I figured that, rather than setting up the Vagrantfile elsewhere, it would be helpful documentation to do it here, even if we scrap the Vagrantfile etc later once we're in a new stable environment.
Emi Matchu
Matchu