I like the idea of using ruby-build + rbenv to build each Ruby version and be able to keep them all cached to quickly run benchmarks through all of them.
I’m currently working on a script to automate the builds for this, and did a few back-of-the-envelope calculations on the storage requirements for keeping this many builds around:
In 2013, there were around 5000 commits to Ruby trunk. Each compiled build of Ruby in that series is ~23 MB (that is, if you build with the --disable-install-doc configure option, which cuts down the size by almost an order of magnitude). So 5000 commits * 23 MB per build = ~112 GB.
The builds compress pretty well with tar + gzip, and not all need to be expanded on disk at once. Tarballs for each build would be ~6.5 MB, for a total of ~31 GB.
I bet that there’s a ton of duplication between builds, so an even better (slightly crazy?) approach might be to store the compiled builds in a git repo - one build per commit, layered on top of each other in chronological order. I believe git would take care of compression and de-duping for us, and I bet it would be able to significantly bring down the store cost to the range of 5-10 GB.
Once I’ve got a goodly number of builds locally (say around 100) I’m going to try the git experiment and see how it goes.