@dmathieu originally suggested using Docker to take care of Runner, I was a bit reluctant initially but the more I think about it the better I feel it fits.
Here is a rough idea for a V1 of runner
- Runner to be controlled by a simple bash file with no dependencies except for Docker
- On execution runner is either to build or download a cached image
- Take advantage of AUFS for image building (keeping diffs small)
- Bash file to control what results runner is to generate, it can submit them optionally using curl
Usage example: ./runner --start 45340 --finish 45357 --step 10 full_suite
(runs full_suite for each 10 builds outputs results on screen and to a file called results.csv)
Runner attempts to use a pre-built image first, if missing it creates one.
Keeping diffs small
Runner can create a clean image every N builds and used diffs for the rest.
So for example it can create a base image with 45300 and then for every subsequent build use that as a starting point (nuke old install do a new install)
If all filesystem changes are kept in one step AUFS should be smart enough to only store the diff between builds, leading to a very minimal storage req and way faster running of specs.
Consider running a custom registry
We can outsource building of all the images across the team and publish to a custom registry
This makes running tests on a clean box very simple.
For version 1 I would suggest we stay away from complex setups like say discourse and focus on simple .rb benches, later on we can integrate discourse bench. It gets trickier cause you need to bundle install stuff and bring up a pile of environment.
Thoughts on the basics here?