I’ve actually started from scratch and have came up with an MVP over at https://railsbench.herokuapp.com/. Currently, I’m running the benchmarks on a bare metal server from Softlayer (sponsors welcome) and would love to have more collaborators on it. I really want this to take off and have plenty of time (I’m a student so time is on my side) to work on the project. Below is a brief overview of what I’ve done so far.
Overview
The following is a rough overview of how I implemented the application:
The Web UI has a Github Event Handler which receives the commits pushed to the repository through a webhook. Once the webhook has been received, the application run jobs against the particular commit which executes the benchmarks on a bare metal server through SSH.
In order to have consistent benchmarks, I’m renting the cheapest bare metal server from Softlayer. I’m not sure if we really need a bare metal server but Sam Saffron mentioned something about it in his blog post. The benchmarks are executed in Docker containers which is similar to what Bert did for GSoC and inline with the discussion here Runner on Docker.
Rails Benchmark
Rails Benchmark URL: https://railsbench.herokuapp.com/tgxworld/rails
For the Rails Benchmark, I forked ko1-test-app by Aaron and modified it to post the results of objects allocation for an index action which fetches and renders 100 users and their attributes. I read the previous thread and felt that creating our own app is the cheapest way to get things going. Discourse is currently running on Rails master but it will be tough if we’re benchmarking “two moving targets”(~ Matthew). Perhaps we might be able to use a fixed code base of Discourse and run the benchmarks against it.
Ruby Benchmark
Ruby Benchmark URL: https://railsbench.herokuapp.com/tgxworld/ruby
For the Ruby Benchmark, I used the existing ruby benchmarks. Not sure if this is relevant to the Rails Community but I thought it would be good to implement it anyway. Currently I’m running the benchmarks once as the whole suite takes close to 6 mins on my machine.
Next Steps
What other metrics do we need to track? (ko1-test-app has scripts for GC, Number of Request/s…)
Setup webhook on Rails Repo so that I don’t have to track the changes from a fork.
Thoughts and feedback in general?
More eyes to look at my code 
Some sort of variation tracking system to notify contributors of anomalies
Per release benchmark of major apps?
Improve design
@sam Would love to have your input on this 