I am loving this progress, I would like to start pushing for sponsors but think we need a few things first.
I went ahead and purchased
rubybench.org I think its a reasonable domain name for the effort, we also have a TON of Digital Ocean credit so I am more than happy to just indefinitely offer a 20$ a month slice for the front end.
I can ping some friends at Heroku who could also possibly sponsor the front end, just don't want to have something that will collapse as soon as this hits hacker news.
Clear contribution guidelines
The front page of rubybench.org can easily be something like https://railsbench.herokuapp.com/tgxworld/ruby?utf8=%E2%9C%93&result_types%5B%5D=discourse_ruby_trunk_categories&commit=Submit for now. However it needs to add
- A clear way for people to see where the GitHub repos involved are with a link to contribution guidelines
- A clear call out to sponsors (what do we need? who do you contact?)
- A clear call out to where stuff is discussed (the topic is fine for now, I can spin off a
discuss.rubybench.org in very short order if we wish.
A clear NO VMs used rule called out
We need to be super clear that all benchmarking runs on bare metal, not on various cloud hosts where CPU and IO is unclear and scheduled in weird ways. This rule needs to be called out in the footer.
A hook to get the discussion started
Even though we are very eager to get long term benchmarks going, I think a great first hook for now is a very simple one.
Run all the existing benchmarks (discourse + ruby) on 1.9.3 (pick 3 p releases) on 2.0.0 (pick 3 p releases) and 2.1.0 -> 2.1.5 and 2.2.0
The goal here would be to give a clean report card to where the various stable releases are performance wise, accounting for the 2 key areas (memory usage and wall/cpu times)
Additionally I would recommend including Github Ruby in the mix which contains the method cache fixes that give a huge perf boost https://github.com/github/ruby/commit/bd002fc9fc3c7236395df29955984b20188eed06
( https://github.com/github/ruby/commits/2.2 )