Maybe you could have an adaptive system so you wouldn’t have to re-run every benchmark: if you ran a benchmark and the datapoint was more than, say, 3% off from the mean of the last 7 runs, you’d run it more times to smooth it out.
I am loving this progress, I would like to start pushing for sponsors but think we need a few things first.
A domain
I went ahead and purchased rubybench.org I think its a reasonable domain name for the effort, we also have a TON of Digital Ocean credit so I am more than happy to just indefinitely offer a 20$ a month slice for the front end.
I can ping some friends at Heroku who could also possibly sponsor the front end, just don’t want to have something that will collapse as soon as this hits hacker news.
###Clear contribution guidelines
The front page of rubybench.org can easily be something like https://railsbench.herokuapp.com/tgxworld/ruby?utf8=✓&result_types[]=discourse_ruby_trunk_categories&commit=Submit for now. However it needs to add
- A clear way for people to see where the GitHub repos involved are with a link to contribution guidelines
- A clear call out to sponsors (what do we need? who do you contact?)
- A clear call out to where stuff is discussed (the topic is fine for now, I can spin off a
discuss.rubybench.orgin very short order if we wish.
###A clear NO VMs used rule called out
We need to be super clear that all benchmarking runs on bare metal, not on various cloud hosts where CPU and IO is unclear and scheduled in weird ways. This rule needs to be called out in the footer.
###A hook to get the discussion started
Even though we are very eager to get long term benchmarks going, I think a great first hook for now is a very simple one.
Run all the existing benchmarks (discourse + ruby) on 1.9.3 (pick 3 p releases) on 2.0.0 (pick 3 p releases) and 2.1.0 -> 2.1.5 and 2.2.0
The goal here would be to give a clean report card to where the various stable releases are performance wise, accounting for the 2 key areas (memory usage and wall/cpu times)
Additionally I would recommend including Github Ruby in the mix which contains the method cache fixes that give a huge perf boost https://github.com/github/ruby/commit/bd002fc9fc3c7236395df29955984b20188eed06
( https://github.com/github/ruby/commits/2.2 )
Thoughts?
@sam -thats what i am currently running localy on my PC, as i have no bare metal servers. I wrapped all tests by runner which also catches the memory usage at the end of each test.
Also i am working on time/memory graph for each version , hopefully i will be able to present it soon.
Here are current results of cpu time / memory usage. Tests were run only once, but still running at night for more data.
http://rubybench-rycco.rhcloud.com/benchmarks/binary-trees
Some little progress in cpu time, but we can see some memory issues in versions 2.1.0 - 2.1.2
Running for more data. Also as soon as I can i will add more tests.
Yup rubybench.org sounds good. Could you give me ownership for ruby-bench organization? I’ll fork the repos over.
If possible, I’ll prefer to stick with Heroku for now since I rather focus on the benchmarks than dealing with dev ops which I have little experience it.
I’m not really concerned about the Front End cost right now since I can still afford it. The bare metal server from SoftLayer ranges from $173 USD onwards per month and would be the main concern for me.
I’ll work on the above soon.
I’ll start implementing the above as well. Should take a few days or so.
Thanks for the feedback @sam
I’ve seen your updated site. Looks good to me. I would like to see it being consolidated into a single App though.
@tgxworld - well, i mentioned that i am benchmarking versions from the beginning. So i think it the best would be if you stick at per commit benchmarking and u can use my page as subdomain or whatever( we will figure out)
If you really want to have the same on your web, you can use my results (because you dont want to benchmark the same thing and spend few days on it). I have build an API (for example http://rubybench-rycco.rhcloud.com/api/v1/benchmarks). Docs will be available soon. 
happy benchmarking 
@sam I’ve added the necessary changes. Please see if you’re ok with it.
I’ve left 1.9.x out for the Discourse benchmarks since it isn’t supported on Discourse stable.
Feel free to let me know if there are any more things you need from me.
Well, i see u did not wait 
… I already updated API with DOC, so you could have used my results (if you are not interested in GCC version just use GCC 4.8 -O3). I have updated results with Ruby official benchmarks.
I think its useless to test the same thing twice. (Also i have available memory results )
Just feeling kind of confused how you travelled from rails bench to ruby bench
@richard_ludvigh I see it as two separate projects right now because I don’t see us unifying both projects anyway in the near future. With regards to the results, I can’t use yours as you’re running on a different builder which will not be consistent with the results I have for Ruby Benchmarks on a per-commit basis.
I don’t think we’re on the same page here. The benchmarks have to be run on the same builder in order for it to be consistent. I wouldn’t say we’re actually testing the same thing twice.
Well, i see what you mean. Then just to save your time, feel free to use my suite. I still had not time to write down doc, but i can do some quick introduction for you. It has rubies from 1.6.8 to 2.2.0. I am able to add a ENV switch for you to turn off GCC comparision.
How did u test the memory usage ? I could not find it in your repo
opps. I forgot to push to Github. Heres is an example: https://github.com/rails-bench/rails-bench-docker/blob/master/ruby_benchmarks/ruby_releases/ruby_memory_benchmarks/bm_so_nsieve_bits.rb.
Ah i see, i was wondering you benchmark only the execution memory.
Sorry for the delay, holidays and scrambling to get stuff finished at work for the last week until I take a few weeks off work.
I just added you as an owner of https://github.com/orgs/ruby-bench, please feel free to tidy up the repo and take it over. recommend getting rid of the whole rails-bench thing.
I also setup rubybench.org to go to https://railsbench.herokuapp.com/ which will go live once dns updates.
Well turned out, the GCC 4.9 O2 is not that much faster on baremetal.
This topic is now closed. New replies are no longer allowed.
Please continue the discussion on http://community.rubybench.org/