Is ruby fast yet?


#1

Someone recently posted a link to this site: http://www.isrubyfastyet.com/ when discussing ruby 2.1 performance. Very interesting, it’s actually open source here: https://github.com/brianhempel/isrubyfastyet

I do like their graphs.


#2

I first noticed this project yesterday, if I would have known about it earlier I would have contacted Brian. (which I will now)

I think it is very important to highlight where this project differs from is ruby fast yet.

Our initial goals are:

  1. Focus on MRI (back-end is to eventually support JRuby and Rubinius etc)

  2. Figure out which builds of MRI 2.1 regressed performance and which improved performance. To answer this questions we should:
    a. Graph every benchmark in the ruby/benchmarks folder, for every commit
    b. Graph the Discourse benchmark, for every commit

  3. Figure out which builds of Rails 4.1 regressed performance and which improved perf.

  4. Run all the tests on realistic production hardware and software stack, x64 Linux / gcc compiled. Support multiple backends to run the tests on.
    a. Ensure power management etc does not cause variance in tests. Run repeated runs, disable power management.

  5. Maintain the ability to backfill benchmarks as required, all ruby builds to live in separate easily addressable folders.

  6. Look at comparing performance of tuned stacks of Ruby as well (eg malloc limit raising, out of band gc for web apps etc)

  7. Build an extensible framework others can easily contribute to.

I do like seeing graphs, however I would like us to focus on a front end that is more similar to pypy speed center where you can compare any two things.

I am much more leaning towards using rbenv than rvm as it takes care of (5) cleanly.

cc @andypike @benweint


#3

IRFY author here. Let me know any questions you have.

Builds are too long for me to benchmark every commit with the hardware I have, so I just went with benching every day. That at least narrows things down a lot. With more hardware, more could be done. You really want dedicated hardware for benchmarks, however.

I went with RVM for a couple reasons. The key reason at the time was that it had better support for more Rubies. I don’t know if the rbenv ecosystem has since caught up.

I have learned that builds can break for a million reasons. External libraries. RVM bugs. Implementation bugs in my build environment. Implementation bugs that aren’t caught by test suites (found an RBX JIT bug once). Gems changing their interfaces and requirements. Continuous, full-stack, ecosystem testing has many points of failure.


#4

Great to hear from you here.

I tried out rbenv this week and have decided to say goodbye to it. The rbenv rehash business is just a time killer for this kind of stuff, just logging on to my machine took 10 seconds due to rbenv business.

Instead I am going with chruby which is lightning fast and supports enough rubies to keep me happy.

I am also keeping all the builds around, I seem to have enough storage for it. When benching Discourse I am able to bundle --standalone which saves a lot on storage (although I do need to do some magic to allow for rubygems changes)

My current bench setup is using an oldish desktop I have, completely dedicated. Its an i7-960 with an ssd, slower than my current desktop but perfectly decent.

I had to spend a fair amount of time with cpufrequtil to get the clock to run at a consistent rate without it benching is rather inconsistent.


#5

#6

@mariovisic @brianhempel

Moving all rubybench discussion to the new home at: http://community.rubybench.org/