Why is memory profiling in ruby so hard?

Question

Or rather, why aren't there better tools for profiling memory in ruby, specifically rails apps?

Recently our rails app (hosted on heroku) has started seeing lots of R14 errors in the worker dynos. This means we're running out of memory. Bumping the dynos to 2x (512mb -> 1GB) only alleviated the problem temporarily, leading me to believe there is a memory leak somewhere. Naturally, my next step was to find a good profiling gem that can help me discover the source of the leak.

Maybe I'm just ignorant of the tools available, or maybe I just don't know how to use the ones I have. My wish is that I could install a gem and then run reports on the memory usage statistics. Hitting an endpoint to get a report is not really viable as my memory issues are isolated to worker dynos running delayed jobs.

I've looked at memprof, but it's 1.8 only.

I've looked at ruby-prof (awesome), but the memory profiling requires a patched ruby intepreter.

I've looked at GC::Profiler, but I don't understand how to find memory leaks with it.

So, is it just plain difficult to find memory leaks in ruby? Or am I missing the point somehow?

Answer 1

Depending on your "type" of leak, You can run valgrind against ruby. Might require recompilation again though. In general it's hard because ruby does method allocation without firing any events, by default, so it's tricky to track. See also perftools.rb project, which somewhat works around this limitation.