Ruby Memory leak (MRI)

Question

I must be missing something but every single application I write in Ruby seems like leaking some memory. I use Ruby MRI 2.3 but I see the same behaviour with other versions.

Whenever I write a test application that does something inside a loop it is slowly leaking memory.

while true
   #do something
   sleep 0.1
end

For instance, I can write to array and then clean it in a loop, or just send http post request.

Here is just one example, but I have many examples like this:

require 'net/http'
require 'json'
require 'openssl'

class Tester

    def send_http some_json
        begin
            @uri = URI('SERVER_URL')
            @http = Net::HTTP.new(@uri.host, @uri.port)
            @http.use_ssl = true
            @http.keep_alive_timeout = 10
            @http.verify_mode = OpenSSL::SSL::VERIFY_NONE
            @http.read_timeout = 30
            @req = Net::HTTP::Post.new(@uri.path, 'Content-Type' => 'application/json')
            @req.body = some_json.to_json
            res = @http.request(@req)
        rescue Exception => e  
                puts e.message  
                puts e.backtrace.inspect  
        end
    end

    def run
        while true
            some_json = {"name": "My name"}
            send_http(some_json)
            sleep 0.1
        end
    end
end


Tester.new.run

The leak I see is very small, it can be 0.5 mb every hour.

I ran the code with MemoryProfiler and with GC::Profiler.enable and it shows that I have no leaks. So it must be 2 options:

1. There is a memory leak in C code. This might be possible but I don't use any external gems so I find it hard to believe that Ruby is leaking.

2. There is no memory leak and this is some sort of Ruby memory management mechanism. The thing is that I can defiantly see the memory growing. Until when will it grow? How much do I need to wait to know if it is a leak or now?

The same code runs perfectly fine with JRuby without any leaks.

I was amazed reading a post:

stack overlflow from Joe Edgar:

Ruby’s history is mostly as a command line tool for text processing and therefore it values quick startup and a small memory footprint. It was not designed for long-running daemon/server processes

If what is written there is true and Ruby doesn't release memory back to OS then... We will always have a leak, right?

For instance:

1. Ruby asks for memory from OS.
2. OS provides the memory to Ruby.
3. Ruby frees the memory but GC still didn't run.
4. Ruby asks for more memory from OS.
5. OS provide more memory to Ruby.
6. Ruby runs GC but it is too late as Ruby already asked twice.
7. And so on and on.

What am I missing here?

Answer 1

Look Into GC Compaction and (Un)frozen String Literals

"Identical" Strings Aren't Necessarily Identical

Prior to Ruby 2.7.0, mainline Ruby didn't have compacting garbage collection. While I don't fully understand all the internals, the gist is that certain objects couldn't be garbage collected. Since you're using Ruby 2.3, that's something to keep in mind as you work on your memory allocation issues. Other non-YARV VMs may handle their internals differently, which is why you might see variation when using alternative engines like JRuby.

Even with Ruby 3.0.0-preview2, String literals aren't frozen by default, so your current implementation is creating a new String object with a unique object ID every tenth of a second. Consider the following:

3.times.map { 'foo'.__id__ }
#=> [240, 260, 280]

Even though the String objects seem identical, Ruby is actually allocating each one as a unique object in memory. Because a loop iteration is not a scope gate, those String objects can't be collected or compacted by YARV.

Enable Frozen String Literals by Default

You may have other issues as well, but it seems likely that your largest issue is keeping all of those String literals in scope indefinitely within your endless while-loop. You may be able to resolve your garbage collection problem (it's not a memory leak) by using frozen String literals instead. Consider the following:

# run irb with universally-frozen string literals
RUBYOPT="--enable-frozen-string-literal" irb

3.times.map { 'foo'.__id__ }
#=> [240, 240, 240]

You can solve this within your code in other ways as well, but reducing the number of String literals that remain in scope seems like a very sensible place to start.