Unicorn restarting on its own - No memory - gets killed

Question

I am running two Rails apps on DigitalOcean with 512MB RAM and with 4 nginx processes.

The rails apps use Unicorn.

One has 2 workers and the other uses 1.

My problem is with the second app that has 1 Unicorn worker (same problem was there when there were 2 workers as well). What happens is, suddenly my app throws a 500 error. When I SSH into the server I would find that the app's unicorn process is not running!

When I start unicorn again everything would be fine.

This is my log file. As you can see, the worker gets reaped and then it is not able to fork it and the reason given is No Memory.

, [2014-01-24T04:12:28.080716 #8820]  INFO -- : master process ready
I, [2014-01-24T04:12:28.110834 #8824]  INFO -- : worker=0 ready

E, [2014-01-24T06:45:08.423082 #8820] ERROR -- : reaped #<Process::Status: pid 8824 SIGKILL (signal 9)> worker=0
E, [2014-01-24T06:45:08.438352 #8820] ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM)
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:523:in `fork'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:523:in `spawn_missing_workers'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:538:in `maintain_worker_count'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:303:in `join'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/bin/unicorn:126:in `<top (required)>'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'

I, [2014-01-24T08:43:53.693228 #26868]  INFO -- : Refreshing Gem list
I, [2014-01-24T08:43:56.283950 #26868]  INFO -- : unlinking existing socket=/tmp/unicorn.hmd.sock
I, [2014-01-24T08:43:56.284840 #26868]  INFO -- : listening on addr=/tmp/unicorn.hmd.sock fd=11
I, [2014-01-24T08:43:56.320075 #26868]  INFO -- : master process ready
I, [2014-01-24T08:43:56.348648 #26872]  INFO -- : worker=0 ready

E, [2014-01-24T09:10:07.251846 #26868] ERROR -- : reaped #<Process::Status: pid 26872 SIGKILL (signal 9)> worker=0
I, [2014-01-24T09:10:07.300339 #27743]  INFO -- : worker=0 ready
I, [2014-01-24T09:18:09.992675 #28039]  INFO -- : executing ["/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn", "-D", "-c", "/home/vocp/projects/hmd/config/unicorn.rb", "-E", "production", {11=>#<Kgio::UNIXServer:/tmp/unicorn.hmd.sock>}] (in /home/vocp/projects/hmd)
I, [2014-01-24T09:18:10.426852 #28039]  INFO -- : inherited addr=/tmp/unicorn.hmd.sock fd=11
I, [2014-01-24T09:18:10.427090 #28039]  INFO -- : Refreshing Gem list
E, [2014-01-24T09:18:13.456986 #28039] ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM)
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:523:in `fork'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:523:in `spawn_missing_workers'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/lib/unicorn/http_server.rb:153:in `start'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/gems/unicorn-4.7.0/bin/unicorn:126:in `<top (required)>'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `load'
/home/vocp/projects/hmd/vendor/bundle/ruby/2.0.0/bin/unicorn:23:in `<main>'
E, [2014-01-24T09:18:13.464982 #26868] ERROR -- : reaped #<Process::Status: pid 28039 exit 1> exec()-ed

This is my unicorn.rb

root = "/home/vocp/projects/hmd"
working_directory root
pid "#{root}/tmp/pids/unicorn.pid"
stderr_path "#{root}/log/unicorn.log"
stdout_path "#{root}/log/unicorn.log"

listen "/tmp/unicorn.hmd.sock"
worker_processes 1
timeout 30
preload_app true

# Force the bundler gemfile environment variable to
# reference the capistrano "current" symlink
before_exec do |_|
  ENV["BUNDLE_GEMFILE"] = File.join(root, 'Gemfile')
end


before_fork do |server, worker|
  defined?(ActiveRecord::Base) && ActiveRecord::Base.connection.disconnect!

  old_pid = Rails.root + '/tmp/pids/unicorn.pid.oldbin'
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      puts "Old master alerady dead"
    end
  end
end

after_fork do |server, worker|
  defined?(ActiveRecord::Base) && ActiveRecord::Base.establish_connection
  child_pid = server.config[:pid].sub('.pid', ".#{worker.nr}.pid")
  system("echo #{Process.pid} > #{child_pid}")
end

I donot have monit or god or any monitoring tools. I find it very odd because generally the used server memory would be 380/490. And nobody uses these two apps apart from me! They are in development.

Have I wrongly configured anything? Why is this happening? please help. Should I configure god to restart unicorn when it crashes?

Answer 1

For Unicorn memory usage the only way is up, unfortunately. Unicorn will allocate more memory if your rails app needs it. But it does not release it even if it doesn't need it anymore. For example if you load a lot of records for an index page at once, unicorn will increase memory usage. Now this is exacerbated by the fact that 512MB are not a huge amount of memory for 2 rails apps with 3 workers.

Furthermore there are memory leaks that increase memory usage too. See this article https://www.digitalocean.com/community/articles/how-to-optimize-unicorn-workers-in-a-ruby-on-rails-app

At the end of the article they refer to the unicorn-worker-killer gem in order to restart the unicorn workers based on either max connections or max memory which looks pretty straightforward.

Personally I have used the bluepill gem to monitor individual unicorn processes and restart them if needed.

In your case I would monitor all unicorn processes and restart them if they reach a certain memory size.