Heroku - why would I get Error R12 (Exit timeout) when pushing a release to Heroku?

Question

Occasionally, whenever I push a release to Heroku shortly after I would receive the following error (I'm running 2 512MB dynos):

2014-11-21 00:38:30.216
188 <45>1 2014-11-21T00:38:29.163459+00:00 heroku web.2 - - Error R12 (Exit timeout) -> At least one process failed to exit within 10 seconds of SIGTERM

I'm using the unicorn app server, unfortunately only 1 per unicorn worker per 512MB dyno (since at its peak, my app RSS is 320MB - yea, go figure, some bloat is happening). Not sure if this helps, but I'm on Cedar14 with Preboot enabled. UNICORN_WORKERS is set to 1.

Here's my unicorn setup. Should I be concerned with this error?

And while we are on this topic, is db pool size 15 too large for my 2 dynos (I'm using Postgres standard which allows up to 120 concurrent connections).

worker_processes Integer(ENV['UNICORN_WORKERS'] || 2)

timeout Integer(ENV['UNICORN_TIMEOUT'] || 25)

preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  # other settings
  if defined?(ActiveRecord::Base)
    config = ActiveRecord::Base.configurations[Rails.env] || Rails.application.config.database_configuration[Rails.env]
    config['reaping_frequency'] = Integer(ENV['DB_REAPING_FREQUENCY'] || 10)
    config['pool'] = ENV['DB_POOL'] || 15
    ActiveRecord::Base.establish_connection(config)
  end

end

Answer 1

Heroku has a rule when deploying that basically says this:

• When your dyno is going to restart, Heroku will ask your processes nicely to shut themselves down. This gives them a chance to do nice stuff like close open db connections, etc.
• If your process doesn't close itself within 10 seconds, you'll get the error above, and Heroku will forcefully kill your process to restart it.

This is done to ensure you don't have an enormous bill running because one of your processes somehow never exited.

What's happening in your case (I'm speculating here), is that you've got a lot of open DB connections, and it's taking more than 10 seconds to close them because either:

• You have some DB latency.
• Your DB is burdened with other stuff.
• Your application simply can't close that many in < 10 seconds.

Overall, it's not a big deal. This problem sorts itself out over time, so I wouldn't worry about it.