What can I do about unicorn::clientshutdown errors on my heroku rails app?

Question

I have an app that accepts image uploads. It's a rails 3 app on heroku using Unicorn. I'm occasionally getting unicorn::clientshutdown exceptions, and I don't really know what causes them or how to handle them. What should I do?

This is my unicorn.rb file:

before_fork do |server, worker|
  # Replace with MongoDB or whatever
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
    Rails.logger.info('Disconnected from ActiveRecord')
  end

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis.quit
    Rails.logger.info('Disconnected from Redis')
  end
end

after_fork do |server, worker|
  # Replace with MongoDB or whatever
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
    Rails.logger.info('Connected to ActiveRecord')
  end

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis = ENV['REDIS_URI']
    Rails.logger.info('Connected to Redis')
  end
end

Answer 1

Image uploads, Heroku, and Unicorn, oh my.

Problem

This is the trifecta for this error. There is probably a correlating H12 error (https://devcenter.heroku.com/articles/error-codes#h12-request-timeout) in your Heroku logs. What is happening is that the request is taking too long to complete (Heroku has an inescapable 30 second timeout), so it disconnected and that unicorn worker was killed. Also, Unicorn isn't great with slow/long-running requests (see http://rainbows.rubyforge.org)

Solution

The trick is to upload the image on the front-end without hitting the server (CORS/AJAX/jquery.fileupload.js/etc), passing that uploaded file location along with the form submission, then performing any processing later as a background job and reuploading, which isn't subject to the 30 second timeout limit. Others have written more extensively about this. Also, you could use a service like Cloudinary to do this for you.

PS

YMMV, but you should add this to your unicorn config as well( https://devcenter.heroku.com/articles/rails-unicorn)

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end
  # ...
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to sent QUIT'
  end
  # ...
end