Unicorn completely ignores USR2 signal

Question

I'm experiencing a rather strange problem with unicorn on my production server. Although the config file states preload_app true, sending USR2 to the master process does not generate any response, and it seems like unicorn is ignoring the signal altogether. On another server sending USR2 changes the master process to and (old) state and starts a new master process successfully. The problematic server is using RVM & bundler, so I'm assuming it's somehow related (the other one is vanilla ruby). Sending signals other than USR2 (QUIT, HUP) works just fine. Is there a way to trace what's going on behind the scenes here? Unicorn's log file is completely empty.

Answer 1

I suspect your issue might be that your Gemfile has changed, but you haven't started your unicorn in a way that allows USR2 to use the new Gemfile. It's therefore crashing when you try to restart the app.

Check your /log/unicorn.log for details of what might be failing.

If you're using Capistrano, specify the BUNDLE_GEMFILE as the symlink, e.g.:

run "cd #{current_path} && BUNDLE_GEMFILE=#{current_path}/Gemfile bundle exec unicorn -c #{config_path} -E #{unicorn_env} -D"

Here's a PR that demostrates this.

Answer 2

I experienced a similar problem, but my logs clearly identified the issue: sending USR2 would initially work on deployments, but as deployments got cleaned up, the release that the Unicorn master was initially started on would get deleted, so attempts at sending a USR2 signal would appear to do nothing / fail, with the error log stating:

forked child re-executing... 53 /var/www/application/releases/153565b36021c0b8c9cbab1cc373a9c5199073db/vendor/bundle/ruby/1.9.1/gems/unicorn-4.3.1/lib/unicorn/http_server.rb:439:in `exec': No such file or directory - /var/www/application/releases/153565b36021c0b8c9cbab1cc373a9c5199073db/vendor/bundle/ruby/1.9.1/bin/unicorn (Errno::ENOENT)

The Unicorn documents mention this potential problem at http://unicorn.bogomips.org/Sandbox.html: "cleaning up old revisions will cause revision-specific installations of unicorn to go missing and upgrades to fail", which in my case meant USR2 appeared to 'do nothing'.

I'm using Chef's application recipe to deploy applications, which creates a symlinked vendor_bundle directory that is shared across deployments, but calling bundle exec unicorn still resulted in the original Unicorn master holding a path reference that included a specific release directory.

To fix it I had to call bundle exec /var/www/application/shared/vendor_bundle/ruby/1.9.1/bin/unicorn to ensure the Unicorn master had a path to a binary that would be valid from one deployment to the next. Once that was done I could deploy to me heart's content, and kill -USR2 PID would work as advertised.

The Unicorn docs mention you can manually change the binary path reference by setting the following in the Unicorn config file and sending HUP to reload Unicorn before sending a USR2 to fork a new master: Unicorn::HttpServer::START_CTX[0] = "/some/path/to/bin/unicorn"

Perhaps this is useful to some people in similar situations, but I didn't implement this as it appears specifying an absolute path to the shared unicorn binary was enough.