Heroku delayed_job workers killed during deployment

Question

On Heroku, I use delayed_job to run asynchronous tasks. All is well until I do a git push heroku master and then the Heroku environment kills any worker threads that are in-process.

The issue here is that those jobs never get re-queued since the delayed_job table in my db shows them as still locked and running, even though the workers that used to be servicing them are long dead.

How do I prevent this situation from occurring? I'd like Heroku to wait for all delayed jobs in progress to complete or error out before closing down, or at least terminate them and allow a new worker to be assigned to them once the server comes back up post-reboot from changes being applied by my update.

Answer 1

Looks like you can configure DJ to handle SIGTERM and mark the in-progress jobs as failed (so they'll be restarted again):

Use this setting to throw an exception on TERM signals by adding this in your initializer:

Delayed::Worker.raise_signal_exceptions = :term

More info in this answer: https://stackoverflow.com/a/16811844/1715829