Delayed jobs restarting + Capistrano best practices

Question

I just wanted to understand best practices for restarting delayed jobs workers using capistrano. I've got a bunch of workers that process long jobs (up to 10minutes).

I've come across two scenarios during a deploy while the workers are processing- 

1)

I stop delayed jobs workers before deploy: restart task and start them again after the deploy:restart task.

But in this scenario, it won't restart my app until the delayed jobs are finished (this may be ok - but the cap deploy script literally sits there until the job is done and it can stop all workers) before proceeding with the app restart task.

2)  I also tried stopping/starting the delayedjobs workers after the restart task - but this caused all sorts of dramas whereby the tasks would be halted without waiting yet the delayedjobs table had them listed and assigned to a worker with a PID that doesn't exist!

Any other options? Or am I forced to wait as mentioned in scenario 1.

Many thanks.

Edit: I just realised with scenario 1 .. it doesn't wait! The stop task will forcefully kill my worker even though it hasn't finished!

 ** [out] delayed_job: trying to stop process with pid 9630...
 ** [out] delayed_job: process with pid 9630 won't stop, we forcefully kill it...
 ** [out] 
 ** [out] delayed_job: process with pid 9630 successfully stopped.

Answer 1

If your deploy changes change the database schema, then you're forced to wait.

If it doesn't, you can set a flag that the workers check at the end of every job. Upon deploy, you set the flag, and once a long-running worker finishes, it replaces itself by exec-ing a new worker. That way you're never really starting and stopping workers - they're just always running, and will load the latest code on starting the next job.