DelayedJob ERROR: there is already one or more instance(s) of the program running

Question

Some backstory:

My server ran out of disk space last night while delayed_job workers were running off jobs processing images.

When I try to stop the workers, I get the response "Terminated".

RAILS_ENV=production script/delayed_job stop
Terminated

Then I run the following to see that the workers were terminated.

ps -ef | grep delayed_job
servername      4474  4274  0 02:37 pts/1    00:00:00 grep --color=auto delayed_job

Now I try to start up new workers.

RAILS_ENV=production script/delayed_job -n2 start
ERROR: there is already one or more instance(s) of the program running
ERROR: there is already one or more instance(s) of the program running

This is the error that I'm getting, I'm not able to start any new workers even though it seems that I have stopped all active workers.

This was confusing me so I ran

RAILS_ENV=production script/delayed_job status
delayed_job: running [pid 0]
delayed_job: running [pid 0]

This seems wrong [pid 0], it seems like my workers have been corrupted and I'm hoping to figure out how to solve this issue. Thanks!

I'm running on Linux Ubuntu LTS 10.4.

EDIT: So I deleted the 2 worker files I found in tmp/pids which allowed me to start and stop workers. However, the workers will not run anything because I believe I need to have PID files generated in the tmp folder.

Answer 1

I ran this command to solve the issue:

RAILS_ENV=production script/delayed_job -n 2 --pid-dir=tmp/pids restart

By specifying the directory it was able to fix the issue of lack of PID file.

Answer 2

Put in some random pid (which is not a pid of an existing process) on that file, and then try to start your daemon. You will see a message similar to:

pid-file for killed process 32323123 found (/path/to/pid/file), deleting.

and your daemon will get started.