How to kill this immortal nginx worker?

Question

I have started nginx and when I stop like root

/etc/init.d/nginx stop

after that I type

ps aux | grep nginx

and get response like tcp LISTEN 2124 nginx WORKER

kill -9 2124  # tried with kill -QUIT 2124, kill -KILL 2124

and after I type again

ps aux | grep nginx

and get response like tcp LISTEN 2125 nginx WORKER and so on.

How to kill this immortal Chuck Norris worker ?

Answer 1

After kill -9 there's nothing more to do to the process - it's dead (or doomed to die). The reason it sticks around is because either (a) it's parent process hasn't waited for it yet, so the kernel holds the process table entry to keep it's status until the parent does so, or (b) the process is stuck on a system call into the kernel that is not finishing (which usually means a buggy driver and/or hardware).

If the first case, getting the parent to wait for the child, or terminating the parent, should work. Most programs don't have a clear way to make them "wait for a child", so that may not be an option.

In the second case, the most likely solution is to reboot. There may be tools that could clear such a condition, but that's not common. Depending on just what that kernel processing is doing, it may be possible to get it to unblock by other means - but that requires knowledge of that processing. For example, if the process is blocked on a kernel lock that some other process is somehow holding indefinitely, terminating that other process could aleviate the problem.

Note that the ps command can distinguish these two states as well. These show up in the 'Z' state. See the ps man page for more info: http://linux.die.net/man/1/ps. They may also show up with the text "defunct".