Shutting down a ThreadPoolExecutor when a worker thread has an Exception

Question

What is the best way for a worker thread to signal that a graceful shutdown should be initiated?

I have a fixed size thread pool which works through a continuous set of tasks, each lasting no more than a few seconds. During normal operation this works well and chugs along with its workload.

The problem I am having is when an exception is thrown in one of the threads. If this happens I would like to bring the whole thing down and have been unable to get this working correctly.

Current approach

The naive approach that I have been using is to have a static method in the "Supervisor" class which shuts down the thread pool using the standard shutdown() and awaitTermination() approach. This is then called by any of the "Worker" classes if they encounter a problem. This was done rather than propagating the exception because execute() requires a Runnable and the run() method cannot throw exceptions.

Here is some pseudo code:

// Finds work to do and passes them on to workers
class Supervisor {

  ThreadPoolExecutor exec;

  static main() {
    exec = new FixedThreadPool(...);
    forever {
      exec.execute(new Worker(next available task));
     }
  }

  static stopThreadPool() {
    exec.shutdown();
    if(!exec.awaitTermination(timeout_value)) {
      print "Timed out waiting on terminate"
    }
  }
}


class Worker {
  run() {
    try {
      // Work goes here
    } catch () {
      Supervisor.stopThreadPool()
    }
  }
}

The effect that I am seeing is that the threads do pause for a while but then I see the timeout message and they all resume their processing. This pattern continues until I manually shut it down. If I put a call to stopThreadPool() in the main method after having broken out of the loop, the shutdown happens correctly as expected.

The approach is clearly wrong because it doesn't work, but it also feels like the design is not right.

To reiterate the question: What is the best way for a worker thread to signal that a graceful shutdown should be initiated?

Additional information

The questions I have looked at on SO have been of two types:

• "How do I kill a thread in a thread pool?"
• "How do I know all my threads are finished?"

That's not what I'm after. They also seem to exclusively talk about a finite set of tasks whereas I am dealing with a continuous feed.

I have read about an alternative approach using exec.submit() and Futures which puts the onus on the supervisor class to check that everything's ok but I don't know enough about it to know if it's a better design. The exception case is, well ... exceptional and so I wouldn't want to add work/complexity to the normal case unnecessarily.

(Minor side note: This is a work project and there are other people involved. I'm saying "I" in the question for simplicity.)

Answer 1

You are not that far from the correct solution, the problem is that you need to handle the interruption caused by the shutdown call properly. So your thread's run method should look like this:

run () {
  try {
    while (Thread.interrupted() == false) {
      doSomeWork();
    }
  } catch (Exception e) { 
    myExecutor.shutdown();
  }
}

Note that I explicitly used the shutdown() without awaitTermination() because otherwise the waiting thread is the one that keeps the Executor from properly terminating, because one thread is still waiting. Perfect single-thread deadlock. ;)

The check for interruption is by the way the hint on how to kill a thread gracefully: get the run method to end by either setting a running boolean to false or by interrupting, the thread will die a moment later.

To check if all of your threads have terminated (= are just about to end their run method), you can use a CountDownLatch for a simple case or the CyclicBarrier/Phaser class for more complex cases.