Application not exiting after terminating Actor and ActorSystem

Question

I have a main program that creates an ActorSystem, an Actor and sends some messages to the actor. When the messages are processed I sent PoisonPill to kill the Actor. Then i shutdown the actor system.

Within the Actor I am calling Await to await for a future to complete. The problem I am facing is that the application is not exiting even though the actor is terminated by PoisonPill and ActorSystem is shutdown.

def main(args: Array[String]): Unit = {

    val actorSystem = ActorSystem("sytem")
    val creationActor = actorSystem.actorOf(Props[MyActor], "MyActor")
    ...
    creationActor ! Message    //may be called multiple times
    creationActor ! PoisonPill
    ...
}

And the Actor code is

class MyActor extends Actor {

  override def receive: Receive = {
    case Message => {
       ...
      Await.result(Dataset.create(datasetId), 30 seconds) 
      //Dataset.create returns a Future. Also this method uses an
      //ExecutionContext of its own.
      ...
    }
  }

  override def postStop() = {
    context.system.shutdown()
  }
}

If I comment out the Await.result part, the program exits.

EDIT:
Looks like I have found the root cause.

The ExecutionContext that is used in Dataset.create(…) is the culprit. When i use a synchronous version of Dataset.create(…) which doesn't use Futures, my application exits.

The ec which Dataset.create() uses is defined like this
implicit val defaultContext = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(100))

Still curious to know why the asynchronous version doesn't exit the application.

EDIT 2: If I change the Await part to

val future = BQDataset.create(datasetId)
future onComplete {
  case Success(d) => ...
  case Failure(e) => ...
}

still I face the same problem. As @cem-catikkas mentioned, the ExecutionContext which gets created when I call BQDataset.create is hanging around. I verified this by using jstack and could see "pool-1-thread-1" prio=5 tid=0x00007ff49aa1e800 nid=0x4e03 waiting

Answer 1

When I call Dataset.create(…), the method creates an ExecutionContext and uses it to execute a third party library. This third party library opens a network connection and it doesn't have proper shutdown() or close() method to cleanup the underlying connection.

Because of this, even if I call ExecutionContext.shutdown(), it wasn't able to shutdown the thread pool. That's why the application was not exiting.

Answer 2

The actor is single threaded and handles messages one at a time. Await is a blocking operation. It sounds like the actor is busy doing CostlyOp. The poison pill won't interrupt that, so the actor doesn't shut down until it processes all the costly ops in front of the poison pill.