I have a big file up to terabytes, and my task is to process line by line. Each line should take 5s to accomplish. To improve the performance I dispatch the process to a fixed thread pool like this
ExecutorService executor = Executors.newFixedThreadPool(5);
while ((line = br.readLine()) != null) {
Runnable worker = new WorkerThread(line);
executor.execute(worker);
}
My question is what happens if I overwhelm the executor's queue by putting so many tasks. Does it throw StackOverflow
?
By default, the MaxThreads of the ThreadPool is very high. Usually you'll never get there, your app will crash first. So when all threads are busy the new tasks are queued and slowly, at most 1 per 500 ms, the TP will allocate new threads.
Once 'max' number of threads are reached, no more will be created, and new tasks will be queued until a thread is available to run them.
ExecutorService must be shutdown explicitly to reclaim the resources (CPU & Memory) occupied by threads which have already finished their job but still exist.
It will throw an OOM error (gc overhead) if you do not process faster than you put. You cannot get a StackOverflow
as the stack doesn't change much.
@StinePike asks a good question. RejectedExecutionException
happens if there are no more threads to process items off the queue and the queue is full. In this case default implementation of Executors.newFixedThreadPool(5);
will use an unbounded LinkedBlockingQueue
. Your our only restriction is memory.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With