Use only a subset of threads in an ExecutorService

Question

In a typical JAVA application, one configures a global ExecutorService for managing a global thread pool. Lets say I configure a fixed thread pool of 100 threads:

ExecutorService threadPool = Executors.newFixedThreadPool(100);

Now lets say that I have a list of 1000 files to upload to a server, and for each upload I create a callable that will handle the upload of this one file.

List<Callable> uploadTasks = new ArrayList<Callable>();
// Fill the list with 1000 upload tasks

How can I limit the max number of concurrent uploads to, lets say, 5?

if I do

threadPool.invokeAll(uploadTasks);

I dont have control on how many threads my 1000 tasks will take. Potentially 100 uploads will run in parallel, but I only want max 5. I would like to create some sort of sub-executor, wich uses a subset of the threads of the parent executor. I dont want to create a new separated executorService just for upload, because i want to manage my thread pool globally.

Does any of you know how do to that or if an existing implementation exists? Ideally something like

ExecutorService uploadThreadPool = Executors.createSubExecutor(threadPool,5);

Many thanks,

Antoine

Answer 1

In a typical JAVA application, one configures a global ExecutorService for managing a global thread pool.

I'm not doing this, but maybe i'm atypical. :-) Back to your question:

As having a guard (possibly using a Semaphore inside your Callables will only clutter your global Executor which tasks waiting for each other, you'll have to either

• use some external logic which ensures only 5 jobs are running at any time, which could in itself be a Callable submitted to your one Executor. This could be done by wrapping your download jobs with some logic which will pull the next job from a queue (containing the URLs or whatever) once one job is completed. Then you submit five of those "drain download queue" jobs to your Executor and call it done. But atypical as I am, I'd just
• use a separate Executor for the downloads. This also gives you the ability to name your threads appropriately (in the Executors ThreadFactory), which might help with debugging and makes nice thread dumps.