Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Java library to maintain a process pool

I'm working on a Java daemon that requires concurrency: an infinite loop that listens on a job queue (redis) and distributes each job to a worker. The worker doesn't necessarily have to return a value.

I found Executors pretty useful and I am using a ThreadPoolExecutor to maintain a number of worker threads.

Though, those workers run 3rd party code which needs to run as isolated as possible, avoiding sharing static properties.

My question: is there any Java library/framework that provides features similar to Executors, like:

  • worker pools
  • automatically adjusted pool size

..but spawning processes instead of threads?

like image 907
dimi Avatar asked Apr 24 '12 19:04

dimi


3 Answers

I understand that you have 3rd party libraries that you want to run isolated somehow -- so they do not have access to static variables for example.

I would run your tasks in another ClassLoader. There are web frameworks that use this mechanism to isolate web requests. Passing data between the classloaders is a bit tricky. I'm not sure how Jetty does it. Maybe using some sort of socket? Here's an interesting article about using System.out to share objects. Certainly a hack.


My question is: is there any Java library/framework that provides features similar to Executors... but spawning processes instead of threads?

Modern operating systems will utilize modern multi-processor architectures to get the most out of Java threads. Spawning another thread inside of your current process will run on another processor if available and will be much more efficient that running that task in a completely separate process, especially something as heavy as another JVM process.

like image 79
Gray Avatar answered Nov 12 '22 08:11

Gray


There is project in github to create process pool. Have a look. https://github.com/shri30/ProcessPool

Here is an example:

Sample code:

public class UsageExample implements JTask{


    public static void main(String args[]) throws Exception{
        //Create a pool with name : UsageExample-Process and number of processes : 3
        ExecutorService pool = ProcessPool.createProcessPool("UsageExample-Process", 3);
        ProcessFuture<TaskStatus> pf1 = pool.submit(Task.class, args);
        ProcessFuture<TaskStatus> pf2 =pool.submit(Task.class, args);
        ProcessFuture<TaskStatus> pf3 =pool.submit(Task.class, args);
        ProcessFuture<TaskStatus> pf4 =pool.submit(Task.class, args);
        ProcessFuture<TaskStatus> pf5 =pool.submit(Task.class, args);
        //the get method blocks the call to get the result
        TaskStatus ts= pf1.get();
        Status status = ts.getStatus();
        if(status == Status.SUCESS){
        //code goes here
        }

        TaskStatus ts2= pf2.get();
        Status status2 = ts.getStatus();
        if(status == Status.SUCESS){
        //code goes here
        }


        TaskStatus ts3= pf3.get();
        Status status3 = ts.getStatus();
        if(status == Status.SUCESS){
        //code goes here
        }


        TaskStatus ts4= pf4.get();
        Status status4 = ts.getStatus();
        if(status == Status.SUCESS){
        //code goes here
        }
        //terminate the pool - pass true for graceful termination, the second parameter is minutes
        //wait for task completion
        pool.terminate(true, 3000);
    }


}

class Task implements JTask{

    public static void main(String[] args) {
        System.out.println("Executing the sample task");
    }

}
like image 35
SVN Avatar answered Nov 12 '22 07:11

SVN


Have you considered something like using the ProcessBuilder to manage your isolated process?

String myJar = MyClass.class.getProtectionDomain().getCodeSource().getLocation().getPath());
ProcessBuilder pb = new ProcessBuilder("java", "-classpath", myJar, "package.MainClass");
Process p = pb.start();

I don't know if it's a good idea to submit Runnable tasks that manage sub-processes to an Executor, but it may be a solution to isolate your workers.

like image 21
Bertil Chapuis Avatar answered Nov 12 '22 07:11

Bertil Chapuis