Need help with performance of Java's ProcessBuilder under Solaris

Question

My question is, do JVM's share some kind of resource related to threading or processes that could cause ProcessBuilder performance to spike after a month or more of normal usage? Using java 6 update 21 for all apps.

Over the past several months, we've noticed that a single server in our data center (Sparc M4000 running Solaris 10) can go about 6-8 weeks with no problems. Quickly, however, performance on an application that utilizes the ProcessBuilder class to run scripts takes a huge performance hit - with ProcessBuilder.start taking over a minute to return sometimes. After a reboot, and for several weeks after, normal return time is in the 10s or maybe 100 millisecond range.

I wrote a separate small application that creates 5 threads, and each thread runs the 'ls' command using ProcessBuilder 10 times serially, then I gather stats from that in order to monitor the original problem. This application exits after each run, and is run from cron only once an hour. It usually only takes a second or two.

Last night, ProcessBuilder times spiked again to over a minute for each ProcessBuilder.start call, after 45 days of uptime and normal behavior.

top shows no memory or CPU hogs. I did try to do a jstack on the test app, but got the error 'Can't create thread_db agent'.

Any ideas?

Answer 1

We had a similar problem on our application which runs in Linux. The Linux JVM code uses a fork which means the address space gets mapped and copied each time you exec. We were executing many small short lived processes. It appears a main difference from your app is that we had a relatively large heap (around 240GB) so I'm sure that had an impact. We ended up implementing our own spawning code using JNI and posix spawn. Here is a link to the question/answer: Slowing process creation under java