Java program is getting slower after running for a while

Question

I have a java program that is a typical machine learning algorithm, updating the values for some parameters by some equations:

for (int iter=0; iter<1000; iter++) {
    // 1. Create many temporary variables and do some computations                         
    // 2. Update the value for the parameters                    
}

The computations of updating parameters are rather complex, and I have to create many temporary objects, but they are not referenced out of the loop. The code in the loop is CPU-intensive, and does not access disk. This program loads a relatively large training dataset, therefore, I granted 10G memory (-Xmx10G) to JVM, which is much larger than it requires (peak at ~6G by "top" command or window's task manager).

I tested it on several linux machines (centos 6, 24G memory) and a window machine (win7, 12G), both with SUN Hotspot JDK/JRE 1.8 installed. I did not specify other JVM parameters except -Xmx. Both machines are dedicated to my program.

On windows, my program runs well: each iteration uses very similar running time. However, the running time on all of the centos machines is weird. It initially runs properly, but slows down dramatically (~10 times slower) at 7th/8th iteration, and then keeps slow down ~10% in each iteration ever after.

I suspect it might be caused by Java's garbage collector. Therefore, I use jconsole to monitor my program. Minor GC happens very frequently on both machines , that is because the program creates many temporary variable in the loop. Furthermore, I used "jstat -gcutil $pid$ 1s" command and captured the statistics:

Centos: https://www.dropbox.com/s/ioz7ai6i1h57eoo/jstat.png?dl=0

Window: https://www.dropbox.com/s/3uxb7ltbx9kpm9l/jstat-winpng.png?dl=0

[Edited] However, the statistics on two kinds of machines differ a lot:

1. "S1” on windows jumps fast between 0 to 50, while stays at "0.00" on centos.
2. "E" on windows changes very rapidly from 0 to 100. As I print the stat for every second, the screenshot does not show its increment to 100. On centos, however, "E" increases rather slowly towards 100, and then reduces to 0, and increases again.

It seems the weird behaviour of my program is due to Java GC? I am new to Java performance monitor and do not have a good idea to optimize GC parameter setting. Do you have any suggestions? Thank you very much!

Answer 1

I'm sorry to post this as an answer but I don't have enough score to comment.

If you think it's a GC related issue I'd change it for the Garbage 1 Collector –XX:+UseG1GC

I found this brief explanation about it: http://blog.takipi.com/garbage-collectors-serial-vs-parallel-vs-cms-vs-the-g1-and-whats-new-in-java-8/

Can you run your software under profiling? Try to use the jprofiler, VisualVM or even the netbeans profiler. It may help you a lot.

I noticed that you have your own encapsulation of a vector and matrix. Maybe your are spending a lot more memory than necessary with that too. But I don't think that is the problem.

Sorry again about not contributing as a comment. (It would be more appropriate)

Answer 2

I would consider declaring the vars outside the loop so mem allocation is done once and eliminate GC completely.