why parallel execution on java compile take linear growth in time

Question

time javac Main.java                                      --> 0m1.050s time javac Main.java & javac Main.java                    --> 0m1.808s time javac Main.java & javac Main.java & javac Main.java  --> 0m2.690s time javac Main.java & ... 8 time                         --> 0m8.309s 

When we run javac command in parallel and with each increase in javac command ~1 sec gets added for all the javac command to complete.

Why is there a linear growth is time ?

Is all javac process while running involved in some kind on locks, if yes how to overcome it so as not to have a linear growth in time

---

PS: I have tried above on single core machine, double core machine, 4 core machine all showed same behaviour.

PS2: environment RedHat7, javac 1.7.0_79

Answer 1

The java compiler already handles dividing its work across available processors, even when only compiling a single file. Therefore running separate compiler instances in parallel yourself won't yield the performance gains you are expecting.

To demonstrate this, I generated a large (1 million lines, 10,000 methods) java program in a single file called Main1.java. Then made additional copies as Main2.java through Main8.java. Compile times are as follows:

Single file compile:

time javac Main1.java &    --> (real) 11.6 sec 

Watching this single file compile in top revealed processor usage mostly in the 200-400% range (indicating multiple CPU usage, 100% per CPU), with occasional spikes in the 700% range (the max on this machine is 800% since there are 8 processors).

Next, two files simultaneously:

time javac Main1.java &    --> (real) 14.5 sec time javac Main2.java &    --> (real) 14.8 sec 

So it only took 14.8 seconds to compile two, when it took 11.6 seconds to compile one. That's definitely non-linear. It was clear by looking at top while these were running that again each java compiler was only taking advantage of at most four CPUs at once (with occasional spikes higher). Because of this, the two compilers ran across eight CPUs mostly in parallel with each other.

Next, four files simultaneously:

time javac Main1.java &    --> (real) 24.2 sec time javac Main2.java &    --> (real) 24.6 sec time javac Main3.java &    --> (real) 25.0 sec time javac Main4.java &    --> (real) 25.0 sec 

Okay, here we've hit the wall. We can no longer out-parallelize the compiler. Four files took 25 seconds when two took 14.8. There's a little optimization there but it's mostly a linear time increase.

Finally, eight simultaneously:

time javac Main1.java &    --> (real) 51.9 sec time javac Main2.java &    --> (real) 52.3 sec time javac Main3.java &    --> (real) 52.5 sec time javac Main4.java &    --> (real) 53.0 sec time javac Main5.java &    --> (real) 53.4 sec time javac Main6.java &    --> (real) 53.5 sec time javac Main7.java &    --> (real) 53.6 sec time javac Main8.java &    --> (real) 54.6 sec 

This was actually a little worse than linear, as eight took 54.6 seconds while four only took 25.0.

So I think the takeaway from all this is to have faith that the compiler will do a decent job trying to optimize the work you give it across the available CPU resources, and that trying to add additional parallelization by hand will have limited (if any) benefit.

Edit:

For reference, there are two entries I found in Oracle's bug database regarding enhancing javac to take advantage of multiple processors:

• Bug ID: JDK-6629150 -- The original complaint, this was eventually marked as a duplicate of:
• Bug ID: JDK-6713663 -- Suggests the resolution, and based on the "Resolved Date" it appears that multi-processor support in javac was added on 2008-06-12.