Is there a way to speed up Javadoc (takes 7 minutes)

Question

I am building a Javadoc for a module with 2,509 classes. This currently takes 7 min or 6 files per second.

I have tried

mvn -T 1C install 

However javadoc only uses 1 CPU. Is there a way to use more and/or speed up?

I am using Oracle JDK 8 update 112. My dev machine has 16 cores and 128 GB of memory.

Running flight recorder I can see that there is only one thread main

For those who are interested, I've used the following options:

<plugin>     <artifactId>maven-javadoc-plugin</artifactId>     <configuration>         <additionalJOptions>             <additionalJOption>-J-XX:+UnlockCommercialFeatures</additionalJOption>             <additionalJOption>-J-XX:+FlightRecorder</additionalJOption>             <additionalJOption>-J-XX:StartFlightRecording=name=test,filename=/tmp/myrecording-50.jfr,dumponexit=true</additionalJOption>             <additionalJOption>-J-XX:FlightRecorderOptions=loglevel=debug</additionalJOption>         </additionalJOptions>     </configuration> </plugin> 

NOTE: One workaround is to do:

-Dmaven.javadoc.skip=true 

Answer 1

Running maven with -T1C will cause maven to try to build modules in parallel, so if you have a multi-module project, at best it will build each module's javadoc in parallel (if your dependency graph between modules allow it).

The javadoc process itself is single-threaded, so you won't be able to use multiple cores to generate the javadoc of one single module.

However, since you have many classes (and possibly many @link doclets or similar ?), maybe the javadoc process could benefit from extended heap. Have you looked into GC activity ? Try adding this in your configuration, see if it helps :

<additionalJOption>-J-Xms2g</additionalJOption> <additionalJOption>-J-Xmx2g</additionalJOption> 

Answer 2

@lbndev is right, at least with the default Doclet (com.sun.tools.doclets.formats.html.HtmlDoclet) that is supplied with Javadoc. A look through the source confirms the single threaded implementation:

• The starting point is in the parent class AbstractDoclet.startGeneration()
• This in turn calls new ClassTree() which calls ClassTree.buildTree() which uses a for loop to iterate the list of classes, generating models of the classes
• The next step is AbstractDoclet.generateClassFiles(), again a for loop on the packages
• This then drills down to HtmlDoclet.generateClassFiles(), which iterates package.allClasses(), again in a for loop.

(Those links are to JDK 8 source. With JDK 11 the classes have moved, but the basic for loops in HtmlDoclet and AbstractDoclet are still there.)

Some sample based profiling confirmed these are the methods that are the bottleneck:

This won't be what you're hoping to hear, but this looks like no option in the current standard Javadoc for multi-threading, at least within a single Maven module.

generateClassFiles() etc would lend themselves well to a bit of multithreading, though this would probably need to be a change in the JDK. As mentioned below AbstractDoclet.isValidDoclet() even actively blocks subclassing of HtmlDoclet. Trying to reimplement some of those loops as a third party would need to pull in a lot of other code.

A scan around other Doclet implementations (e.g. javadown) only found a similar implementation style around the package and class drilldown. It's possible others on this thread will know more.

Thinking a bit more widely, there might be room for tuning around DocFileFactory. It's clearly marked up as an internal class (not even public in the package), but it does abstract the writing of the (HTML) files. It seems possible an alternative version of this could buffer the HTML in memory, or stream directly to a zip file, to improve the IO performance. But clearly this would also need to understand the risk of change in the JDK tools.