Can't make Maven test-scoped dependencies work with Java 9 (nor 10) modules

Question

Edit: I analyzed the given answers. In particular I tested Till Brychcy's hypothesis, which seems to hold, but opens more questions. I added that analysis to the end of the question, after the following big separator: "-----------Analysis of Till Brychcy's answer----------"

Somehow it compiles and runs in IJ, but mvn clean install fails to compile the test

Explanation of the 2 modules, and what fails

I have a java 9 maven project with 2 modules: apimod and clientmod. Module clientmod depends on module apimod (those modules are both Maven modules and Java 9 modules).

Also, i want module clientmod to be able to reuse not only production code from apimod, but also test code. This is a common pattern, that I used many times with Java 8. With Java 9 (it's the same with Java 10) it also works fine, as long as i don't declare module-info.java (that is, as long as I don't run with the module system).

But as soon as I do, enabling the test dependency seems to disable the production dependency: api.Base (an src/main class of module apimod) is no longer visible from client.test.DerivedTest (an src/test class of module clientmod). The test doesn't compile anymore.

Is this is a bug in Maven or in Java 9? This is with the most recent releases: Java 9.0.4 (it's the same with Java 10), Maven 3.5.3, maven-compiler-plugin 3.7.0

My analysis so far

The code

The source code is at:

git clone https://github.com/vandekeiser/wires.git

I "dichotomized" the issue with a failing test in a branch:

git checkout MINIMIZE_ISSUE
`mvn clean install` 

-> BUILD FAIL (compilation error in the test of clientmod)

Maven test-scoped dependency

I want module clientmod to be able to reuse not only production code from apimod but also test code. With Maven you do it like that (clientmod/pom.xml):

<dependency>
    <groupId>fr.cla</groupId>
    <artifactId>apimod</artifactId>
    <version>${project.version}</version>
    <classifier>tests</classifier>
    <scope>test</scope>
</dependency>

Java 9 modules

module apimod {
    exports api;
}

module clientmod {
    requires apimod;
}

The failure when trying to enable both module systems

With Java 9 if I declare both the test-scoped dependency and the Java 9 modules the test doesn't compile anymore (mvn clean install output):

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:testCompile (default-testCompile) on project clientmod: Compilation failure
[ERROR] /G:/projets/wires/wires/wires/clientmod/src/test/java/client/test/DerivedTest.java:[8,22] cannot access api.Base
[ERROR]   class file for api.Base not found

Reproducing the issue with javac: a bug in module patching?

It's as if enabling the test dependency (src/test) disables the production dependency (src/main). I know that in this scenario Maven is supposed to use the javac --patch-module flag. So i reproduced the issue using just javac (using the debug output of mvn -X):

Same compilation, skipping Maven:

javac "G:\projets\wires\wires\wires\clientmod\src\test\java\client\test\DerivedTest.java" \
-d "G:\projets\wires\wires\wires\clientmod\target\test-classes" \
-classpath "G:\projets\wires\wires\wires\clientmod\target\test-classes;" \
--module-path "G:\projets\wires\wires\wires\apimod\target\apimod-1.0-SNAPSHOT-tests.jar;G:\projets\wires\wires\wires\clientmod\target\classes;G:\projets\wires\wires\wires\apimod\target\apimod-1.0-SNAPSHOT.jar;" \
-sourcepath "G:\projets\wires\wires\wires\clientmod\src\test\java;" \
--release 9 \
-Xlint:all \
--patch-module clientmod="G:\projets\wires\wires\wires\clientmod\target\classes;G:\projets\wires\wires\wires\clientmod\src\test\java;"

Same compilation error:

G:\projets\wires\wires\wires\clientmod\src\test\java\client test\DerivedTest.java:8: error: cannot access Base
new Derived().equals(null);                         ^
class file for api.Base not found
1 error

I tried to use the javac flags that are supposed to disable the module system but they don't seem to exist in my 64 bits Windows Oracle JVM? (Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode):

javac --illegal-access=warn
javac: invalid flag: --permit-illegal-access
javac --permit-illegal-access
javac: invalid flag: --illegal-access=warn

Adding (logically unneeded, done in desperation) exports or reads doesn't change anything either:

--add-reads apimod=ALL-UNNAMED \
--add-reads clientmod=ALL-UNNAMED \
--add-exports apimod/api=ALL-UNNAMED \
--add-exports clientmod/client=ALL-UNNAMED \

Output of mvn -version:

Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T20:49:05+01:00)
Maven home: G:\software\apache-maven-3.5.3
Java version: 9.0.4, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk-9.0.4
Default locale: fr_FR, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows" 

Analysis of Till Brychcy's answer

Thanks for your detailed answer, which is essentially that "It looks like maven doesn't support this use case yet" -->So let's try to reproduce the issue without maven. I created branch TRY_ADAPT_khmarbaise-MINIMIZE_ISSUE for these trials (sorry for the confusing branch name).

1. Adapting my former command line which is logged by maven, that is:

   javac "G:\projets\wires\wires\wires\clientmod\src\test\java\client\test\DerivedTest.java"
   -d "G:\projets\wires\wires\wires\clientmod\target\test-classes"
   -classpath "G:\projets\wires\wires\wires\clientmod\target\test-classes;"
   --module-path "G:\projets\wires\wires\wires\apimod\target\apimod-1.0-SNAPSHOT-tests.jar;G:\projets\wires\wires\wires\clientmod\target\classes;G:\projets\wires\wires\wires\apimod\target\apimod-1.0-SNAPSHOT.jar;"
   -sourcepath "G:\projets\wires\wires\wires\clientmod\src\test\java;"
   --release 9
   -Xlint:all
   --patch-module clientmod="G:\projets\wires\wires\wires\clientmod\target\classes;G:\projets\wires\wires\wires\clientmod\src\test\java;"
   --add-reads apimod=ALL-UNNAMED
   --add-reads clientmod=ALL-UNNAMED
   --add-exports apimod/api=ALL-UNNAMED
   --add-exports clientmod/client=ALL-UNNAMED
   --add-modules apimod
   

2. I remove G:\projets\wires\wires\wires\apimod\target\apimod-1.0-SNAPSHOT-tests.jar; from --module-path

3. I add the same to --patch-module clientmod, giving me:

   javac "G:\projets\wires\wires\wires\clientmod\src\test\java\client\test\DerivedTest.java" \
   -d "G:\projets\wires\wires\wires\clientmod\target\test-classes" \
   -classpath "G:\projets\wires\wires\wires\clientmod\target\test-classes;" \
   --module-path "G:\projets\wires\wires\wires\clientmod\target\classes;G:\projets\wires\wires\wires\apimod\target\apimod-1.0-SNAPSHOT.jar;" \
   -sourcepath "G:\projets\wires\wires\wires\clientmod\src\test\java;" \
   --release 9 \
   -Xlint:all \
   --patch-module clientmod="G:\projets\wires\wires\wires\clientmod\target\classes;G:\projets\wires\wires\wires\clientmod\src\test\java;G:\projets\wires\wires\wires\apimod\target\apimod-1.0-SNAPSHOT-tests.jar;" \
   --add-reads apimod=ALL-UNNAMED \
   --add-reads clientmod=ALL-UNNAMED \
   --add-exports apimod/api=ALL-UNNAMED \
   --add-exports clientmod/client=ALL-UNNAMED \
   --add-modules apimod
   

-->OK, it compiles now! So your hypothesis that maven-compiler-plugin, or maven, doesn't support this yet seems validated. But I think it is supposed to be supported in the version I use, which is the latest. I wonder where to start to check that..

Anyway, meanwhile, I tried configuring maven-compiler-plugin explicitly but to no avail. The general structure is I tried is (maven-compiler-plugin.version = 3.7.0):

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>${maven-compiler-plugin.version}</version>
    <configuration>
        <source>${java.version}</source>
        <target>${java.version}</target>
        <release>${java.version}</release>
        <compilerArgs>
            [...]
        </compilerArgs>
    </configuration>
</plugin>

I tried the following compilerArgs (corresponding mvn clean install errors in xml comments):

1:

<!--1. Syntaxically OK, but:-->
<!--[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project apimod: Compilation failure: Compilation failure:-->
<!--[ERROR] /G:/projets/wires/wires/wires/apimod/src/main/java/api/Base.java:[1,1]-->
<!--file should be on source path, or on patch path for module-->
<!--[ERROR] /G:/projets/wires/wires/wires/apimod/src/main/java/module-info.java:[1,1]-->
<!--file should be on source path, or on patch path for module-->
<compilerArgs>
    <arg>--class-path=/G/projets/wires/wires/wires/clientmod/target/test-classes;</arg>
    <arg>
        --module-path=/G/projets/wires/wires/wires/clientmod/target/classes;/G/projets/wires/wires/wires/apimod/target/apimod-1.0-SNAPSHOT.jar;
    </arg>
    <arg>--source-path=/G/projets/wires/wires/wires/clientmod/src/test/java;</arg>
    <arg>-Xlint:all</arg>
    <arg>
        --patch-module=clientmod=/G/projets/wires/wires/wires/clientmod/target/classes;/G/projets/wires/wires/wires/clientmod/src/test/java;/G/projets/wires/wires/wires/apimod/target/apimod-1.0-SNAPSHOT-tests.jar;
    </arg>
    <arg>--add-reads=apimod=ALL-UNNAMED</arg>
    <arg>--add-reads=clientmod=ALL-UNNAMED</arg>
    <arg>--add-exports=apimod/api=ALL-UNNAMED</arg>
    <arg>--add-exports=clientmod/client=ALL-UNNAMED</arg>
    <arg>--add-modules=apimod</arg>
</compilerArgs>

2:

<!--2.-->
<!--[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.-->
<!--[ERROR] There was an error in the forked process-->
<!--[ERROR] api/foo/BaseTest (wrong name: apimod/api/foo/BaseTest)-->
<!--[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process-->
<!--[ERROR] api/foo/BaseTest (wrong name: apimod/api/foo/BaseTest)-->
<!--[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:673)-->
<compilerArgs>
    <arg>--module-source-path=./*/src/main/java;./*/src/test/java/;</arg>
    <arg>
        --source-path=/G/projets/wires/wires/wires/apimod/src/main/java;/G/projets/wires/wires/wires/apimod/src/test/java;/G/projets/wires/wires/wires/clientmod/src/test/java;/G/projets/wires/wires/wires/clientmod/src/main/java;
    </arg>
    <arg>-Xlint:all</arg>
    <arg>
        --patch-module=clientmod=/G/projets/wires/wires/wires/clientmod/target/classes;/G/projets/wires/wires/wires/clientmod/src/test/java;/G/projets/wires/wires/wires/apimod/target/apimod-1.0-SNAPSHOT.jar;/G/projets/wires/wires/wires/apimod/target/apimod-1.0-SNAPSHOT-tests.jar;
    </arg>
    <arg>--add-reads=apimod=ALL-UNNAMED</arg>
    <arg>--add-reads=clientmod=ALL-UNNAMED</arg>
    <arg>--add-exports=apimod/api=ALL-UNNAMED</arg>
    <arg>--add-exports=clientmod/client=ALL-UNNAMED</arg>
    <arg>--add-modules=apimod</arg>
</compilerArgs>    

Answer 1

It works now! It's all explained there: https://issues.apache.org/jira/browse/MCOMPILER-348

The problem was a name clash: changing the name of the maven modules to something different from the java modules fixed it, in the problem-reduction branch and in the main-dev branch.

I still have a warning though (see link), for now I ignore it but it needs to go.

-----------Copy of the link's contents in case the JIRA disappears-----------

Robert Scholte: My guess is that apimod-1.0-SNAPSHOT-tests.jar should have been on the classpath and not on the modulepath. The only reason I can think of that it was added to the modulepath is when its automatic module name is also apimod. Changing the name to something that doesn't collide with the automatic modulename fixes the issue ( e.g. info.example.apimod ) In general, please pick a more unique modulename, it must be globally unique (just like the classes on the classpath). Add reverseDNS or the shared package. Permalink Edit Delete

Me: Yep, it was that, thanks a lot!I thought for sure it was a bug (but, there might yet be some smaller bug? I have yet to look at it more, but there is still a javac warning - see the end of this comment).

It fixed the dichotomy branch and the main branch: -In the dichotomy branch, I added a prefix to the java modules -When I got back to the main branch, I renamed the Maven modules wires.core --> wires-core and so on, and the java modules to the shared root package, also avoiding the name collision

To put it in "Java puzzlers" terms, maybe the lesson here is twofold: -For users of the API (Maven): follow the naming conventions: I shouldn't have named the Maven modules with a '.' separator. I would have avoided the name clash without thinking about it.. -For designers of the API: maybe log a meaningful message if you can detect this clash, because it's hard to see when it does occur

Finally, I still get a javac warning. So for now I have to ignore an exports warning.

I didn't investigate it yet, but here goes:

[WARNING] COMPILATION WARNING :
[INFO] -------------------------------------------------------------
[WARNING] /G:/projets/wires/wires/wires/wires-core/src/test/java/fr/cla/wires/core/MavenVsJavaModulesReproduceTest.java:[9,54] class fr.cla.wires.support.oo.ddd.AbstractValueObjectTest in module  is not exported
[INFO] 1 warning
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /G:/projets/wires/wires/wires/wires-core/src/test/java/fr/cla/wires/core/MavenVsJavaModulesReproduceTest.java: warnings found and -Werror specified
[INFO] 1 error

To reproduce it, checkout tag WARN_IF_UNCOMMENT in the master branch (https://github.com/vandekeiser/wires.git) and remove the exports exception in: -Xlint:all,-serial,-exports

I can't (and shouldn't have to) add exports fr.cla.wires.support.oo.ddd; into module-info.java because that refers to a package which is in src/test

That's where it is for now

Answer 2

It looks like maven doesn't support this use case yet.

The problem is that apimod-1.0-SNAPSHOT-tests.jar is treated as automatic module and its automatic module name (derived from the file name) is "apimod", so the actual module apimod in apimod-1.0-SNAPSHOT.jar (which cames later on the modulepath) is ignored.

Maven should detect that apimod-1.0-SNAPSHOT-tests.jar belongs to apimod and use --patch-module apimod=G:\projets\wires\wires\wires\apimod\target\apimod-1.0-SNAPSHOT-tests.jar instead of making it part of --module-path