Use maven antlr4 plugin to generate grammars from other artifacts

Question

I want to use antlr4 to generate a java parser api for a grammar. I realize other people maybe be interested in the grammar part so I want to keep the parser artifact in maven separate from the grammar artifact.

I have almost got everything working the problem seems to be that it generates the *.java files with the right package but puts them in a directory representing a different package.

I have a multi module maven project with a parent and two modules; grammar and parser. grammar produces a simple jar with this contents:

META-INF/
META-INF/MANIFEST.MF
org/
org/boazglean/
org/boazglean/dabar/
org/boazglean/dabar/grammer/
org/boazglean/dabar/grammer/DabarLexer.g4
org/boazglean/dabar/grammer/DabarParser.g4
META-INF/maven/
META-INF/maven/org.boazglean.dabar/
META-INF/maven/org.boazglean.dabar/grammer/
META-INF/maven/org.boazglean.dabar/grammer/pom.xml
META-INF/maven/org.boazglean.dabar/grammer/pom.properties

Now I want to produce a parser jar, with a bunch of parser classes in the package org.boazglean.dabar.parser. Here is the pom configuration I use:

<plugin>
    <groupId>org.antlr</groupId>
    <artifactId>antlr4-maven-plugin</artifactId>
    <version>4.0</version>
    <executions>
        <execution>
            <goals>
                <goal>antlr4</goal>
            </goals>
            <configuration>
                <sourceDirectory>${antlr.dir}</sourceDirectory>
                <listener>true</listener>
                <visitor>true</visitor>
                <arguments>
                    <argument>-package</argument>
                    <argument>org.boazglean.dabar.parser</argument>
                </arguments>
            </configuration>
        </execution>
    </executions>
</plugin>
<plugin>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>extract-grammer</id>
            <phase>initialize</phase>
            <goals>
                <goal>unpack-dependencies</goal>
            </goals>
            <configuration>
                <includeArtifactIds>grammer</includeArtifactIds>
                <outputDirectory>${antlr.dir}</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

Now the problem is that the generated sources come out in the wrong directory (but they do have the right package).

head -n 2 parser/target/generated-sources/antlr4/org/boazglean/dabar/grammer/DabarLexer.java 

// Generated from org/boazglean/dabar/grammer/DabarLexer.g4 by ANTLR 4.0
package org.boazglean.dabar.parser;

Am I missing some command I should be configuring antlr4 with?

Answer 1

1. Make the grammar folder structure /src/main/antlr4/org/nimy/antlr4/xml/xxxx.g4

2. in pom.xml

<plugin>
    <groupId>org.antlr</groupId>
    <artifactId>antlr4-maven-plugin</artifactId>
    <version>4.1</version>
    <configuration>
        <listener>true</listener>
        <visitor>true</visitor>
        <arguments>
            <argument>-package</argument>
        </arguments>
    </configuration>
    <executions>
        <execution>
            <id>antlr-generate</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>antlr4</goal>
            </goals>
        </execution>
    </executions>
</plugin>

3. Java code is generated in package： package org.nimy.antlr4.xml

antlt4 maven plugin source code on github. Try to read it and get the way to configure the project. antlr4 source code

Answer 2

For the grammar project. The folder structure should look like:

ls -R grammar/
grammar/:
pom.xml  src

grammar/src:
main  test

grammar/src/main:
resources

grammar/src/main/resources:
org

grammar/src/main/resources/org:
boazglean

grammar/src/main/resources/org/boazglean:
dabar

grammar/src/main/resources/org/boazglean/dabar:
grammar

grammar/src/main/resources/org/boazglean/dabar/grammar:
DabarLexer.g4  DabarParser.g4

Now when you build this project you'll have a jar structured like this:

jar -tf grammar/target/grammar-1.0-SNAPSHOT.jar 
META-INF/
META-INF/MANIFEST.MF
org/
org/boazglean/
org/boazglean/dabar/
org/boazglean/dabar/grammar/
org/boazglean/dabar/grammar/DabarParser.g4
org/boazglean/dabar/grammar/DabarLexer.g4
META-INF/maven/
META-INF/maven/org.boazglean.dabar/
META-INF/maven/org.boazglean.dabar/grammar/
META-INF/maven/org.boazglean.dabar/grammar/pom.xml
META-INF/maven/org.boazglean.dabar/grammar/pom.properties

Now for the parser project. It is going to consume the grammars in org.boazglean.dabar.grammar and produce a parser in org.boazglean.dabar.parser

For the parser project. The folder structure should look like:

ls -R parser/
parser/:
pom.xml  src

parser/src:
test

parser/src/test:
java

parser/src/test/java:
org

parser/src/test/java/org:
boazglean

parser/src/test/java/org/boazglean:
dabar

parser/src/test/java/org/boazglean/dabar:
parser

parser/src/test/java/org/boazglean/dabar/parser:
DabarLexerTest.java

Now the heavy lifting comes in the pom.xml For use later:

<properties>
    <antlr.grammar.dir>${project.build.directory}/grammar/</antlr.grammar.dir>
    <antlr.parser.dir>${project.build.directory}/generated-sources/antlr/</antlr.parser.dir>
</properties>

First we are going to need to extract the grammar, from the grammar jar. This allows antlr to run against it.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>extract-grammar</id>
            <phase>initialize</phase>
            <goals>
                <goal>unpack-dependencies</goal>
            </goals>
            <configuration>
                <includeArtifactIds>grammar</includeArtifactIds>
                <outputDirectory>${antlr.grammar.dir}</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

Now that grammar is downloaded we can generate the parser. In order to do that we need to point antlr not at the root of the package (${antlr.grammar.dir}) but at the top of the package ${antlr.grammar.dir}/org/boazglean/dabar/grammar/ Likewise we'll have to produce the java files not at the root of the package ${project.build.directory}/generated-sources/antlr/ , but at the top of the package ${project.build.directory}/generated-sources/antlr/org/boazglean/dabar/parser. To make this consistent with the groupId and artifactId of the project we'll use the build helper plugin to generate some new properties for use by replace all the '.' in the groupId with '/' to form the path.

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>antlr.grammar.package.dir</id>
            <goals>
                <goal>regex-property</goal>
            </goals>
            <configuration>
                <name>antlr.grammar.package.dir</name>
                <regex>\.</regex>
                <value>${antlr.grammar.dir}/${project.groupId}/grammar</value>
                <replacement>/</replacement>
            </configuration>
        </execution>
        <execution>
            <id>antlr.parser.package.dir</id>
            <goals>
                <goal>regex-property</goal>
            </goals>
            <configuration>
                <name>antlr.parser.package.dir</name>
                <regex>\.</regex>
                <value>${antlr.parser.dir}/${project.groupId}/${project.artifactId}</value>
                <replacement>/</replacement>
            </configuration>
        </execution>
    </executions>
</plugin>

Now we have two new properties; antlr.grammar.package.dir, antlr.parser.package.dir that give us the correct path needed by antlr. Now we can call antlr

<plugin>
    <groupId>org.antlr</groupId>
    <artifactId>antlr4-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>generate-antlr-sources</id>
            <goals>
                <goal>antlr4</goal>
            </goals>
            <configuration>
                <sourceDirectory>${antlr.grammar.package.dir}</sourceDirectory>
                <listener>true</listener>
                <visitor>true</visitor>
                <outputDirectory>${antlr.parser.package.dir}</outputDirectory>
                <arguments>
                    <argument>-package</argument>
                    <argument>${project.groupId}.${project.artifactId}</argument>
                </arguments>
            </configuration>
        </execution>
    </executions>
</plugin>

One last thing, antlr generates token files that are quite useful. But at this point they won't get into your parser.jar. So we'll add it as a resource. We do not want to add the java files which are sitting right next to the token files so we'll add an exclude to the resource:

<resource>
    <directory>${antlr.parser.dir}</directory>
    <includes>
        <include>**/*.tokens</include>
    </includes>
</resource>

Now you can build, your test will be able to use the antlr generated parser and your parser jar will look like this:

jar -tf parser/target/parser-1.0-SNAPSHOT.jar 
META-INF/
META-INF/MANIFEST.MF
org/
org/boazglean/
org/boazglean/dabar/
org/boazglean/dabar/parser/
org/boazglean/dabar/parser/DabarLexer.tokens
org/boazglean/dabar/parser/DabarParser.tokens
org/boazglean/dabar/parser/DabarLexer.class
org/boazglean/dabar/parser/DabarParserListener.class
org/boazglean/dabar/parser/DabarParser$ProgramContext.class
org/boazglean/dabar/parser/DabarParser$PhraseContext.class
org/boazglean/dabar/parser/DabarParser$CallContext.class
org/boazglean/dabar/parser/DabarParser$PassContext.class
org/boazglean/dabar/parser/DabarParser$ReferenceContext.class
org/boazglean/dabar/parser/DabarParser$SentenceContext.class
org/boazglean/dabar/parser/DabarParser$CompoundContext.class
org/boazglean/dabar/parser/DabarParser$BlockContext.class
org/boazglean/dabar/parser/DabarParser.class
org/boazglean/dabar/parser/DabarParserBaseVisitor.class
org/boazglean/dabar/parser/DabarParserVisitor.class
org/boazglean/dabar/parser/DabarParserBaseListener.class
META-INF/maven/
META-INF/maven/org.boazglean.dabar/
META-INF/maven/org.boazglean.dabar/parser/
META-INF/maven/org.boazglean.dabar/parser/pom.xml
META-INF/maven/org.boazglean.dabar/parser/pom.properties

Success.