Log4j2 custom plugins - annotation processing with Maven Assembly Plugin

Question

I am not very familiar with Maven, I started using it just yesterday, but I like it. In my project I use Log4j2 library for logging and because of insufficiecy of advanced plugins (like appenders, converters) I need to use custom plugins. log4j-api and log4j-core (also with a bunch of other libraries) are added as dependencies in pom.xml associated with my project. Actually I am using version 2.0 of Log4j.

Log4j uses annotation processing to pre-load classes marked as @Plugin. As far as I know, in older releases of log4j, additional plugin entry had to be specified in pom.xml to trigger plugin processing, or the packages with custom plugins had to be typed into the packages attribute in the configuration file (https://logging.apache.org/log4j/2.x/manual/configuration.html#ConfigurationSyntax). But this is not supported since 2.0-rc2.

In v2.0 this should be done automatically, as long as log4j-core is available to the building engine. There is a file Log4j2Plugins.dat in myproject-0.0.1-SNAPSHOT.jar/META-INF/org/apache/logging/log4j/core/config/plugins/ that contains mappings of my custom plugins - that's OK.

For building with Maven I use also Maven Assembly Plugin. Its goal single is binded to package phase. After packaging the project I naturally have one additional jar in the target directory - myproject-0.0.1-SNAPSHOT-jar-with-dependencies.jar. However, Log4j2Plugins.dat file in this jar contains mappings of original plugins, the same file as in the log4j-core library. And that's the problem, since it doesn't hold any references to my custom plugins. It seems that the file from myproject-0.0.1-SNAPSHOT.jar is being overwritten with the original file from the log4j library, but I am not sure what's the case.

So when I run myproject-0.0.1-SNAPSHOT-jar-with-dependencies.jar, log4j can't find the plugin classes from my project. I think that myproject-0.0.1-SNAPSHOT.jar would run ok, but I can't run it without the dependencies.

The packages attribute in configuration should be re-enabled in 2.0.1 release, but if I don't want to wait for the release, I have to use the annotation processing method.

Do you have an idea how to fix it?

---

I tried to run it with release 2.0-rc1 of log4j, where the packages attribute of the configuration element was usable. The result is: log4j successfully loaded the class of my custom plugin. However, there were so many other errors (that arised in this specific release) that make the program even more unusable.

This is one good point, that ensures me that if the packages attribute will be enabled in next release 2.0.1, my plugin will work. It should be reinstated according to this issue tracking: https://issues.apache.org/jira/browse/LOG4J2-741

---

Added my pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.jjurm</groupId>
  <artifactId>twbot</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>twbot</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.1</version>
          <configuration>
            <source>1.7</source>
            <target>1.7</target>
          </configuration>
        </plugin>
      </plugins>
    </pluginManagement>
    <plugins>
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>2.4</version>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
          <archive>
            <manifest>
              <mainClass>com.jjurm.twbot.system.Run</mainClass>
            </manifest>
          </archive>
        </configuration>
        <executions>
          <execution>
            <id>make-assembly</id> <!-- this is used for inheritance merges -->
            <phase>package</phase> <!-- bind to the packaging phase -->
            <goals>
              <goal>single</goal>
            </goals>
            <configuration>
              <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
              </descriptorRefs>
              <archive>
                <manifest>
                  <mainClass>com.jjurm.twbot.system.Run</mainClass>
                </manifest>
              </archive>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.mariadb.jdbc</groupId>
        <artifactId>mariadb-java-client</artifactId>
        <version>1.1.7</version>
    </dependency>
    <dependency>
        <groupId>joda-time</groupId>
        <artifactId>joda-time</artifactId>
        <version>2.3</version>
    </dependency>
    <dependency>
        <groupId>commons-configuration</groupId>
        <artifactId>commons-configuration</artifactId>
        <version>1.10</version>
    </dependency>
    <dependency>
        <groupId>net.snaq</groupId>
        <artifactId>dbpool</artifactId>
        <version>6.0</version>
    </dependency>
    <dependency>
        <groupId>org.fusesource.jansi</groupId>
        <artifactId>jansi</artifactId>
        <version>1.11</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>2.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-api</artifactId>
        <version>2.0</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.7</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-slf4j-impl</artifactId>
        <version>2.0</version>
    </dependency>
    <dependency>
        <groupId>commons-jxpath</groupId>
        <artifactId>commons-jxpath</artifactId>
        <version>1.3</version>
    </dependency>
    <dependency>
        <groupId>net.sourceforge.htmlunit</groupId>
        <artifactId>htmlunit</artifactId>
        <version>2.15</version>
    </dependency>
  </dependencies>
</project>

Answer 1

I think the problem stems from packaging the dependencies into the jar. Doing a quick dive into the code, it looks like the plugin processor overwrites the plugin dat file for each set of plugins it handles. My guess is that during the packaging process, your custom plugins are processed and written to the dat file, and then overwritten when your log4j dependency is processed for inclusion in the package. There may be better solutions, but off the top of my head I would suggest that you do one of the following:

1. Do not package the dependencies into your jar. Just package your project and then include the dependencies on your classpath when you execute. Even if you want to package everything in a single portable jar, doing this would allow you to at least confirm if your plugins are being overwritten, or if there is something else wrong.

2. Create a separate project for your custom plugins, package it separately from your main project, and then include the resulting jar as a dependency. As with option 1, make sure that you do not include the log4j jars in this package. Once you have created your custom plugin jar, you can package it along with the other dependencies in your main jar and it should work fine since your custom plugin jar will have its own plugin dat file.

Good luck!

Answer 2

Another solution is mentioned in Log4j 2 issue 673. Use maven shade plugin with specific transformer instead of maven assembly plugin.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>2.4.3</version>
  <configuration>
    <transformers>
      <transformer implementation="com.github.edwgiz.mavenShadePlugin.log4j2CacheTransformer.PluginsCacheFileTransformer" />
    </transformers>
  </configuration>
  <dependencies>
    <dependency>
      <groupId>com.github.edwgiz</groupId>
      <artifactId>maven-shade-plugin.log4j2-cachefile-transformer</artifactId>
      <version>2.6.1</version>
    </dependency>
  </dependencies>
  <executions>
    <execution>
      <id>make-assembly</id>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
    </execution>
  </executions>
</plugin>

If I understand it accurately, transformer creates Log4j2Plugins.dat file by correctly merging Log4j2Plugins.dat from all dependencies and the main jar, i.e. all plugins will be included.