NoClassDefFoundError for io.netty.buffer.PooledByteBufAllocator

Question

I have created a Cassandra Client written with Achilles Object mapping in java (using IntelliJ + Gradle). My Client works fine locally in IntelliJ, but throws the exception when deployed in a docker container. I am currently stuck with the below exception in my docker container.

java.lang.NoClassDefFoundError: Could not initialize class io.netty.buffer.PooledByteBufAllocator at com.datastax.driver.core.NettyOptions.afterBootstrapInitialized(NettyOptions.java:144) at com.datastax.driver.core.Connection$Factory.newBootstrap(Connection.java:903) at com.datastax.driver.core.Connection$Factory.access$100(Connection.java:751) at com.datastax.driver.core.Connection.initAsync(Connection.java:139) at com.datastax.driver.core.Connection$Factory.open(Connection.java:807) at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:252) at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:201) at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79) at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1631) at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1549) at com.datastax.driver.core.Cluster.init(Cluster.java:160) at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:342) at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:317) at com.datastax.driver.core.Cluster.connect(Cluster.java:259) at java.util.Optional.orElseGet(Optional.java:267) at info.archinnov.achilles.configuration.ArgumentExtractor.initSession(ArgumentExtractor.java:186) at info.archinnov.achilles.configuration.ArgumentExtractor.initConfigContext(ArgumentExtractor.java:96) at info.archinnov.achilles.bootstrap.AbstractManagerFactoryBuilder.buildConfigContext(AbstractManagerFactoryBuilder.java:60) at info.archinnov.achilles.generated.ManagerFactoryBuilder.build(ManagerFactoryBuilder.java:38) at com.ds.db.cassandra.AchillesClient.(AchillesClient.java:22) at com.ds.message.RabbitMQMsgClient$1.open(RabbitMQMsgClient.java:114) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)

But the class, io.netty.buffer.PooledByteBufAllocator which is part of netty-buffer-4.0.56.Final.jar is already part of the classpath.

When I tried testing thing locally from my Intellij IDE, things are working fine. But after deployment, I am facing this issue in my docker container.

The service is started in my docker container like this:

java -server -XX:HeapDumpPath=/opt/ds/srv/diagnostics/msgreader-1589749851-2s89z.heapdump -Xmx614m -Xms614m -XX:MaxMetaspaceSize=126M -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC -XX:CICompilerCount=4 -XX:MaxGCPauseMillis=1000 -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -Dsun.net.inetaddr.ttl=60 -XX:OnOutOfMemoryError=kill -9 %p -Djava.library.path=/usr/local/lib -Djava.net.preferIPv4Stack=true -Dapp.dir=/opt/ds/sw/apps/msgreader -Dserver.name=msgreader -Dlog.dir=/opt/ds/var/log/msgreader -cp /opt/ds/sw/apps/javacontainer/resources:/opt/ds/sw/apps/msgreader/lib/*:/opt/ds/sw/apps/msgreader/resources:/opt/ds/sw/apps/javacontainer/lib/* com.ds.msg.Server start

From the above cmd, you can notice the -cp argument mentioning the class path. And this path contains netty-buffer-4.0.56.Final.jar.

I later found that netty-all-4.0.51.Final.jar is also part of the class path and this jar also contains the same class file. I even tried removing jars, with all possible combination. But still I am facing the same issue.

Even in case of multiple versions of a jar file, we should be getting NoSuchMethodError, Can anyone please help me understand the problem.

Answer 1

I have finally found the answer, the issue is what I guessed in my question. Multiple versions of same jar, had caused the failure. To find it, I used the following in my gradle file:

apply plugin: 'project-report'

And ran,

gradle htmlDependencyReport

It will give us a good HTML report on the dependencies Tree. We can even use the below cmd, but it will tough to follow up in a multi module gradle projects

gradle dependencies

In the HTML report, I found achilles-core module had dependency on netty-buffer-4.0.56.Final.jar and another module had dependency on netty-all-4.0.51.Final.jar. So when I tried the following for achilles in build.gradle, things were working fine:

compile(group:'info.archinnov', name: 'achilles-core', version: '6.0.0'){
    exclude module: 'netty-buffer'
}

As netty-all-4.0.51.Final.jar already had the classes required for achilles Object mapping, my project started working on deployment.

Another reason for Failure, even after removing the duplicate jars files from the docker container: (Hard)Restarting the pod, in turn created a new pod, which pulls the same Dockerimage from docker repo.

IntelliJ some how, resolves the PATH issue, when running locally :/