IllegalAccessError in Spark caused by async-http-client

Question

Context: I am working on a Spark streaming job that writes data to InfluxDB, using this library. Here's the environment.

• Scala 2.11.8
• Spark 2.1.0 (Dockerized standalone cluster)

relevant dependencies:

"org.apache.spark" %% "spark-core" % "2.1.0" % "provided",
"org.apache.spark" %% "spark-streaming" % "2.1.0" % "provided",
"org.apache.spark" %% "spark-streaming-kafka-0-8" % "2.1.0",
"com.paulgoldbaum" %% "scala-influxdb-client" % "0.5.2" // which uses "org.asynchttpclient" % "async-http-client" % "2.0.24"

Everything compiles and runs fine on my local computer, but when I submit the assembly jar to the Spark cluster, I get this error in the driver:

Exception in thread "main" java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
  at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.IllegalAccessError: tried to access field io.netty.handler.ssl.JdkSslContext.SUPPORTED_CIPHERS from class io.netty.handler.ssl.NettySslPackageAccessor
  at io.netty.handler.ssl.NettySslPackageAccessor.jdkSupportedCipherSuites(NettySslPackageAccessor.java:24)
  at org.asynchttpclient.config.AsyncHttpClientConfigDefaults.defaultEnabledCipherSuites(AsyncHttpClientConfigDefaults.java:85)
  at org.asynchttpclient.DefaultAsyncHttpClientConfig$Builder.<init>(DefaultAsyncHttpClientConfig.java:635)
  at org.asynchttpclient.DefaultAsyncHttpClient.<init>(DefaultAsyncHttpClient.java:67)
  at com.paulgoldbaum.influxdbclient.HttpClient.<init>(HttpClient.scala:21)
  at com.paulgoldbaum.influxdbclient.InfluxDB$.connect(InfluxDB.scala:16)
  ...

The problem disappear if I remove the code for writing to InfluxDB.

What I've learned after some looking around is that the class io.netty.handler.ssl.NettySslPackageAccessor actually belongs to the async-http-client library. It seems to be a hack-class to access protected member in io.netty.handler.ssl.JdkSslContext.

I messed about with this problem for a few days. The solution I got to make it work is overriding the async-http-client to earlier version that doesn't include the offending code.

dependencyOverrides ++= Set("org.asynchttpclient" % "async-http-client" % "2.0.12")

Question: Why is the IllegalAccessError happening only on the cluster and not in my local run? Is there a better way to solve this issue?

If my SBT can compile fine, then there shouldn't be any such IllegalAccessError, so it means there are differences between my local code and the cluster code, which is probably the provided spark dependencies, but it is the same version as the cluster.

I'm kind of OK leaving things as it is, but it would be better if the newer versions could be used. Or at the very least I want to understand why this problem occurs and avoid it in the future.

Answer 1

This is caused by having both io.netty:nettyand org.asynchttpclient:async-http-client in your classpath. If you want to use netty and assync-http-client,add the following dependencies to your gradle (similar for maven pom.xml) build script:

compile 'org.asynchttpclient:async-http-client:2.0.38' compile 'org.asynchttpclient:async-http-client-netty-utils:2.0.38'

Answer 2

I ran into the same issue today and found this issue on github which explains the problem. Basically you have multiple ClassLoaders when using Spark

and io.netty.handler.ssl.NettySslPackageAccessor and io.netty.handler.ssl.JdkSslContext are loaded by different ClassLoaders.

If this is the case, the attempt access package-private static field JdkSslContext.SUPPORTED_CIPHERS will fail with an IllegalAccessError since the package-private fields are 'scoped' at the ClassLoader level.

Oh, and your solution also worked for me, thanks.