Spark forcing log4j

Question

I have trivial spark project in Scala and would like to use logback, but spark/hadoop seems to be forcing log4j on me.

1. This seems at odds with my understanding of the purpose of slf4j; is it not a oversight in spark/hadoop?

2. Do I have to give up on logback and use log4j, or is there a workaround?

In build.sbt I tried exclusions ...

"org.apache.spark" %% "spark-core" % "1.4.1" excludeAll(
    ExclusionRule(name = "log4j"),
    ExclusionRule(name = "slf4j-log4j12")
),
"org.slf4j" % "slf4j-api" % "1.7.12",
"ch.qos.logback" % "logback-core" % "1.1.3",
"ch.qos.logback" % "logback-classic" % "1.1.3"

... but this results in an exception ...

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Level
    at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:354)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:344)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1659)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:91)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.security.Groups.<init>(Groups.java:55)
    at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2162)
    at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2162)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2162)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:301)
    at spike.HelloSpark$.main(HelloSpark.scala:19)
    at spike.HelloSpark.main(HelloSpark.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level
    at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 20 more

Answer 1

I encountered the same exception as yours. I think you should also add log4j-over-slf4j as dependency besides excluding log4j and slf4j-log4j12. It works for me.

log4j-over-slf4j is a drop in replacement for log4j as it provides exactly same api as log4j and actually routes all calls to log4j to slf4j which in turn routes everything to the underneath log framework. https://www.slf4j.org/legacy.html gives detail explanations.