I have recently found a way to use logback instead of log4j in Apache Spark (both for local use and spark-submit
). However, there is last piece missing.
The issue is that Spark tries very hard not to see logback.xml
settings in its classpath. I have already found a way to load it during local execution:
Basically, checking for System property logback.configurationFile
, but loading logback.xml
from my /src/main/resources/
just in case:
// the same as default: https://logback.qos.ch/manual/configuration.html
private val LogbackLocation = Option(System.getProperty("logback.configurationFile"))
// add some default logback.xml to your /src/main/resources
private lazy val defaultLogbackConf = getClass.getResource("/logback.xml").getPath
private def getLogbackConfigPath = {
val path = LogbackLocation.map(new File(_).getPath).getOrElse(defaultLogbackConf)
logger.info(s"Loading logging configuration from: $path")
path
}
And then when I initialize my SparkContext...
val sc = SparkContext.getOrCreate(conf)
sc.addFile(getLogbackConfigPath)
I can confirm it works locally.
spark-submit
spark-submit \
...
--master yarn \
--class com.company.Main\
/path/to/my/application-fat.jar \
param1 param2
This gives an error:
Exception in thread "main" java.io.FileNotFoundException: Added file file:/path/to/my/application-fat.jar!/logback.xml does not exist
Which I think is nonsense, because first the application, finds the file (according to my code)
getClass.getResource("/logback.xml").getPath
and then, during
sc.addFile(getLogbackConfigPath)
it turns out... whoa! no file there!? What the heck!? Why would it not find the file inside the jar. It obviously is there, I did triple checked it.
spark-submit
So I thought, OK. I will pass my file, as I could specify the system property. I put the logback.xml
file next to my application-fat.jar
and:
spark-submit \
...
--conf spark.driver.extraJavaOptions="-Dlogback.configurationFile=/path/to/my/logback.xml" \
--conf spark.executor.extraJavaOptions="-Dlogback.configurationFile=/path/to/my/logback.xml" \
--master yarn \
--class com.company.Main\
/path/to/my/application-fat.jar \
param1 param2
And I get the same error as above. So my setting is completely ignored! Why? How to specify
-Dlogback.configurationFile
properly and pass it as properly to driver and executors?
Thanks!
java.io.FileNotFoundException
This is probably unsolvable.
Simply, SparkContext.addFile
can not read the file from inside the Jar. I believe it is treated as it was in some zip
or alike.
Fine.
-Dlogback.configurationFile
This was not working due to my misunderstanding of the configuration parameters.
Because I am using --master yarn
parameter, but I do not specify --deploy-mode
to cluster
it is by default client
.
Reading https://spark.apache.org/docs/1.6.1/configuration.html#application-properties
spark.driver.extraJavaOptions
Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file.
So passing this setting with --driver-java-options
worked:
spark-submit \
...
--driver-java-options "-Dlogback.configurationFile=/path/to/my/logback.xml" \
--master yarn \
--class com.company.Main\
/path/to/my/application-fat.jar \
param1 param2
--driver-java-options
In contrast to --conf
multiple parameters have to be passed as one parameter, example:
--driver-java-options "-Dlogback.configurationFile=/path/to/my/logback.xml -Dother.setting=value" \
And the following will not work
--driver-java-options "-Dlogback.configurationFile=/path/to/my/logback.xml" \
--driver-java-options "-Dother.setting=value" \
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With