How to override Spark's log4j.properties per driver?

Question

I'm trying to override Spark's default log4j.properties, but haven't had any luck. I tried the adding the following to spark-submit:

--conf "spark.executor.extraJavaOptions=Dlog4j.configuration=/tmp/log4j.properties"   --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=/tmp/log4j.properties" 

But that didn't seem to work. I also tried using --files option in spark-submit and that also didn't seem to work. Has anyone got logging setup so you have a log4j.properties file per driver and not using the default?

I'm using Mesos and Marathon to run the Spark driver. I wasn't sure of the --files option and I couldn't find any examples of how it's used and what it does exactly.

I would also like to mention that I manually uploaded the log4j.properties file to all my nodes that had my changes for testing.

Version of Spark is 1.1.0 as of right now.

Answer 1

For the driver/shell you can set this with the --driver-java-options when running spark-shell or spark-submit scripts.

In Spark you cannot set --conf spark.driver.extraJavaOptions because that is set after the JVM is started. When using the spark submit scripts --driver-java-options substitutes these options into the launch of the JVM (e.g. java -Dblah MyClass) that runs the driver.

Note that the -Dlog4j.configuration property should be a valid URL, so if its from somewhere on your file system use file: URL. If the resource variable cannot be converted to a URL, for example due to a MalformedURLException, then log4j will search for the resource from the classpath.

For example, to use a custom log4j.properties file;

./spark-shell --driver-java-options "-Dlog4j.configuration=file:///etc/spark/my-conf/log4j.warnonly.properties"