Apache Spark custom log4j configuration for application

Question

I would like to customize the Log4J configuration for my application in a standalone Spark cluster. I have a log4j.xml file which is inside my application JAR. What is the correct way to get Spark to use that configuration instead of its own Log4J configuration?

I tried using the --conf options to set the following, but no luck.

spark.executor.extraJavaOptions -> -Dlog4j.configuration=log4j.xml
spark.driver.extraJavaOptions -> -Dlog4j.configuration=log4j.xml

I am using Spark 1.4.1 and there's no log4j.properties file in my /conf.

Answer 1

If you are using SBT as package manager/builder:

There is a log4j.properties.template in $SPARK_HOME/conf

• copy it in your SBT project's src/main/resource
• remove the .template suffix
• edit it to fit your needs
• SBT run/package/* will include this in JAR and Spark references it.

Works for me, and will probably include similar steps for other package managers, e.g. maven.

Answer 2

Try using driver-java-options. For example:

spark-submit --class my.class --master spark://myhost:7077 --driver-java-options "-Dlog4j.configuration=file:///opt/apps/conf/my.log4j.properties" my.jar