How to suppress Spark logging in unit tests?

Question

So thanks to easily googleable blogs I tried:

import org.specs2.mutable.Specification  class SparkEngineSpecs extends Specification {   sequential    def setLogLevels(level: Level, loggers: Seq[String]): Map[String, Level] = loggers.map(loggerName => {     val logger = Logger.getLogger(loggerName)     val prevLevel = logger.getLevel     logger.setLevel(level)     loggerName -> prevLevel   }).toMap    setLogLevels(Level.WARN, Seq("spark", "org.eclipse.jetty", "akka"))    val sc = new SparkContext(new SparkConf().setMaster("local").setAppName("Test Spark Engine"))    // ... my unit tests 

But unfortunately it doesn't work, I still get a lot of spark output, e.g.:

14/12/02 12:01:56 INFO MemoryStore: Block broadcast_4 of size 4184 dropped from memory (free 583461216) 14/12/02 12:01:56 INFO ContextCleaner: Cleaned broadcast 4 14/12/02 12:01:56 INFO ContextCleaner: Cleaned shuffle 4 14/12/02 12:01:56 INFO ShuffleBlockManager: Deleted all files for shuffle 4 

Answer 1

Add the following code into the log4j.properties file inside the src/test/resources dir, create the file/dir if not exist

# Change this to set Spark log level log4j.logger.org.apache.spark=WARN  # Silence akka remoting log4j.logger.Remoting=WARN  # Ignore messages below warning level from Jetty, because it's a bit verbose log4j.logger.org.eclipse.jetty=WARN 

When I run my unit tests (I'm using JUnit and Maven), I only receive WARN level logs, in other words no more cluttering with INFO level logs (though they can be useful at times for debugging).

I hope this helps.