How to turn off INFO from logs in PySpark with no changes to log4j.properties?

Question

I working in a cluster where I do not have the permission to change the file log4j.properties to stop the info logging while using pyspark (as explained in first answer here.) The following solution as explained in the above question's first answer work for spark-shell (scala)

import org.apache.log4j.Logger
import org.apache.log4j.Level

But for spark with python (ie pyspark), it didn't work nor the following

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

How can I stop the verbose printing of info in pyspark WITHOUT changing log4j.properties file?

Answer 1

I used sc.setLogLevel("ERROR") because I didn't have write access to our cluster's log4j.properties file. From the docs:

Control our logLevel. This overrides any user-defined log settings. Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN

Answer 2

This helps for me:

import logging
s_logger = logging.getLogger('py4j.java_gateway')
s_logger.setLevel(logging.ERROR)
spark_context = SparkContext()