how to hide "py4j.java_gateway:Received command c on object id p0"?

Question

Once logging is started in INFO level I keep getting bunch of py4j.java_gateway:Received command c on object id p0 on your logs. How can I hide it?

Answer 1

using the logging module run:

logging.getLogger("py4j").setLevel(logging.ERROR)

Answer 2

None of these answers have worked for me. Even after using the above solutions, I was still getting the logging errors.

After a long search I came across the following that solved my issue.

import logging
logger = spark._jvm.org.apache.log4j
logging.getLogger("py4j.java_gateway").setLevel(logging.ERROR)

I found this solution in the Databricks Knowledge base article