In spark," INFO metrics.MetricsSaver: Saved 10:24 records to ...."

Question

I am processing a 500M text file(.gz) in amazon sc3 using spark and spark sql on amazon EMR(6 node clusters). And my program keep printing out " INFO metrics.MetricsSaver: Saved 10:24 records to /mnt/var/em/raw/i-d1bfb6dd_20150301_SparkSubmit_05013_raw.bin" for several hours.

My questions are:

1. what is the meaning of the log?
2. How can I solve this issue?

Thanks in advance.

Answer 1

As Sean said, it's just metrics and it's nothing to be concerned about. It will log messages like that on a regular basis even if you're not actively doing anything.

To suppress those messages, you can update the following line in /home/hadoop/spark/conf/log4j.properties.

Change log4j.logger.amazon.emr.metrics=INFO to log4j.logger.amazon.emr.metrics=WARN.