Get Full Stack Trace in Spark Log

Question

I want to see the full stack trace in my spark executor logs.

I have for example:

Caused by: java.lang.RuntimeException: java.lang.Long is not a valid external type for schema of int at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply_0$(generated.java:434) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(generated.java:737) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290) ... 18 more

I want to see the 18 more that are missing. I have tried changing to logging level etc but it did not help.

Answer 1

Solusion 1. In your spark install folder, there has a conf folder contains log4j.properties file, you could configure that file to log levels.

Spark default used the INFO, buy you could change to DEBUG or TRACE to get all spark logs. The configure template : https://github.com/apache/spark/blob/master/conf/log4j.properties.template

Other options for Level include: all, debug, error, fatal, info, off, trace, trace_int, warn

Solusion 2. Put those logger to your SparkContext() function

import org.apache.log4j.Logger; 
import org.apache.log4j.Level;

Logger.getLogger("org").setLevel(Level.INFO); 
Logger.getLogger("akka").setLevel(Level.INFO);

The level.Info could change to DEBUG OR TRACE etc.