Spark ML Pipeline Causes java.lang.Exception: failed to compile ... Code ... grows beyond 64 KB

Question

Using Spark 2.0, I am trying to run a simple VectorAssembler in a pyspark ML pipeline, like so:

feature_assembler = VectorAssembler(inputCols=['category_count', 'name_count'], \
                                    outputCol="features") 
pipeline = Pipeline(stages=[feature_assembler])
model = pipeline.fit(df_train)
model_output = model.transform(df_train)

When I try to look at the output using

model_output.select("features").show(1)

I get the error

Py4JJavaError                             Traceback (most recent call last)
<ipython-input-95-7a3e3d4f281c> in <module>()
      2 
      3 
----> 4 model_output.select("features").show(1)

/usr/local/spark20/python/pyspark/sql/dataframe.pyc in show(self, n, truncate)
    285         +---+-----+
    286         """
--> 287         print(self._jdf.showString(n, truncate))
    288 
    289     def __repr__(self):

/usr/local/spark20/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
    931         answer = self.gateway_client.send_command(command)
    932         return_value = get_return_value(
--> 933             answer, self.gateway_client, self.target_id, self.name)
    934 
    935         for temp_arg in temp_args:

/usr/local/spark20/python/pyspark/sql/utils.pyc in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/usr/local/spark20/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    310                 raise Py4JJavaError(
    311                     "An error occurred while calling {0}{1}{2}.\n".
--> 312                     format(target_id, ".", name), value)
    313             else:
    314                 raise Py4JError(

Py4JJavaError: An error occurred while calling o2875.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure:   
Task 0 in stage 1084.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1084.0 (TID 42910, 169.45.92.174): 
java.util.concurrent.ExecutionException: java.lang.Exception: failed to 
compile: org.codehaus.janino.JaninoRuntimeException: Code of method "

  (Lorg/apache/spark/sql/catalyst/InternalRow;Lorg/apache/spark/sql/catalyst/InternalRow;)I" of class    
"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" 
grows beyond 64 KB

It then lists the generated code, which looks like:

/* 001 */ public SpecificOrdering generate(Object[] references) {
/* 002 */   return new SpecificOrdering(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificOrdering extends org.apache.spark.sql.catalyst.expressions.codegen.BaseOrdering {
/* 006 */
/* 007 */   private Object[] references;
/* 008 */
/* 009 */
/* 010 */   public int compareStruct1(InternalRow a, InternalRow b) {
/* 011 */     InternalRow i = null;
/* 012 */
/* 013 */     i = a;
/* 014 */     boolean isNullA836;
/* 015 */     byte primitiveA834;
/* 016 */     {
/* 017 */
/* 018 */       boolean isNull834 = i.isNullAt(0);
/* 019 */       byte value834 = isNull834 ? (byte)-1 : (i.getByte(0));
/* 020 */       isNullA836 = isNull834;
/* 021 */       primitiveA834 = value834;
/* 022 */     }
/* 023 */     i = b;

…

/* 28649 */     return 0;
/* 28650 */   }
/* 28651 */ }

Why would this simple VectorAssembler generate 28,651 lines of code and exceed 64KB?

Answer 1

It seems like there may be a limit, namely 64KB, to Spark's lazy evaluation. In other words, it's being a little to lazy in this case which is causing it to hit that limit.

I was able to work around this same exception, which I was triggering via a join rather than with a VectorAssembler, by calling cache on one of my Datasets about half way through my pipeline. I don't know (yet) exactly why this solved the issue however.