spark executor lost failure

Question

I am using the databricks spark cluster (AWS), and testing on my scala experiment. I have some issue when training on a 10 GB data with LogisticRegressionWithLBFGS algorithm. The code block where I met the issue is as follows:

import org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
val algorithm = new LogisticRegressionWithLBFGS()
algorithm.run(training_set)

First I got a lot executor lost failure and java out of memory issues, then I repartitioned my training_set with more partitions and the out of memory issues are gone, but Still get executor lost failure.

My cluster has 72 cores and 500GB ram in total. Can any one give some idea on this?

Answer 1

LBFGS uses dense vector for storing betas (feature weights) internally and everything is in memory. So regardless of sparsity of features in training set, the total count of features is something to be mindful about.

So to solve this user should either increase executor memory or limit total count of features in the training set.