How to set heap size in spark within the Eclipse environment?

Question

I am trying to run the simple following code using spark within Eclipse:

import org.apache.spark.sql.SQLContext
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
object jsonreader {  
  def main(args: Array[String]): Unit = {
    println("Hello, world!")
    val conf = new SparkConf()
      .setAppName("TestJsonReader")
      .setMaster("local")
      .set("spark.driver.memory", "3g") 
    val sc = new SparkContext(conf)

    val sqlContext = new SQLContext(sc)
    val df = sqlContext.read.format("json").load("text.json")

    df.printSchema()
    df.show   
  }
}

However, I get the following errors:

16/08/18 18:05:28 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.

I followed different tutorials like this one: How to set Apache Spark Executor memory. Most of time either I use --driver-memory option (not possible with Eclipse) or by modifiying the spark configuration but there is no corresponding file.

Does anyone have any idea about how to solve this issue within Eclipse environment?

Answer 1

In Eclipse go to Run > Run Configurations... > Arguments > VM arguments and set max heapsize like -Xmx512m.

Answer 2

I had this issue as well and this is how I solved it. Thought it might be helpful.

val conf: SparkConf = new SparkConf().setMaster("local[4]").setAppName("TestJsonReader").set("spark.driver.host", "localhost")
conf.set("spark.testing.memory", "2147480000")