Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to increase Java heap space on Spark Amazon EC2 cluster?

I wrote a program using Java Spark API. Given my data is big, I am getting the following error

java.lang.OutOfMemoryError: Java heap space

Any idea how to increase java heap space of Spark EC2 cluster on Aws? I can give details about my code and cluster setup if necessary. Thanks.

like image 523
Rajiur Rahman Avatar asked Sep 29 '22 03:09

Rajiur Rahman


1 Answers

I was able to increase the Heap space by adding two flags while submitting the application jar to spark-submit .

--executor-memory 10g --driver-memory 2g

like image 106
Rajiur Rahman Avatar answered Oct 13 '22 00:10

Rajiur Rahman