Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Specify options for the jvm launched by pyspark

How /where are the jvm options used by the pyspark script when launching the jvm it connects to specified?

I am specifically interested in specifying jvm debugging options e.g.

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

Thanks.

like image 439
WestCoastProjects Avatar asked Mar 16 '23 12:03

WestCoastProjects


1 Answers

pyspark uses the bin/spark-class script to start the client that you see running in your terminal / console. You can just append whatever options necessary to JAVA_OPTS:

JAVA_OPTS="$JAVA_OPTS -Xmx=2g -Xms=1g -agentlib:jdwp=transport=dt_socket,server=y..."
like image 144
Lyuben Todorov Avatar answered Mar 19 '23 13:03

Lyuben Todorov