How /where are the jvm options used by the pyspark script when launching the jvm it connects to specified?
I am specifically interested in specifying jvm debugging options e.g.
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
Thanks.
pyspark
uses the bin/spark-class
script to start the client that you see running in your terminal / console. You can just append whatever options necessary to JAVA_OPTS
:
JAVA_OPTS="$JAVA_OPTS -Xmx=2g -Xms=1g -agentlib:jdwp=transport=dt_socket,server=y..."
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With