Eclipse remote debug spark-submit

Question

I want to do a remote debug of an application which is submitted to Spark using the Command Prompt in Windows.

How could I provide Spark the remote debug port to start.

Answer 1

you can always do the Remote Debugging using Spark with Java/scala. If you want to check with small dataset with localmode then you can use the step debugger. Have a look on the Java Debugger and how it works first.

Java step debugger

if you want to use the Spark specific Debugging in Remotemode Then check this link it has very nice example with step by step process.

spark-remote-debugging

For debugging the driver you can add the following to your spark-submit command. Then set your remote debugger to connect to the node you launched your driver program on.

--driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

You can customize the port according to your need. In this case, start your debugger in listening mode, then start your spark program and wait for the executor to attach to your debugger. It's important to set the number of executors to 1 or multiple executors will all try to connect to your debugger, likely causing problems. check the above example for more details.

If you are using jetbrains then you can use this example Spark remote debugging