pySpark: Get executor id

Question

How can I get the executor id when running pySpark code? I am aware that with scala I can use SparkEnv.get().executorId(), but I cannot find the equivalent when using pySpark.

Answer 1

You can use the REST API to query the executors, I have used it in pySparkUtils to find the executor IPs

Boaz