Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How does Apache spark handle python multithread issues?

According to python's GIL we cannot use threading in CPU bound processes so my question is how does Apache Spark utilize python in multi-core environment?

like image 334
Seyed Vahid Hashemi Avatar asked Jun 26 '16 08:06

Seyed Vahid Hashemi


1 Answers

Multi-threading python issues are separated from Apache Spark internals. Parallelism on Spark is dealt with inside the JVM.

enter image description here

And the reason is that in the Python driver program, SparkContext uses Py4J to launch a JVM and create a JavaSparkContext.

Py4J is only used on the driver for local communication between the Python and Java SparkContext objects; large data transfers are performed through a different mechanism.

RDD transformations in Python are mapped to transformations on PythonRDD objects in Java. On remote worker machines, PythonRDD objects launch Python sub-processes and communicate with them using pipes, sending the user's code and the data to be processed.

PS: I'm not sure if this actually answers your question completely.

like image 96
eliasah Avatar answered Sep 19 '22 09:09

eliasah