Menu
I have an Akka system written in scala that needs to call out to some Python code, relying on Pandas and Numpy, so I can't just use Jython. I noticed that Spark uses CPython on its worker nodes, so I'm curious how it executes Python code and whether that code exists in some re-usable form.
690
PySpark applications are executed using a standard CPython interpreter in order to support Python modules that use C extensions. We have not tested PySpark with Python 3 or with alternative Python interpreters, such as PyPy or Jython.
Spark comes with an interactive python shell. The PySpark shell is responsible for linking the python API to the spark core and initializing the spark context. bin/PySpark command will launch the Python interpreter to run PySpark application. PySpark can be launched directly from the command line for interactive use.
Standalone PySpark applications should be run using the bin/pyspark script, which automatically configures the Java and Python environment using the settings in conf/spark-env.sh or . cmd . The script automatically adds the bin/pyspark package to the PYTHONPATH .
PySpark architecture is described here https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals.
As @Holden said Spark uses py4j to access Java objects in JVM from the python. But this is only one case - when driver program is written in python (left part of diagram there)
The other case (the right part of the diagram) - when Spark Worker starts Python process and sends serialized Java objects to python program to be processed, and receives output. Java objects are serialized into pickle format - so python could read them.
Looks like what you are looking for is the latter case. Here some links to the Spark's scala core that could be useful for you to get started:
Pyrolite library that provides Java interface to Python's pickle protocols - used by Spark to serialize Java objects into pickle format. For example such conversion is required for accessing Key part of Key, Value pairs for the PairRDD.
Scala code that starts python process and iterates with it: api/python/PythonRDD.scala
SerDeser utils that do picking of the code: api/python/SerDeUtil.scala
Python side: python/pyspark/worker.py
176
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
