How does Apache Spark support different language APIs

Question

I have been wondering the high level architecture of a system like Apache Spark which supports different language APIs. For example Spark supports APIs in Scala, Python and Java.

Although the software is written in Scala, it is supporting multiple language APIs. How is it done in a programmatic level? Can someone explain the high level architecture of a system like this?

Answer 1

Spark uses an RPC server to expose API to other languages. When you look at the source code you'll find that all objects from PySpark and SparkR are in fact JVM objects wrappers. Take a look at R DataFrame and Python DataFrame. In R, there is a lot of callJMethod() calls (which are calling JVM objects via some kind of proxy). In Python, each object has a handle to JVM object. In case of DataFrame it is _jdf (Java DataFrame). Under the hood, there is Spark (Scala implementation) running and there is an RPC server running which exposes JVM objects (i.e. SparkContext) to external processes (PySpark and SparkR in this case).

The question is, how to implement map()/mapPartitions() methods which take code in Python or R? As you might know the map() method creates an RDD which remembers the transformation. To support Python, in Scala Spark there is a special kind of an RDD called PythonRDD.

Also, there is a document about PySpark internals, if you're interested, of course.