Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Which of the many Spark/Scala kernels for Jupyter/IPython to choose? [closed]

There are a lot of Scala/Spark kernels for IPython/Jupyter:

  1. IScala
  2. ISpark
  3. Jupyter Scala
  4. Apache Toree(prev Spark Kernel)

Does anybody know wich of them is most compatible with IPython/Jupyter and most comfortable to use with:

  1. Scala
  2. Spark(Scala)
like image 499
Lunigorn Avatar asked Sep 30 '15 05:09

Lunigorn


People also ask

What are IPython kernels?

A 'kernel' is a program that runs and introspects the user's code. IPython includes a kernel for Python code, and people have written kernels for several other languages. When IPython starts a kernel, it passes it a connection file. This specifies how to set up communications with the frontend.

What is a spark kernel?

spark-kernel (homepage) The Spark Kernel enables remote applications to dynamically interact with Apache Spark. It serves as a remote Spark Shell that uses the IPython message protocol to provide a common entrypoint for applications (including IPython itself).

Can I use Scala in Jupyter notebook?

Jupyter notebook is widely used by almost everyone in the data science community. While it's a tool with extensive support for python-based development of machine learning projects, one can also use it for Scala development as well, using the spylon-kernel.


2 Answers

I can't speak for all of them, but I use Spark Kernel and it works very well for using both Scala and Spark.

I found IScala and Jupyter Scala less stable and less polished. Jupyter Scala always prints every variable value after I execute a cell; I don't want to see this 99% of the time.

Spark Kernel is my favourite. Both for Spark and plain old Scala.

like image 140
Al M Avatar answered Sep 22 '22 18:09

Al M


Spark Kernel has been accepted into Apache Incubator and has moved all development to Apache Toree.

like image 21
artyomboyko Avatar answered Sep 21 '22 18:09

artyomboyko