scala spark notebook inside IntelliJ

Question

I use andypetrella/spark-notebook, and it's great:

docker run -p 9001:9001 andypetrella/spark-notebook:0.7.0-scala-2.11.8-spark-2.1.0-hadoop-2.7.3

However, I would like to use such a notebook from inside InteliJ to use completion and debugging. How to achieve this?

I see the IntelliJ you can have New -> jupyter, but I think it is only python. Or does it work on scala and spark?

I also see that IntelliJ has a Scala worksheet. With its experimental REPL, IntelliJ evaluates incrementally the new lines added, but it evaluates all the file from the beginning if you change a line in the middle of the file. So, not very convinient compared to a jupyter notebook.

I see other questions about running scala spark notebook inside IntelliJ, such as this one, but they do not answer the question here. How to run Spark scala application inside Intellij

The question is how to have the equivalent of andypetrella/spark-notebook from inside IntelliJ (so, only evaluate the lines changed)?

Answer 1

As for spark notebook in Intellij I am afraid you are out of luck. If you want any notebook with Spark in Intellij then you can test Apache Zeppelin notebook. The plugin works but is outdated.

docker run --rm -p 8080:8080 dylanmei/zeppelin

• In Intellij Plugins->Browse repositories-> search IntelliJ Zeppelin
• Create an empty scala class and inside it type 'NotebookName'
• Hit ctrl+shift+alt+enter (shift+option+cmd+enter) (see tools-> Zeppelin in the intellij menu for shortcuts)- this should give you a window asking for the address of zepellin
• Type the code for the paragraph
• shift+alt+enter (shift+option+enter) to add the paragraph. This should give you the result of the paragraph
• Navigate to http://localhost:8080 (or whatever port you gave it) and look for 'NotebookName' (if you do not give it a name it will spawn as 'IntelliJ Notebook')

There is a demo of this on the plugins github: https://github.com/chilang/intellij-zeppelin

If you try to use this plugin with spark-notebook it will yield a nasty error.