Starting Ipython with Spark 2

Question

I have my ipthon startup script as follows

IPYTHON_OPTS="notebook --port 8889 \
--notebook-dir='/usr/hdp/2.3.2.0-2950/spark/' \
--ip='*' --no-browser" pyspark

It runs fine for older Spark. But when I switch to Spark2, I get the following error:

IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+. Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS instead.

It is not clear what command I should replace it with. Can someone help?

Answer 1

For Spark < 2.0 the command to start pyspark with IPython was:

IPYTHON=1 pyspark

The equivalent updated command for Spark >= 2.0 is:

PYSPARK_DRIVER_PYTHON=ipython pyspark

according to Spark's source code.