How can I get Zeppelin to restart cleanly on an EMR cluster?

Question

I am running an EMR cluster and trying to use a Zeppelin notebook for data analysis.

Versions: Release label:emr-5.2.1
Hadoop distribution: Amazon 2.7.3
Hive 2.1.0
Spark 2.0.2
Zeppelin 0.6.2

I am consistently having problems with Zeppelin hanging up while running a query and I can never recover it. I have tried:
- restarting the interpreter
- SSH'ing into the master node and running zeppelin_daemon.sh restart (have tried running as hadoop / root / zeppelin, and also running the script with options reload, start/stop, upstart)

Every time I use the daemon shell script, it tells me that it has stopped/started OK, but then I get this when I run a status:
Zeppelin running but process is dead [FAILED]

The only thing that I can seem to do is kill my cluster and stand up a new one, which is crazy on many levels.

So...what is the officially supported way to restart Zeppelin???

Answer 1

Have you tried:

sudo /usr/lib/zeppelin/bin/zeppelin-daemon.sh stop/start?

Answer 2

I ssh'ed into the master node, then ran

$ sudo stop zeppelin
zeppelin stop/waiting

$ sudo start zeppelin
zeppelin start/running, process 24434

Immediately after, refreshing Zeppelin gave me a 503, but after ~2 minutes or so, the UI was available again. I checked the logs in /var/log/zeppelin and it takes some time to configure settings.

Answer 3

Services on EMR use upstart, and the supported way to restart them is to use "sudo stop <service-name>; sudo start <service-name>". (The start and stop commands are in /sbin, which is in the PATH by default.)

Note: currently on EMR there is a limitation that prevents /sbin/restart from working properly, so you must use stop then start instead of restart.