cannot start spark history server

Question

I am running spark on yarn cluster. I tried to start the history server

./start-history-server.sh

but got the following errors.

starting org.apache.spark.deploy.history.HistoryServer, logging to /home/abc/spark/spark-1.5.1-bin-hadoop2.6/sbin/../logs/spark-abc-org.apache.spark.deploy.history.HistoryServer-1-abc-Efg.out
failed to launch org.apache.spark.deploy.history.HistoryServer:
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:47)
... 6 more
full log in /home/abc/spark/spark-1.5.1-bin-hadoop2.6/sbin/../logs/spark-abc-org.apache.spark.deploy.history.HistoryServer-1-abc-Efg.out

I have set spark.eventLog.enabled = true and spark.history.fs.logDirectory and spark.eventLog.dir to the hdfs logging directory.

Why can't I get the history server to start?

Update 1:

Thank you stf for telling me to look at the log file; I didn't know it exists!

I realise my problem is in my setting in spark-env.sh

 export SPARK_HISTORY_OPTS="-Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs:///localhost/eventLogging spark.history.fs.logDirectory=hdfs:///localhost/eventLogging"

The forward slash becomes dot

 Error: Could not find or load main class spark.history.fs.logDirectory=hdfs:...localhost.eventLogging

Any idea how to prevent this from happening?

Update 2: Solved this problem thanks to stf's help. Correct setting in spark-env.sh is

 SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.eventLog.enabled=true -Dspark.eventLog.dir=hdfs://localhost/eventLogging -Dspark.history.fs.logDirectory=hdfs://localhost/eventLogging"

Answer 1

For those still getting this error and not helped by the comment discussion. The following resolved this issue for me. Make sure that you have the following in in spark/conf/spark-defaults.conf

spark.eventLog.enabled          true
spark.eventLog.dir              /path/to/spark/logs
spark.history.fs.logDirectory   /path/to/spark/logs

Then run spark/sbin/start-history-server.sh /path/to/spark/logs