How to configure Spark Standalone's web UI for HTTPS?

Question

I'd like to set up Spark Standalone's Web UI so it can be accessed through HTTPS.

Spark is being run on a cluster external to the computer I'm using to access the browser.

Here's what I've done so far, and it's not working:

• Used OpenSSL to make a self-signed certificate and key and then keytool to make a keystore
• Packaged the certificate and key as a .p12 file
• In Chrome's settings under Manage Certificates, imported said p12 file

• In spark-defaults.conf, added options

  spark.ui.https.enabled                   true
  spark.ui.ssl.server.keystore.location    /path/to/spark.keystore
  spark.ui.ssl.server.keystore.keypassword password
  spark.ui.ssl.server.keystore.password    password
  

• In spark-env.sh, added

  export SPARK_MASTER_OPTS="-Dspark.ui.https.enabled=true \
    -Dspark.ui.ssl.server.keystore.location=/path/to/spark.keystore \
    -Dspark.ui.ssl.server.keystore.keypassword=password \
    -Dspark.ui.ssl.server.keystore.password=password"
  
  export SPARK_WORKER_OPTS="-Dspark.ui.https.enabled=true \
    -Dspark.ui.ssl.server.keystore.location=/path/to/spark.keystore \
    -Dspark.ui.ssl.server.keystore.keypassword=password \
    -Dspark.ui.ssl.server.keystore.password=password"
  

I've tried to connect to the server before, after, and in between each of these steps, and I keep getting the error "This site can’t provide a secure connection". What am I missing here?

Answer 1

According to this line I think that it's not possible to set up the Spark Standalone's web UI with HTTPS.

masterWebUiUrl = "http://" + masterPublicAddress + ":" + webUi.boundPort

My recommendation is to file an issue in Spark's JIRA and find a Spark developer to fix it.

Answer 2

Below config worked for me, try putting this in "spark-defaults.conf" and restart the Spark service. Also check logs for which port Spark UI is listening on, as "spark.ssl.ui.port" is set to "0". In my case, it was running on port 8480.

spark.ssl.enabled true
spark.ssl.ui.port 0
spark.ssl.keyStore <path_to_keystore>
spark.ssl.keyStorePassword <keystore_password>
spark.ssl.keyPassword <key_password>
spark.ssl.trustStore <path_to_truststore>
spark.ssl.trustStorePassword <truststore_password>
spark.ssl.enabledAlgorithms ECDHE-RSA-AES256-SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
spark.ssl.protocol TLSv1.2
spark.ssl.trustStoreType JKS

See the screenshot for reference.