Why does pyspark give "we couldn't find any external IP address" on macOS?

Question

I use pyspark and got a warning below. Could someone tell me how to fix it? Is this something I should be worried about?

Code:

lines = sc.textFile("README.md") #worked
lines.count() #error 

Warning:

16/02/24 08:20:39 WARN : Your hostname, a.local resolves to a loopback/non-reachable address: fe80:0:0:0:f09c:b1ff:fef2:170c%awdl0, but we couldn't find any external IP address!

Answer 1

Spark resolves your hostname to an address to establish communication channels between nodes (the driver and executors) for a Spark application.

It looks like your machine has various names (possibly in /etc/hosts) and some of them are not resolved which Spark can use.

You can explicitly set the local hostname to localhost (which is probably resolvable) for spark-shell using SPARK_LOCAL_HOSTNAME environment variable as follows:

SPARK_LOCAL_HOSTNAME=localhost ./bin/spark-shell

Or:

./bin/spark-shell -c spark.driver.host=localhost

Refer to Environment Variables in the documentation for more info.

Answer 2

Add local path in $SPARK_HOME/conf/spark-env.sh:

SPARK_LOCAL_IP=127.0.0.1