Apache Spark - Connection refused for worker

Question

Hi I was new to apache spark and i was trying to learn it

While creating a new standalone cluster I met with this error.

I started my master and it is active in port 7077, i can see that in the ui (port 8080)

While startting the server using the command

 ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://192.168.0.56:7077

I am meeting with a connection refused error

14/07/22 13:18:30 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker@node-        physical:55124] -> [akka.tcp://[email protected]:7077]: Error [Association failed     with [akka.tcp://[email protected]:7077]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:7077]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:     Connection refused: /192.168.0.56:7077

Please help me with the error i am sruck here for a long time.

I hope the information is enough. Please help

Answer 1

In my case, I went to /etc/hosts and :

• removed the line with 127.0.1.1 and it worked.
• wrote "MASTER_IP MACHINE_NAME"

Answer 2

Try "./sbin/start-master -h ". It works, when I specify the host name as IP address.

Answer 3

Change the SPARK_MASTER_HOST=< ip> in the spark-env.sh of the master node.

Then restart the master, if you grep the process you will see it changes from

java -cp /spark/conf/:/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host < HOST NAME> --port 7077 --webui-port 8080

to

java -cp /spark/conf/:/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host < HOST IP> --port 7077 --webui-port 8080