Zookeeper unable to listen on port 3888

Question

I've got 3 servers on aws. each with open jdk 7 and zookeeper 3.4.6 all have unique elastic ip's. each conf/zoo.cfg has

• clientPort=2181
• server.1=server1:2888:3888
• server.2=server2:2888:3888
• server.3=server3:2888:3888

then i start it with ./zkServer.sh start (says STARTED) and the zookeeper.out says

2015-01-14 09:27:55,919 [myid:1] - INFO  [Thread-1:QuorumCnxManager$Listener@504] - My election bind port: /server1ipaddress:3888
2015-01-14 09:27:55,920 [myid:1] - ERROR [/server1ipaddress:3888:QuorumCnxManager$Listener@517] - Exception while listening
    java.net.BindException: Cannot assign requested address
            at java.net.PlainSocketImpl.socketBind(Native Method)
            at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
            at java.net.ServerSocket.bind(ServerSocket.java:376)
            at java.net.ServerSocket.bind(ServerSocket.java:330)
            at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:507)

So it cant open the port. i've eventually opened all ports on aws security to rule that out.

• telnet into 2181 with ruok gets imok.
• telnet into 2888 cannot connect. connection refused.
• telnet into 3888 cannot connect. connection refused.

netstat shows that nothing is blocking 2888 and 3888

i've even tried this with all 3 servers having zookeeper started. whats going on? how do i get those ports open for use.

Answer 1

Cross verify myid's on all the nodes based on the zoo.cfg. The same issue happened to me, upon looking myid pattern got changed on 2 of the nodes.

Answer 2

Your problem is answered here.

In a few words: on each ZooKeeper machine, at your conf/zoo.cfg, you have to set the current server's IP to 0.0.0.0. For example: if you are currently on server1, the config should contain the following lines:

server.1=0.0.0.0:2888:3888
server.2=server2:2888:3888
server.3=server3:2888:3888

This step solved the problem in my case.