Zookeeper connection error

Question

We have a standalone zookeeper setup on a dev machine. It works fine for every other dev machine except this one testdev machine.

We get this error over and over again when trying to connect to zookeeper through testdev:

2012-11-09 14:06:53,909 - INFO  [main-SendThread(zk01.dev.bunchball.net:2181):ClientCnxn$SendThread@947] - Socket connection established to zk01.dev.bunchball.net/192.168.8.58:2181, initiating session 2012-11-09 14:06:53,911 - INFO  [main-SendThread(zk01.dev.bunchball.net:2181):ClientCnxn$SendThread@1183] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2012-11-09 14:06:55,366 - INFO  [main-SendThread(zk01.dev.bunchball.net:2181):ClientCnxn$SendThread@1058] - Opening socket connection to server zk01.dev.bunchball.net/192.168.8.58:2181 2012-11-09 14:06:55,368 - INFO  [main-SendThread(zk01.dev.bunchball.net:2181):ClientCnxn$SendThread@947] - Socket connection established to zk01.dev.bunchball.net/192.168.8.58:2181, initiating session 2012-11-09 14:06:55,368 - INFO  [main-SendThread(zk01.dev.bunchball.net:2181):ClientCnxn$SendThread@1183] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2012-11-09 14:06:57,271 - INFO  [main-SendThread(zk01.dev.bunchball.net:2181):ClientCnxn$SendThread@1058] - Opening socket connection to server zk01.dev.bunchball.net/192.168.8.58:2181 2012-11-09 14:06:57,274 - INFO  [main-SendThread(zk01.dev.bunchball.net:2181):ClientCnxn$SendThread@947] - Socket connection established to zk01.dev.bunchball.net/192.168.8.58:2181, initiating session 2012-11-09 14:06:57,275 - INFO  [main-SendThread(zk01.dev.bunchball.net:2181):ClientCnxn$SendThread@1183] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 

We tried restarting the test dev machine, and also restart zookeeper host but nothing worked. We are totally confused why it works perfectly fine with other machines except this one. What might be the cause of this?

Answer 1

I just have the same situation as you and I have just fixed this problem.

It is the reason that you have configured even number of zookeepers which directly result in this problem,try to change your number of zookeeper node to a odd one.

for example the original status of my zookeeper cluster is consisted of 4 nodes,then just remove one of them which result in the number of node to be 3 well ,now its ok to startup zookeeper cluster

below is the output of successfully connect to zookeeper server

2013-04-22 22:07:05,654 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@1321ed6 Welcome to ZooKeeper! 2013-04-22 22:07:05,704 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@966] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled 2013-04-22 22:07:05,727 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@849] - Socket connection established to localhost/127.0.0.1:2181, initiating session [zk: localhost:2181(CONNECTING) 0] 2013-04-22 22:07:05,846 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1207] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13e3211c06e0000, negotiated timeout = 30000 

Answer 2

I faced the same issue and found it was due to zookeeper cluster nodes needs ports opened to communicate with each other.

server.1=xx.xx.xx.xx:2888:3888  server.2=xx.xx.xx.xx:2888:3888  server.3=xx.xx.xx.xx:2888:3888 

once i allowed these ports through aws security group and restarted. All worked fine for me