Hazelcast in multinode docker environments with TCPIP

Question

I am currently struggling with the following problem:

I have a Java EE application that shall be able to run in a cluster (it actually does). For the data exchange between the nodes I use Hazelcast 3.3.3. Since our customers are afraid of UDP we use TCP. So I configure 3 hazelcast parameters per node: The network interface to use, the port and the members of the cluster. In real life (not a docker environment) everything works. Now I want to put my Java EE servers into docker containers (one per node) and also link them via Hazelcast.

What I did was:

On the host machine running the docker containers I created several virtual network interfaces (172.21.0.X). for each docker container I gave network mappings like -p 172.21.0.X:5190:5190 (5190 is the port I use for Hazelcast)

In my Java EE application within the container I configured hazelcast to listen to interface 172.21.0.X, port 5190 members 172.21.0.X:5190,...... The problem now is: docker will assign an internal IP number to each cointainer. In advance I cannot be sure which IP number docker will assign. It definitly is not the IP I would like to use (172.21.0.X). Hazelcast starts up and tells me: no matching interface found. Of course not. Hazelcast will check the IP interfaces of the docker container which is something like 172.4.XX. This does not fit to the config I gave to hazelcast (use 172.21.X.X).

So far it seems not to be possible to specify the internal IP number of a docker container. I could query it from a running container, but that is too late since I need to do the configuration of my Java EE server BEFORE I start the container.

Any hints on that? Is it possible to get the Hazelcast running with TCPIP?

Answer 1

This thread provides an answer and an example: https://groups.google.com/d/msg/orient-database/ehfKcInWZXs/wxs8xLUOGB8J

Basically you use hazelcast variables in your hazelcast file, i.e.

<property name="hazelcast.local.localAddress">${hazelcast.ip}</property>

Then when you're running the docker contained you add:

-e IP=`ip route | awk '/default/ { print  $9 }'`

Which sets an environment variable in the docker container. Then in there you do something like this:

CMD ["java", "-Dhazelcast.ip=${IP}", ...

So it's taking the docker env variable, and smashing into java as a definition, which hazelcast then picks out and substitutes into it's config file at runtime.

Answer 2

I want to make @gremwell's answer up-to-date, because now it's not quite correct. Property hazelcast.local.localAddress is no longer valid. Instead one should put this snippet inside hazelcast.xml

   <network>
        <public-address>${hazelcast.ip}:${hazelcast.port}</public-address>
        <port auto-increment="false" port-count="100">${hazelcast.port}</port>
        ...
   </network>

Then pass the params while starting docker container (I use official hazelcast image and mount folder with my config using -v flag)

sudo docker run -p 5701:5701 -e JAVA_OPTS="-Dhazelcast.config=/etc/conf/hazelcast.xml -Dhazelcast.ip=`ip route get 8.8.8.8 | awk '{print $NF; exit}'` -Dhazelcast.port=5701" -v /data/docker/hazelcast:/etc/conf -ti hazelcast/hazelcast

Command -Dhazelcast.ip=ip route get 8.8.8.8 | awk '{print $NF; exit}' takes host machine ip and pass it inside config file as hazelcast.ip variable. Inner port and docker port mapping should be similar 5701 in 3 places. Next container run with 5702 and so on.

Hope this helps.