Docker-swarm overlay network is not working for containers in different hosts

Question

We have a networking problem in docker-swarm. The problem is below;

• we have virtualized environment over wmware ( vsphere 6.02)
• our servers are created from vmware say server1 and server2
• we have a docker compose file defining a couple of services
• we have an overlay-network definition within docker-compose for docker-swarm
• when we deploy system using docker-swarm deployment is finished successfully, all containers gets ip from overlay network range.
• But the problem is if 2 containers (say cnt1 and cnt2) are deployed to different servers they can not ping each other
• I check tcpdump and see that ARP communication is successfull so they know each other mac correctly
• But when you try to ping to container, ICMP Echo messages are send but are not delivered to second machine..

Where should I check, any advices?

    server-1:~$ docker version
    Client:
     Version:      17.03.0-ce
     API version:  1.26
     Go version:   go1.7.5
     Git commit:   3a232c8
     Built:        Tue Feb 28 08:01:32 2017
     OS/Arch:      linux/amd64

    Server:
     Version:      17.03.0-ce
     API version:  1.26 (minimum version 1.12)
     Go version:   go1.7.5
     Git commit:   3a232c8
     Built:        Tue Feb 28 08:01:32 2017
     OS/Arch:      linux/amd64
     Experimental: true

ps: I checked this post but I have latest version of docker / docker-swarm so the issue should be fixed..

ps-2: similar problem; https://github.com/docker/swarm/issues/2687

Answer 1

"VTEP Port is reserved or restricted for VMware use, any virtual machine cannot use this port for other purpose or for any other application."

But we can change docker swarm data-path-port(the default port number 4789 is used) to another:

docker swarm init --data-path-port=7789