Log client's "real" IP address in Docker Swarm 1.12 when accessing a service

Question

I have nginx container running as a service in Docker Swarm inside user created overlay network. Both created with:

docker network create --driver overlay proxy
docker service create --name proxy --network proxy -p 80:80 nginx

When accessing nginx site through a browser, in nginx access log remote address is logged as 10.255... formatted address, what I presume to be the Swarm load balancer address. The question is how to know/log the address of the end client accessing the site and not the load balancer address.

Answer 1

So the way I've worked around this for now is by using the mode=host option for publishing the port and pinning the proxy container to a single node like so:

docker service create \
--name proxy \
--network proxy \
--publish mode=host,target=80,published=80 \
--publish mode=host,target=443,published=443 \
--constraint 'node.hostname == myproxynode' \
--replicas 1 \
letsnginx

Answer 2

Good catch!, Most people analyzing the nginx access.log and client ip is important part of it.

As docker version 1.12.1 the problem exists. nginx will log swarm overlay ip. But client ip logs fine as standalone container. As a work around, you can have a reverse proxy pointing to swarm service. I know this is against High availablity and Self Healing concept of swarm, but seems to be the only work around right now.

sample config: (lets assume swarm service is listening on 8081 on localhost)

server {
  listen 80 default_server;
  location / {
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_pass          http://localhost:8181;
    proxy_read_timeout  90;
  }
}

More info can be found on this github issue.

Another Option:

You can use networking in host mode.

docker service create \
--name nginx \
--network <your overlay network> \
--publish mode=host,target=80,published=80 \
--publish mode=host,target=443,published=443 \
--replicas 1 \
nginx

Answer 3

We had to solve the issue of identifying clients real IPs for our production deployment of Docker Swarm running https://www.newsnow.co.uk/ (mode=host was not an option as we needed to run multiple replicas across multiple nodes).

To solve it, we created: https://github.com/newsnowlabs/docker-ingress-routing-daemon, a daemon that modifies the ingress mesh network routing to expose true client IPs to service containers.