Hiding a docker container behind OpenVPN, in docker swarm, with an overlay network

Question

The goal: To deploy on docker swarm a set of services, one of which is only available for me when I am connected to the OpenVPN server which has also been spun up on docker swarm.

How can I, step by step, only connect to a whoami example container, with a domain in the browser, when I am connected to a VPN?

Background

The general idea would be have, say, kibana and elasticsearch running internally which can only be accessed when on the VPN (rather like a corporate network), with other services running perfectly fine publicly as normal. These will all be on separate nodes, so I am using an overlay network.

I do indeed have OpenVPN running on docker swarm along with a whoami container, and I can connect to the VPN, however it doesn't look like the IP is changing and I have no idea how to make it so that the whoami container is only available when on the VPN, especially considering I'm using an overlay network which is multi-host. I'm also using traefik, a reverse proxy which provides me with a mostly automatic letsencrypt setup (via DNS challenge) for wildcard domains. With this I can get:

https://traefik.mydomain.com

But I also want to connect to vpn.mydomain.com (which I can do right now), and then be able to visit:

https://whoami.mydomain.com

...which I cannot. Yet. I've posted my traefik configuration in a different place in case you want to take a look, as this thread will grow too big if I post it here.

Let's start with where I am right now.

OpenVPN

Firstly, the interesting thing about OpenVPN and docker swarm is that OpenVPN needs to run in privileged mode because it has to make network interfaces changes amongst other things, and swarm doesn't have CAP_ADD capabilities yet. So the idea is to launch the container via a sort of 'proxy container' that will run the container manually with these privileges added for you. It's a workaround for now, but it means you can deploy the service with swarm.

Here's my docker-compose for OpenVPN:

    vpn-udp:
        image: ixdotai/swarm-launcher:latest
        hostname: mainnode
        environment:
            LAUNCH_IMAGE: ixdotai/openvpn:latest
            LAUNCH_PULL: 'true'
            LAUNCH_EXT_NETWORKS: 'app-net'
            LAUNCH_PROJECT_NAME: 'vpn'
            LAUNCH_SERVICE_NAME: 'vpn-udp'
            LAUNCH_CAP_ADD: 'NET_ADMIN'
            LAUNCH_PRIVILEGED: 'true'
            LAUNCH_ENVIRONMENTS: 'OVPN_NATDEVICE=eth1'
            LAUNCH_VOLUMES: '/etc/openvpn:/etc/openvpn:rw'
        volumes:
            - '/var/run/docker.sock:/var/run/docker.sock:rw'
        networks:
            - my-net
        deploy:
            placement:
                constraints:
                    - node.hostname==mainnode

I can deploy the above with: docker stack deploy --with-registry-auth --compose-file docker/docker-compose.prod.yml my-app-name and this is what I'm using for the rest. Importantly I cannot just deploy this as it won't load yet. OpenVPN configuration needs to exist in /etc/openvpn on the node, which is then mounted in the container, and I do this during provisioning:

// Note that you have to create the overlay network with --attachable for standalone containers
docker network create -d overlay app-net --attachable

// Create the config
docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn ovpn_genconfig -u udp://vpn.mydomain.com:1194 -b

// Generate all the vpn files, setup etc
docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn bash -c 'yes yes | EASYRSA_REQ_CN=vpn.mydomain.com ovpn_initpki nopass'

// Setup a client config and grab the .ovpn file used for connecting
docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn easyrsa build-client-full client nopass

docker run -v /etc/openvpn:/etc/openvpn --log-driver=none --rm ixdotai/openvpn ovpn_getclient client > client.ovpn

So now, I have an attachable overlay network, and when I deploy this, OpenVPN is up and running on the first node. I can grab a copy of client.ovpn and connect to the VPN. Even if I check "send all traffic through the VPN" though, it looks like the IP isn't being changed, and I'm still nowhere near hiding a container behind it.

---

Whoami

This simple container can be deployed with the following in docker-compose:

    whoami:
        image: "containous/whoami"
        hostname: mainnode
        networks:
            - ${DOCKER_NETWORK_NAME}
        ports:
            - 1337:80
        deploy:
            placement:
                constraints:
                    - node.hostname==mainnode

I put port 1337 there for testing, as I can visit my IP:1337 and see it, but this doesn't achieve my goal of having whoami.mydomain.com only resolving when connected to OpenVPN.

---

I can ping a 192.168 address when connected to the vpn

I ran the following on the host node:

ip -4 address add 192.168.146.16/24 dev eth0

Then when connected to the VPN, I can resolve this address! So it looks like something is working at least.

How can I achieve the goal stated at the top? What is required? What OpenVPN configuration needs to exist, what network configuration, and what container configuration? Do I need a custom DNS solution as I suggest below? What better alternatives are there?

---

Some considerations:

• I can have the domains, including the private one whoami.mydomain.com public. This means I would have https and get wildcard certificates for them easily, I suppose? But my confusion here is - how can I get those domains only on the VPN but also have tls certs for them without using a self-signed certificate?

• I can also run my own DNS server for resolving. I have tried this but I just couldn't get it working, probably because the VPN part isn't working properly yet. I found dnsmasq for this and I had to add the aforementioned local ip to resolve.conf to get anything working locally for this. But domains would still not resolve when connected to the VPN, so it doesn't look like DNS traffic was going over the VPN either (even though I set it as such - my client is viscosity.

• Some mention using a bridge network, but a bridge network does not work for multi-host

Resources thus far (I will update with more)

- Using swarm-launcher to deploy OpenVPN

- A completely non-explanatory answer on stackexchange which I have seen referenced as basically unhelpful by multiple people across other Github threads, and one of the links is dead

Answer 1

So I was banging my head head against a brick wall about this problem and just sort of "solved" it by pivoting your idea:

Basically I opened the port of the vpn container to its host. And then enable a proxy. This means that I can reach that proxy by visiting the ip of the pc in which the vpn resides (AKA the Docker Host of the VPN container/stack).

Hang with me:

I used gluetun vpn but I think this applies also if you use openvpn one. I just find gluetun easier.

Also IMPORTANT NOTE: I tried this in a localhost environment, but theoretically this should work also in a multi-host situation since I'm working with separated stacks. Probably, in a multi-host situation you need to use the public ip of the main docker host.

1. Create the network

So, first of all you create an attachable network for this docker swarm stacks:

docker network create --driver overlay --attachable --scope swarm vpn-proxy

By the way, I'm starting to think that this passage is superfluous but need to test it more.

2. Set the vpn stack

Then you create your vpn stack file, lets call it stack-vpn.yml:

(here I used gluetun through swarm-launcher "trick". This gluetun service connects through a VPN via Wireguard. And it also enables an http proxy at the port 8888 - this port is also mapped to its host by setting LAUNCH_PORTS: '8888:8888/tcp')

version: '3.7'

services:
  vpn_launcher:
    image: registry.gitlab.com/ix.ai/swarm-launcher
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock:rw'
    networks:
      - vpn-proxy
    environment:
      LAUNCH_IMAGE: qmcgaw/gluetun
      LAUNCH_PULL: 'true'
      LAUNCH_EXT_NETWORKS: 'vpn-proxy'
      LAUNCH_PROJECT_NAME: 'vpn'
      LAUNCH_SERVICE_NAME: 'vpn-gluetun'
      LAUNCH_CAP_ADD: 'NET_ADMIN'
      LAUNCH_ENVIRONMENTS: 'VPNSP=<your-vpn-service> VPN_TYPE=wireguard WIREGUARD_PRIVATE_KEY=<your-private-key> WIREGUARD_PRESHARED_KEY=<your-preshared-key> WIREGUARD_ADDRESS=<addrs> HTTPPROXY=on HTTPPROXY_LOG=on'
      LAUNCH_PORTS: '8888:8888/tcp'
    deploy:
      placement:
        constraints: [ node.role == manager ]
      restart_policy:
        condition: on-failure

networks:
  vpn-proxy:
    external: true

Notice that either the swarm-launcher and the gluetun containers are using the network previously created vpn-proxy.

3. Set the workers stack

For the time being we will set an example with 3 replicas of alpine image here (filename stack-workers.yml):

version: '3.7'

services:
  alpine:
    image: alpine
    networks:
      - vpn-proxy
    command: 'ping 8.8.8.8'
    deploy:
      replicas: 3

networks:
  vpn-proxy:
    external: true

They also use the vpn-proxy overlay network.

4. Launch our stacks

docker stack deploy -c stack-vpn.yml vpn
docker stack deploy -c stack-workers workers

Once they are up you can access any worker task and try to use the proxy by using the host ip where the proxy resides.

As I said before, theoretically this should work on a multi-host situation, but probably you need to use the public ip of the main docker host (although if they share the same overlay network it could also work with the internal ip address (192...) ).