Docker containers unable to connect to internet

Question

I have a docker compose file that starts up few containers including prometheus, alertmanager and grafana. These containers are not able to connect to internet. I have tried multiple solutions but to no avail. I am on a digitalocean ubuntu droplet.

My docker-compose file:

version: '3'

services:

    prometheus:
      image: prom/prometheus:v2.20.1
      container_name: prometheus
      ports:
        - 9090:9090
      volumes:
        - /data/prometheus:/prometheus
        - ./prometheus/:/etc/prometheus/
      restart: always
    
    alertmanager:
      image: prom/alertmanager:v0.21.0
      container_name: alertmanager
      ports:
        - 9093:9093
        - 6783:6783
      command:
        - '--log.level=debug'
        - '--config.file=/etc/alertmanager/alertmanager_config.yml'
        - '--storage.path=/alertmanager'
      volumes:
        - ./alertmanager:/etc/alertmanager
        - /data/alertmanager:/alertmanager
      restart: always


    grafana:
      image: grafana/grafana:7.1.5
      container_name: grafana
      ports:
        - 3000:3000
      volumes:
        - ./grafana.ini:/etc/grafana/grafana.ini
      restart: always

I have tried multiple things

• Installed resolvconf and restarted docker service docker restart
• Changed /etc/resolv.conf on host machine to point to google or openDNS servers.
• Added DNS in /etc/docker/daemon.json and restarted docker

{
    "dns" : ["172.24.100.50", "8.8.8.8"]
}

• Changed DNS nameserver inside the containers from

nameserver 127.0.0.11
options ndots:0

to

nameserver 127.0.0.11
nameserver 172.24.100.50
nameserver 8.8.8.8

Commands run inside the container

/alertmanager $ wget http://curl.haxx.se/download/curl-7.36.0.tar.gz
wget: bad address 'curl.haxx.se'
/alertmanager $ nslookup google.com
;; connection timed out; no servers could be reached

/alertmanager $ 

While sending alerts, alertmanager gives error:

lookup api.<my website>.com on 172.24.100.50:53: read udp 172.18.0.5:44178->172.24.100.50:53: i/o timeout"

I tried to run alertmanager on host network and it still doesn't work

docker run --net host -d prom/alertmanager:v0.21.0
docker exec -it <container_id> sh

/alertmanager $ cat /etc/resolv.conf 
nameserver 172.24.100.50
nameserver 8.8.8.8
/alertmanager $ ls
/alertmanager $ wget http://curl.haxx.se/download/curl-7.36.0.tar.gz
wget: bad address 'curl.haxx.se'
/alertmanager $ set vc
/alertmanager $ nslookup google.com
;; connection timed out; no servers could be reached

I have tried many options but haven't found the solution yet. Anyone who can help me with this? let me know if more details are required.

Answer 1

I was able to solve the issue. It turns out in the digitalocean firewall for the droplet, the outbound traffic were blocked for UDP. Only TCP traffic was allowed. And hence the dns resolution was not working.

DNS uses TCP for Zone transfer and UDP for name queries either regular (primary) or reverse. UDP can be used to exchange small information whereas TCP must be used to exchange information larger than 512 bytes. DNS requires port 53 for name resolution and from the docker logs it can be seen port 53 is being used but since udp outbound traffic were blocked, dns was not working.

However, I did try to force docker to use TCP by setting dns_opt=use-vc setting. This didn't work. UDP traffic was allowed and now it is working.