docker nginx proxy nginx connect() failed (111: Connection refused) while connecting to upstream

Question

I'm trying to run a nginx container as the main entry point for all of my websites and web services. I managed to run a portainer as a container, and I'm able to reach it from the internet. Right now I'm trying to reach a static website hosted by another nginx container, but I fail doing so - when I go to the URL, I get

502 Bad Gateway

I've tried adding the upstream section to my main nginx's config, but nothing changed (after every config change, I reload my main nginx service inside the container).

On the other hand, adding upstream is something I'd like to avoid if it's possible because spawning multiple different applications would require adding an upstream for each application - and that's much more work than I'd expect.

Here is my main nginx's configuration file:

events {
    worker_connections 1024;
}
http {
    server {
        listen 80;

        location /portainer/ {
            proxy_pass http://portainer:9000/;
        }
        location /helicon/ {
            proxy_pass http://helicon:8001/;
        }
    }
}

Here is how I start my main nginx container:

docker run -p 80:80 --name nginx -v /var/nginx/conf:/etc/nginx:ro --net=internal-net -d nginx

Here is my static website's nginx configuration file:

events {
    worker_connections 1024;
}
http {
    server {
        listen  80;

    server_name helicon;
        root /var/www/html/helicon;

        error_log /var/log/nginx/localhost.error.log;
        access_log /var/log/nginx/localhost.access.log;
    }
}

Here is how the docker-compose file to create and start that container:

version: '3.5'
services:
    helicon:
        build: .
        image: helicon
        ports:
            - "127.0.0.1:8001:80"
        container_name: helicon
        networks:
            - internal-net
networks:
    internal-net:
        external: true

I'm using internal-net network to keep all apps in the same network instead of deprecated --link option for docker run

When I go to http://my.server.ip.address/helicon I get 502. Then I check logs with docker logs nginx and there is an information

2018/06/24 11:15:28 [error] 848#848: *466 connect() failed (111: Connection refused) while connecting to upstream, client: Y.Y.Y.Y, server: , request: "GET /helicon/ HTTP/1.1", upstream: "http://172.18.0.2:8001/", host: "X.X.X.X"

The helicon container indeed has an IP address of 172.18.0.2.

What am I missing? Maybe my approach should be completely different from using networks?

Kind regards,

Daniel

Answer 1

To anyone coming across this page here is my little contribution to your understanding of docker networking. I would like to illustrate with an example scenario. We are running several contianers with docker-compose such as the following:

1. Docker container client
2. Docker container nginx reverse proxy
3. Docker container service1
4. Docker container service2 etc ...

to make sure you are setup correctly, make sure of the following:

1. All containers are on same network!

• first run: "docker network ls" to find your network name for your stack
• secondly run: "docker network inspect [your_stack_network_name]"

2. note that the ports you expose in docker-compose have nothing to do with nginx reverse proxying! that means that any ports you exposed in your docker-compose file are available on your actual host machine i.e your latptop or pc via your browser BUT for proxying purposes you must point nginx to actual ports of your services.

A walkthrough: lets say service1 runs inside container 1 on port 3000, you mapped port 8080 in your docker-compose file like so: "8080:3000", with this configuration on your local machine you can access the container via your browser on port 8080 (localhost:8080) BUT for nginx reverse proxy container, when trying to proxy to service1, port 8080 is not relevant! the dockerized nginx reverse-proxy will use docker DNS to map service1 to its ip within then docker network and look at it entirely independently from your local host. to nginx reverse proxy inside docker network, service1 only listens on port 3000!!! so make sure to point nginx to the correct port!

Answer 2

Solved. I was working on this for hours thinking it was a nginx config issue. I modified nginx.conf endlessly but couldn't fix it. I was getting 502 Bad Gateway and the error description was:

failed (111: Connection refused) while connecting to upstream

I was looking in the wrong place. It turns out that the http server in my index.js file was listening on the url 'localhost'.

httpServer.listen(PORT, 'localhost', async err => {

This works fine on your development machine, but when running inside a container it must be named the url of the container itself. My containers are networked, and in my case the container is named 'backend'

I changed the url from 'localhost' to 'backend' and everything works fine.

httpServer.listen(PORT, 'backend', async err => {

Answer 3

I was getting the same error. In docker-compose.yml my service port was mapped to a different port (e.g. 1234:8080) and I was using the mapped port number (1234) inside nginx.conf.

However, inside the docker network, the containers do not use their mapped port numbers. In order to solve this problem, I changed the proxy_pass statement to use the correct port number (8080).

To make it clear the working configuration is like this: (Check the used port number in nginx.conf!)

docker-compose.yml

version: '3.8'

services:

...

  web1:
    ports:
      - 1234:8080
    networks:
      - net1
    
...

  proxy:
    image: nginx:latest
    networks:
      - net1

...

  networks:
    net1:
      driver: bridge

nginx.conf

... 

location /api
    ...
    proxy_pass http://web1:8080/;

I must thank user8458126 for pointing me in the right direction.