Connection refused on API request between containers with docker compose

Question

I'm developing a multi-container Docker application and I want a container to make an HTTP request to API of other container using Docker Compose. I'm getting Connection Refused error.

Both containers run ASP.NET Core 2.2. The IDE I'm using is Visual Studio 2017. I'm testing API calls with Postman.

Things I've tried:
✔️ Ports are exposed in Dockerfiles
✔️ Ports are declared in Docker Compose configuration
✔️ Network is declared in Docker Compose and containers are connected to it
✔️ Http request URI uses service name and not localhost or local IP address
✔️ Http request URI uses container port (80) and not host port
✔️ Firewall is disabled
✔️ Custom network with custom subnet
✔️ Custom IPv4 address for services
✔️ Docker Compose downgraded to v2.4 (to specify gateway on custom networks)
✔️ Delete and recreate serviceB project
✔️ Switched from basic usage of HttpClient to typed clients (custom services)
✔️ Switched from GetAsync(Uri) to GetAsync(Uri,HttpCompletionOption,CancellationToken)

Dockerfiles

FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
...

Docker Compose configuration:

version: '3.4'

services:
  servicea:
    ...
    ports:
      - "51841:80"
      - "44364:443"
    networks:
      - local

  serviceb:
      ...
    ports:
      - "65112:80"
      - "44359:443"
    networks:
      - local

networks:
  local:
    driver: bridge

serviceA controller action:

[Route("[action]")]
[HttpGet]
public IActionResult foo()
{
   HttpClient client = _clientFactory.CreateClient();

   var result = client.GetAsync("http://serviceb:80/api/bar").Result;
   var response = result.Content.ReadAsStringAsync().Result;

   return new OkObjectResult(response);
}

If I do an Http Get request to serviceA (with host port 51841) with Postman at http://localhost:51841/api/foo I'd like to get the response of serviceB's Bar action.

But I'm getting Connection refused

Raw exception details:

System.Net.Http.HttpRequestException: Connection refused ---> System.Net.Sockets.SocketException: Connection refused
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

PING

If I access to serviceA's bash and do ping to serviceB (and specifically serviceB's :80 port) it works:

root@serviceA_id:/app# ping -p 80 serviceb
PATTERN: 0x80
PING serviceb(10.168.0.3) 56(84) bytes of data.
64 bytes from ... (10.168.0.3): icmp_seq=1 ttl=64 time=0.117 ms
64 bytes from ... (10.168.0.3): icmp_seq=2 ttl=64 time=0.101 ms
64 bytes from ... (10.168.0.3): icmp_seq=3 ttl=64 time=0.083 ms
--- serviceb ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2057ms rtt min/avg/max/mdev = 0.083/0.100/0.117/0.016 ms

CURL

I can also stablish connection with API REST endpoint with CURL but Content-Length recieved is 0

root@serviceA_id:/app# curl -v http://serviceb/api/bar
*   Trying 10.168.0.3...
* TCP_NODELAY set
* Connected to serviceb (10.168.0.3) port 80 (#0)
> GET /api/bar/ HTTP/1.1
> Host: serviceb
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 307 Temporary Redirect
< Date: Mon, 17 Jun 2019 07:11:31 GMT
< Server: Kestrel
< Content-Length: 0
< Location: https://serviceb:44359/api/bar/
<
* Curl_http_done: called premature == 0
* Connection #0 to host serviceb left intact

So we can see serviceB is telling serviceA that its request will be redirected to https://serviceb:44359/api/bar/ but the desired connection port is 80 (container port), not 44359 (host port)

If I let curl follow redirections then Connection Refused appears (the redirected port is closed)

root@serviceA_id:/app# curl -v -L http://serviceb/api/bar
*   Trying 10.168.0.3...
* TCP_NODELAY set
* Connected to serviceb (10.168.0.3) port 80 (#0)
> GET /api/bar HTTP/1.1
> Host: serviceb
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 307 Temporary Redirect
< Date: Wed, 19 Jun 2019 08:48:33 GMT
< Server: Kestrel
< Content-Length: 0
< Location: https://serviceb:44359/api/bar
<
* Curl_http_done: called premature == 0
* Connection #0 to host serviceb left intact
* Issue another request to this URL: 'https://serviceb:44359/api/bar'
*   Trying 10.168.0.3...
* TCP_NODELAY set
* connect to 10.168.0.3 port 44359 failed: Connection refused
* Failed to connect to serviceb port 44359: Connection refused
* Closing connection 1
curl: (7) Failed to connect to serviceb port 44359: Connection refused

Why I am being redirected to host port?

In Startup.cs my services were using app.UseHttpsRedirection(); so removing that line solved the problem

What if I still need to use HTTPS? Add option to use container port. More info in answer

Answer 1

ServiceA's HTTP requests were being redirected (HTTP 307 status code) to https://serviceb:44359/api/bar being :44359 the host port for HTTPS. Host ports are not accessible between containers, container ports do. So if I access to serviceA's terminal and send an HTTP request with curl verbose -v following redirections -L to the URI http://serviceb/api/bar I got the Connection Refused error:

root@serviceA_id:/app# curl -v -L http://serviceb/api/bar
*   Trying 10.168.0.3...
* TCP_NODELAY set
* Connected to serviceb (10.168.0.3) port 80 (#0)
> GET /api/bar HTTP/1.1
> Host: serviceb
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 307 Temporary Redirect
< Date: Wed, 19 Jun 2019 08:48:33 GMT
< Server: Kestrel
< Content-Length: 0
< Location: https://serviceb:44359/api/bar
<
* Curl_http_done: called premature == 0
* Connection #0 to host serviceb left intact
* Issue another request to this URL: 'https://serviceb:44359/api/bar'
*   Trying 10.168.0.3...
* TCP_NODELAY set
* connect to 10.168.0.3 port 44359 failed: Connection refused
* Failed to connect to serviceb port 44359: Connection refused
* Closing connection 1
curl: (7) Failed to connect to serviceb port 44359: Connection refused

Why I were being redirected to host port?

In Startup.cs my services were using app.UseHttpsRedirection();, that line was causing the problem.

The default configuration of HttpsPolicyBuilderExtensions.UseHttpsRedirection(IApplicationBuilder) method redirects to HTTPS host port by default. If you want to use a different port for redirection you need to add that option so Startup.cs will look like this:

public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        public void ConfigureServices(IServiceCollection services)
        {

            ...

            services.AddHttpsRedirection(options =>
            {
                options.HttpsPort = 443;
            });

            ...

        }

        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {

            ...

            app.UseHttpsRedirection();

            ...

        }
    }

Answer 2

but it doesn't when I do ping to serviceB's API REST:

root@serviceA_id:/app# ping -p 80 serviceb/api/bar
PATTERN: 0x80
ping: serviceb/api/bar: Temporary failure in name resolution

That may be the cause of Connection Refused error when doing API requests from ASP.NET

That's not how ping works. Ping uses ICMP, not TCP, and not HTTP which works on top of TCP. You ping a host, not a TCP port or HTTP API. So the above is expected to fail.

I can also stablish connection with API REST endpoint with CURL but Content-Length recieved is 0

root@serviceA_id:/app# curl -v http://serviceb/api/bar
*   Trying 10.168.0.3...
* TCP_NODELAY set
* Connected to serviceb (10.168.0.3) port 80 (#0)

This indicates you have correctly configured docker containers to communicate, there's nothing more to debug in the configuration of your networks, and you know your serviceb is is listening.

The only things left to debug are:

• Make sure servicea is connecting to serviceb on port 80, not 65112 or any other host port. Containers communicate between each other on container ports.
• Make sure you are running the code posted, and not a previous version. This is easy to mistake when building and deploying images, especially if you aren't changing your image tag for each build.
• Make sure you give serviceb time to start. I've often seen these errors when servicea starts before serviceb.

If you still have issues, you can start debugging with tools like tcpdump. E.g.

docker run -it --rm --net container:$container_id \
  nicolaka/netshoot tcpdump -i any port 80

Replace the $container_id with the id of serviceb to see any TCP requests to port 80 on that container.