Docker blocking outgoing connections on high load?

Question

We have a node.js web server that makes some outgoing http requests to an external API. It's running in docker using dokku.
After some time of load (30req/s) these outgoing requests aren't getting responses anymore.

Here's a graph I made while testing with constant req/s: incoming and outgoing is the amount of concurrent requests (not the number of initialized requests). (It's hard to see in the graph, but it's fairly constant at ~10 requests for each.)
response time is for external requests only. You can clearly see that they start failing all of a sudden (hitting our 1000ms timeout).

---

The more req/s we send, the faster we run into this problem, so we must have some sort of limit we're getting closer to with each request.

---

I used netstat -ant | tail -n +3 | wc -l on the host to get the number of open connections, but it was only ~450 (most of them TIME_WAIT). That shouldn't hit the socket limit. We aren't hitting any RAM or CPU limits, either.

---

I also tried running the same app on the same machine outside docker and it only happens in docker.

Answer 1

It could be due to the Docker userland proxy. If you are running a recent version of Docker, try running the daemon with the --userland-proxy=false option. This will make Docker handle port forwarding with just iptables and there is less overhead.