docker-compose fails at adding veth (virtual interface) to bridge docker0

Question

Issue:
Here's my docker-compose.yaml:

version: '2'

services:
    elasticsearch:
        image: 'elasticsearch:latest'
        ports:
            - "9200:9200"

    logstash:
        build: './logstash_image'
        links:
            - elasticsearch
        ports:
            - "5000:5000"
            - "5001:5001"
            - "5201:5201"
            - "5202:5202"
            - "5203:5203"

    kibana:
        image: 'kibana:latest'
        ports:
             - '5601:5601'
        links:
             - elasticsearch
        volumes:
             - ./kibana.yml:/opt/kibana/config/kibana.yml

And here's my Dockerfile:

FROM logstash:latest

LABEL maintainer1.name="anon" \
      maintainer1.email="[email protected]" \
      maintainer2.name="myname" \
      maintainer2.email="[email protected]"

RUN /opt/logstash/bin/logstash-plugin install logstash-input-log4j2-logstash2 logstash-output-syslog
# FIX: use `logstash-input-log4j2` instead of `logstash-input-log4j2-logstash2`?
# other plugins: logstash-input-http, logstash-input-log4j
COPY logstash.conf /etc/logstash/conf.d/logstash.conf

ENTRYPOINT ["logstash","-f","/etc/logstash/conf.d/logstash.conf", "--debug"]

This used to build just fine a few days ago, however currently I'm getting the following error message:

[root@somemachine elk_stack]# docker-compose up -d
Creating network "elkstack_default" with the default driver
Building logstash
Step 1 : FROM logstash:latest
 ---> 1ca34df702f8
Step 2 : LABEL maintainer1.name ...
 ---> Using cache
 ---> 6dd78ac216f2
Step 3 : RUN /opt/logstash/bin/logstash-plugin install logstash-input-log4j2-logstash2 logstash-output-syslog
 ---> Running in 885b0429b3a4
ERROR: Service 'logstash' failed to build: failed to create endpoint focused_almeida on network bridge: adding interface veth0491201 to bridge docker0 failed: operation not supported

ERROR: Service 'logstash' failed to build: failed to create endpoint focused_almeida on network bridge: adding interface veth0491201 to bridge docker0 failed: operation not supported

So it seems like (1) docker-compose is assigning a randomly generated name to the logstash service instead of elkstack_logstash_1 presumably because the compose did not fully go through.
(2) veth0491201 is unable to attach to the bridge docker0 for some reason, because of this I can't reach the internet from within the container as it's not connected through docker0.

Investigation:
docker ps -a after the compose:

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                                        NAMES
885b0429b3a4        6dd78ac216f2                "/bin/sh -c '/opt/log"   18 minutes ago      Created                                                          focused_almeida

docker network ls after the compose:

NETWORK ID          NAME                          DRIVER            
23bcc8c01ad4        bridge                        bridge              
4aea66903c6b        none                          null                
67cb26c508d9        host                          host                
9fc4a42bbcf6        elkstack_default              bridge  

docker network inspect elkstack_default:

[
    {
        "Name": "elkstack_default",
        "Id": "9fc4a42bbcf6800889f8e9bc71ab394ebf6f97b21e0ce2345253b055d17138aa",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1/16"
                }
            ]
        },
        "Containers": {},
        "Options": {}
    }
]

route -n:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.120.1      0.0.0.0         UG    100    0        0 docker0
10.0.120.0      0.0.0.0         255.255.255.0   U     100    0        0 docker0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-83e64c848669
172.19.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-9fc4a42bbcf6

Troubleshooting leads:

1. docker --version: Docker version 1.10.2, build c3959b1
   docker-compose --version: docker-compose version 1.8.0, build f3628c7
   Some networking bug in Docker?
2. The machine at which I'm doing this has recently done some config changes in regards to its network, firewalls etc.
   I think it's using 172.18.0.0 for something else, so when Docker tries to use that route it gets blocked. How can I troubleshoot or determine if it's an issue with the machines network settings?
3. Following up on point made in (2), I tried to make a docker-compose where the network is explicitly defined, this didn't seem to work either:

networks:
    net:
        driver: bridge
        ipam:
            driver: default
            config:
            - subnet: 172.30.0.0/16
              ip_range: 172.30.5.0/24

• I'm able to docker-compose another stack of services on the machine, this leads me to believe that it's a Docker-related or docker-compose-related flaw, either in the application itself or my configuration.

• I'm able to get this to work on my local machine, this leads me to believe that it's an issue with the remote machine where I'm having issues.

  uname -a: Linux somemachine 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  cat /etc/centos-release: CentOS Linux release 7.2.1511 (Core)

Related info & issues:
- http://54.71.194.30:4017/articles/networking/#how-docker-networks-a-container
- https://github.com/docker/docker/issues/23047
- https://github.com/docker/docker/issues/15341

Answer 1

Your problem looks to be related with your system configuration.

Please, could you try to add manually, with unix command brctl the veth into the bridge ?

The command should be:

brctl addif veth0491201 docker0

This is the command done by the script to start your VM that is failling. By troubleshooting that step may explain the reason.

I suspect the type of veth is not compatible on YOUR system, and you have to know why. May be a bug into your system regarding the bridge software part.

For sûre, your problem is not linked with your ip setting.

You may try to upgrade OR downgrade your kernel, by testing different kernel releases.

Without more details about what exactly happens during the command "brctl addif veth0491201 docker0" I cannot say more things.

That will return the same "Operation not supported" error message, but by checking everything about both interfaces, you may find either, a good reason why the "operation is not supported" (wrong type interface) or a kernel bug with (good type of interface).