Gitlab CI runner not able to expose ports of nested Docker containers

Question

When using GitLab CI, as well as the gitlab-ci-multi-runner, I'm unable to get internally-started Docker containers to expose their ports to the "host", which is the Docker image in which the build is running.

My .gitlab-ci.yml file:

test:   image: docker   stage: test   services:     - docker:dind   script:     - APP_CONTAINER_ID=`docker run -d --privileged -p "9143:9143" appropriate/nc nc -l 9143`     - netstat -a     - docker exec $APP_CONTAINER_ID netstat -a     - nc -v localhost 9143 

My command:

gitlab-ci-multi-runner exec docker --docker-privileged test 

The output:

$ netstat -a Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address           Foreign Address         State tcp        0      0 runner--project-1-concurrent-0:54664 docker:2375             TIME_WAIT tcp        0      0 runner--project-1-concurrent-0:54666 docker:2375             TIME_WAIT Active UNIX domain sockets (servers and established) Proto RefCnt Flags       Type       State         I-Node Path  $ docker exec $APP_CONTAINER_ID netstat -a Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address           Foreign Address         State tcp        0      0 0.0.0.0:9143            0.0.0.0:*               LISTEN Active UNIX domain sockets (servers and established) Proto RefCnt Flags       Type       State         I-Node Path  $ nc -v localhost 9143 ERROR: Build failed: exit code 1 FATAL: exit code 1 

What am I doing wrong here?

Original Question Follows - above is a shorter, easier-to-test example

I have an application image that listens on port 9143. Its startup and config is managed via docker-compose.yml, and works great on my local machine with docker-compose up - I can access localhost:9143 without issue.

However, when running on GitLab CI (the gitlab.com version) via a shared runner, the port doesn't seem to be exposed.

The relevant portion of my .gitlab-ci.yml:

test:   image: craigotis/buildtools:v1   stage: test   script:     - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com/craigotis/myapp     - docker-compose up -d     - sleep 60 # a temporary hack to get the logs     - docker-compose logs     - docker-machine env     - docker-compose port app 9143     - netstat -a     - docker-compose ps     - /usr/local/bin/wait-for-it.sh -h localhost -p 9143 -t 60     - cd mocha     - npm i     - npm test     - docker-compose down 

The output is:

$ docker-compose logs ... app_1  | [Thread-1] INFO spark.webserver.SparkServer - == Spark has ignited ... app_1  | [Thread-1] INFO spark.webserver.SparkServer - >> Listening on 0.0.0.0:9143 app_1  | [Thread-1] INFO org.eclipse.jetty.server.Server - jetty-9.0.z-SNAPSHOT app_1  | [Thread-1] INFO org.eclipse.jetty.server.ServerConnector - Started ServerConnector@6919dc5{HTTP/1.1}{0.0.0.0:9143} ...  $ docker-compose port app 9143 0.0.0.0:9143  $ netstat -a Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address           Foreign Address         State        tcp        0      0 runner-e11ae361-project-1925166-concurrent-0:53646 docker:2375             TIME_WAIT    tcp        0      0 runner-e11ae361-project-1925166-concurrent-0:53644 docker:2375             TIME_WAIT    tcp        0      0 runner-e11ae361-project-1925166-concurrent-0:53642 docker:2375             TIME_WAIT    Active UNIX domain sockets (servers and established) Proto RefCnt Flags       Type       State         I-Node Path  $ docker-compose ps stty: standard input: Not a tty     Name                  Command               State                Ports                ---------------------------------------------------------------------------------------- my_app_1   wait-for-it.sh mysql_serve ...   Up      8080/tcp, 0.0.0.0:9143->9143/tcp  mysql_server   docker-entrypoint.sh --cha ...   Up      3306/tcp       $ /usr/local/bin/wait-for-it.sh -h localhost -p 9143 -t 60 wait-for-it.sh: waiting 60 seconds for localhost:9143 wait-for-it.sh: timeout occurred after waiting 60 seconds for localhost:9143 

The contents of my docker-compose.yml:

version: '2'  networks:     app_net:         driver: bridge  services:     app:         image: registry.gitlab.com/craigotis/myapp:latest         depends_on:         - "db"         networks:         - app_net         command: wait-for-it.sh mysql_server:3306 -t 60 -- java -jar /opt/app*.jar         ports:         - "9143:9143"      db:         image: mysql:latest         networks:         - app_net         container_name: mysql_server         environment:         - MYSQL_ALLOW_EMPTY_PASSWORD=true 

It seems like my application container is listening on 9143, and it's properly exposed to the shared GitLab runner, but it doesn't seem to actually be exposed. It works fine on my local machine - is there some special workaround/tweak I need to make this work inside a Docker container running on GitLab?

Answer 1

When using docker:dind a container is created and your docker-compose containers get setup within it. It exposes the ports to localhost within the docker:dind container. You cannot access this as localhost from the environment that your code is executing in.

A hostname of docker is setup for you to reference this docker:dind container. You can check by using cat /etc/hosts.

Instead of referencing localhost:9143 you should use docker:9143.

Answer 2

The offical gitab-ci on gitlab.com documentation refers to the example of PostgreSQL

Its working CI does not try to connect to localhost, but rather to the service name

The services keyword defines just another docker image that is run during your build and is linked to the docker image that the image keyword defines. This allows you to access the service image during build time.

The service container for MySQL will be accessible under the hostname mysql.
So, in order to access your database service you have to connect to the host named mysql instead of a socket or localhost.

You could check if this applies in your case, and try accessing your application service in app:9143 instead of localhost:9143.