Using docker-compose in a GitLab CI pipeline

Question

I'm trying to implement a GitLab continuous integration (CI) pipeline with the following .gitlab-ci.yml file:

image: docker:latest

# When using dind, it's wise to use the overlayfs driver for
# improved performance.
variables:
  DOCKER_DRIVER: overlay

services:
  - docker:dind

before_script:
  - docker info
  - curl -L "https://github.com/docker/compose/releases/download/1.10.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  - chmod +x /usr/local/bin/docker-compose
  - export PATH=/usr/local/bin:$PATH
  - echo $PATH

stages:
  - build
  - test

build:
  stage: build
  script:
    - docker-compose build
    - docker-compose up -d

test:
  stage: test
  script:
    - cd tests/tester
    - pytest test_run_tester.py
  except:
    - /^.*skip_tests$/

However, in GitLab I'm getting the following error message:

Running with gitlab-ci-multi-runner 1.10.4 (b32125f)
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Waiting for services to be up and running...
Pulling docker image docker:latest ...
Running on runner-2e54fd37-project-13-concurrent-0 via scw-de9c9c...
Fetching changes...
HEAD is now at 2504a08 Update CI config
From https://lab.iperlane.com/gio/ipercron-compose
   2504a08..5c2f23f  CI         -> origin/CI
Checking out 5c2f23f1 as CI...
Skipping Git submodules setup
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.8.14-docker-2
Operating System: Alpine Linux v3.5 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.751 GiB
Name: 90395a030c02
ID: 7TKR:5PQN:XLFM:EJST:NF2V:NLQC:I2IZ:6OZG:TR4U:ZEAK:EVXE:HIF7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
$ curl -L "https://github.com/docker/compose/releases/download/1.10.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   600    0   600    0     0   1175      0 --:--:-- --:--:-- --:--:--  1176

  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  3 7929k    3  254k    0     0   116k      0  0:01:08  0:00:02  0:01:06  258k
 17 7929k   17 1358k    0     0   433k      0  0:00:18  0:00:03  0:00:15  704k
 61 7929k   61 4861k    0     0  1164k      0  0:00:06  0:00:04  0:00:02 1636k
100 7929k  100 7929k    0     0  1703k      0  0:00:04  0:00:04 --:--:-- 2300k
$ chmod +x /usr/local/bin/docker-compose
$ export PATH=/usr/local/bin:$PATH
$ docker-compose build
/bin/sh: eval: line 51: docker-compose: not found
ERROR: Build failed: exit code 127

The system seems not to be able to find docker-compose, even after installing it and adding it to the path. Is there perhaps another image such that docker-compose commands can be made, or a different way to install it in the .gitlab-ci.yml?

Answer 1

The problem

This is complex problem.

The docker:latest image is based on alpine (Alpine Linux), which is built using musl-libc. This system is very barebones, and as such doesn't have everything a full-fledged desktop Linux might have. In fact, dynamic executables need to be compiled specifically for this system.

docker-compose is a Python application, bundled into a Linux executable using PyInstaller. These executables are really only expected to be able to run on the system which they were built. If you run an alpine container, and apk add file, you can see that the (dynamically-linked) executable you downloaded is expecting a specific interpreter.

# file /usr/local/bin/docker-compose
/usr/local/bin/docker-compose.pyinstaller: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=00747fe5bcde089bb4c2f1ba7ab5348bc02ac5bf, stripped

The problem is /lib64/ld-linux-x86-64.so.2 doesn't exist in alpine. In fact, /lib64 doesn't even exist.

The solution

Because docker-compose is a Python application, it can also be installed using pip:

Testing this inside an alpine container:

$ docker run --rm -it alpine /bin/bash

Here are the commands you can run manually, or add to your .gitlab-ci.yml before_script:

# apk add --no-cache python py2-pip
...
# pip install --no-cache-dir docker-compose
...
# docker-compose -v
docker-compose version 1.11.1, build 7c5d5e4

---

It turns out there is actually an open issue for this:

• https://github.com/docker/compose/issues/3465