How to run a Single CircleCi job that requires multiple primary languages? (Using Docker Executor)

Question

My development team maintains two separate web applications that interact via REST.

One application is built in Go and the other in Python.

In order confirm that the two apps interact properly prior to Prod releases, we aimed to create an automated CircleCI Integration testing job that does the following:

Integration Test Requirements:

1. Check out Go and Python App
2. Build and start both apps on localhost
3. Run live integration tests where the Go App calls the Python app

Problem: The Pre-Built Docker Images in CircleCi (https://circleci.com/docs/2.0/circleci-images/) only support one primary language.

Solution: In order to run both a Go and Python server in the same CircleCI job, we created a custom Docker image that combines the public circleCI Golang and Python images.

In order to help other teams who may run into a similiar situation (i.e. a team that wants a Java + Python CircleCi docker container), sharing steps we took to solve this problem.

Steps to create a custom Go + Python docker image and use in CircleCi

1. Create a public repository in Docker Hub and link it to an existing Github account.
2. Here's the Docker Hub repo we used for this example: https://hub.docker.com/r/ejparz/circleci-images/dockerfile
3. We created our DockerFile based on the public CircleCi Go and Python docker images available here:

GO: https://github.com/CircleCI-Public/circleci-dockerfiles/blob/master/golang/images/1.11.0/Dockerfile

Python: https://github.com/CircleCI-Public/circleci-dockerfiles/blob/master/python/images/3.7.2-stretch/Dockerfile

Public go docker base image here: https://github.com/docker-library/golang/blob/master/1.11/stretch/Dockerfile

Our custom DockerFile:

    FROM circleci/golang:1.11-stretch
    FROM circleci/python:3.7.2-stretch

    #Create regular go directory
    COPY --from=0 /go /go

    #Copy go binaries
    COPY --from=0 /usr/local/go /usr/local/go

    ENV GOPATH /go
    ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH

    #Use Root to set path permissions
    USER root
    RUN chmod -R 777 "$GOPATH"

    #Add circleci user
    USER circleci

    #Add Shell
    CMD ["/bin/sh"]

4. After linking our Github Dockerfile to Docker Hub and auto building, we were able to reference the image in our CircleCi file

   run-system-integration-tests:
      docker:
         - image: ejparz/circleci-images:latest
   

From there, we were able to start our Python app on one port.

  - run:
        name: Start Python Django Server in background 
        command: |
          cd ~/app
          pipenv run python manage.py runserver
        background: true
  - run:
        name: Waiting for python server to be ready
        command: |
          for i in `seq 1 10`;
          do
            nc -z localhost 8000 && echo Success && exit 0
            echo -n .
            sleep 1
          done
          echo Failed waiting for server && exit 1

Then call that port from our golang app. (Or curl for basic test purposes)

     - run:
        name: Test Port connectivity
        command: |
          curl http://localhost:8000

That's it.

With some small tweaks to the above DockerFile template, other teams/users should be able to easily compose different CircleCi Primary language containers.

Answer 1

Create a custom Docker Image that combines the desired primary languages :

FROM circleci/golang:1.11-stretch
FROM circleci/python:3.7.2-stretch

#Create regular go directory
COPY --from=0 /go /go

#Copy go binaries
COPY --from=0 /usr/local/go /usr/local/go

ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH

#Use Root to set path permissions
USER root
RUN chmod -R 777 "$GOPATH"

#Add circleci user
USER circleci

#Add Shell
CMD ["/bin/sh"]