Can't run AWS CodeBuild local build script

Question

I'm trying to run the CodeBuild locally for my golang project. I've pulled the docker image amazon/aws-codebuild-local:latest and run the CodeBuild script ./codebuild_build.sh -i aws/codebuild/standard:2.0 -s "/project/src", there is nothing happened. Then I captured the docker command generated in the script: docker run -it -v //var/run/docker.sock:/var/run/docker.sock -e "IMAGE_NAME=aws/codebuild/standard:2.0" -e "SOURCE=/project/src" -e "INITIATOR=me" amazon/aws-codebuild-local:latest.

When I run the docker command, I got the following error:

Removing network agent-resources_default
Removing volume agent-resources_source_volume
Removing volume agent-resources_user_volume
Creating network "agent-resources_default" with the default driver
Creating volume "agent-resources_source_volume" with local driver
Creating volume "agent-resources_user_volume" with local driver
Pulling build (aws/codebuild/standard:2.0)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]y
Pulling build (aws/codebuild/standard:2.0)...
ERROR: pull access denied for aws/codebuild/standard, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

I'm sure my docker is logged in, don't know what should I do next to solve the issue.

Answer 1

An example for using local build platform of Ubuntu 18.04 with docker image identfier/tag aws/codebuild/standard:3.0 to run few python unit tests (see below) as an example.

1. Make local docker image of aws/codebuild/standard:3.0

The Ubuntu 18.04 platform is identified as aws/codebuild/standard:3.0 in docker

# download definition of curated docker codebuild images
git clone https://github.com/aws/aws-codebuild-docker-images.git

# got ubuntu version of intrest
cd aws-codebuild-docker-images/ubuntu/standard/3.0/

# build the image (this will take a time as the final image is > 7GB)
docker build -t aws/codebuild/standard:3.0 .

2. Download codebuild_build.sh

wget https://raw.githubusercontent.com/aws/aws-codebuild-docker-images/master/local_builds/codebuild_build.sh

chmod u+x codebuild_build.sh

3. Run the local build job

./codebuild_build.sh -i aws/codebuild/standard:3.0 -a /tmp/artifacts -s ./application/

where ./application/ should be changed to your folder with the application you want to build. The folder should contain your buildspec.yml. My example buildspec.yml was:

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.8
  pre_build:
    commands:
      - echo Nothing to do in the pre_build phase...
  build:
    commands:
      - echo Running my python unit tests on `date`
      - python -m unittest test.py  
  post_build:
    commands:
      - echo Build completed on `date`

Notice - python -m unittest test.py as I was executing unit tests in the build phase only. The test.py was just an example from the Python docs: https://docs.python.org/3/library/unittest.html#basic-example

The ./codebuild_build.sh will also pull Docker image amazon/aws-codebuild-local first time you execute it.

4. Example output:

Removing agent-resources_build_1 ... done
Removing agent-resources_agent_1 ... done
Removing network agent-resources_default
Removing volume agent-resources_source_volume
Removing volume agent-resources_user_volume
Creating network "agent-resources_default" with the default driver
Creating volume "agent-resources_source_volume" with local driver
Creating volume "agent-resources_user_volume" with local driver
Creating agent-resources_agent_1 ... done
Creating agent-resources_build_1 ... done
Attaching to agent-resources_agent_1, agent-resources_build_1
agent_1  | 2020/01/05 07:50:34 [Customer Config] Couldn't open specified customer config file: open /root/.aws/config: no such file or directory
agent_1  | 2020/01/05 07:50:34 [Customer Config] Error parsing supplied customer config file: invalid argument
agent_1  | [Container] 2020/01/05 07:50:35 Waiting for agent ping
agent_1  | [Container] 2020/01/05 07:50:36 Waiting for DOWNLOAD_SOURCE
agent_1  | [Container] 2020/01/05 07:50:36 Phase is DOWNLOAD_SOURCE
agent_1  | [Container] 2020/01/05 07:50:36 CODEBUILD_SRC_DIR=/codebuild/output/src628986230/src
agent_1  | [Container] 2020/01/05 07:50:36 YAML location is /codebuild/output/srcDownload/src/buildspec.yml
agent_1  | [Container] 2020/01/05 07:50:36 No commands found for phase name: INSTALL
agent_1  | [Container] 2020/01/05 07:50:36 Processing environment variables
agent_1  | [Container] 2020/01/05 07:50:36 Moving to directory /codebuild/output/src628986230/src
agent_1  | [Container] 2020/01/05 07:50:36 Registering with agent
agent_1  | [Container] 2020/01/05 07:50:36 Phases found in YAML: 4
agent_1  | [Container] 2020/01/05 07:50:36  INSTALL: 0 commands
agent_1  | [Container] 2020/01/05 07:50:36  PRE_BUILD: 1 commands
agent_1  | [Container] 2020/01/05 07:50:36  BUILD: 2 commands
agent_1  | [Container] 2020/01/05 07:50:36  POST_BUILD: 1 commands
agent_1  | [Container] 2020/01/05 07:50:36 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
agent_1  | [Container] 2020/01/05 07:50:36 Phase context status code:  Message: 
agent_1  | [Container] 2020/01/05 07:50:36 Entering phase INSTALL
agent_1  | [Container] 2020/01/05 07:50:36 Running command echo "Installing Python version 3.8 ..."
agent_1  | Installing Python version 3.8 ...
agent_1  | 
agent_1  | [Container] 2020/01/05 07:50:36 Phase complete: INSTALL State: SUCCEEDED
agent_1  | [Container] 2020/01/05 07:50:36 Phase context status code:  Message: 
agent_1  | [Container] 2020/01/05 07:50:36 Entering phase PRE_BUILD
agent_1  | [Container] 2020/01/05 07:50:36 Running command echo Nothing to do in the pre_build phase...
agent_1  | Nothing to do in the pre_build phase...
agent_1  | 
agent_1  | [Container] 2020/01/05 07:50:36 Phase complete: PRE_BUILD State: SUCCEEDED
agent_1  | [Container] 2020/01/05 07:50:36 Phase context status code:  Message: 
agent_1  | [Container] 2020/01/05 07:50:36 Entering phase BUILD
agent_1  | [Container] 2020/01/05 07:50:36 Running command echo Runing my python unittests on `date`
agent_1  | Runing my python unittests on Sun Jan 5 07:50:36 UTC 2020
agent_1  | 
agent_1  | [Container] 2020/01/05 07:50:36 Running command python -m unittest test.py
agent_1  | ...
agent_1  | ----------------------------------------------------------------------
agent_1  | Ran 3 tests in 0.000s
agent_1  | 
agent_1  | OK
agent_1  | 
agent_1  | [Container] 2020/01/05 07:50:37 Phase complete: BUILD State: SUCCEEDED
agent_1  | [Container] 2020/01/05 07:50:37 Phase context status code:  Message: 
agent_1  | [Container] 2020/01/05 07:50:37 Entering phase POST_BUILD
agent_1  | [Container] 2020/01/05 07:50:37 Running command echo Build completed on `date`
agent_1  | Build completed on Sun Jan 5 07:50:37 UTC 2020
agent_1  | 
agent_1  | [Container] 2020/01/05 07:50:37 Phase complete: POST_BUILD State: SUCCEEDED
agent_1  | [Container] 2020/01/05 07:50:37 Phase context status code:  Message: 
agent-resources_agent_1 exited with code 0
Stopping agent-resources_build_1 ... done
Aborting on container exit...

5. Check if the build job was successful or not?

In my case I just wanted to know if all unit tests in test.py were successful or not. For this you can just check the exit code from the codebuild_build.sh

echo ${?}

This will return 0 if tests were successful, or 1 if not. Can verify this by modifying test.py so that the unit tests fail, and re-running codebuild_build.sh. This works because python -m unittest test.py exits with code of 0 when all tests pass, and 1 otherwise.

Answer 2

First build and tag the CodeBuild docker image as described here:

https://github.com/aws/aws-codebuild-docker-images

Then run the following command making sure to update the image name and tag in the command:

./codebuild_build.sh -i <image_name>:<image_tag> -a /home/ec2-user/environment/artifacts -s /home/ec2-user/environment/sample-web-app