Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to use AWS CodeArtifact *within* A Dockerfile in AWSCodeBuild

I am trying to do a pip install from codeartifact from within a dockerbuild in aws codebuild.

This article does not quite solve my problem: https://docs.aws.amazon.com/codeartifact/latest/ug/using-python-packages-in-codebuild.html

The login to AWS CodeArtifct is in the prebuild; outside of the Docker context.

But my pip install is inside my Dockerfile (we pull from a private pypi registry).

How do I do this, without doing something horrible like setting an env variable to the password derived from reading ~/.config/pip.conf/ after running the login command in prebuild?

like image 522
Tommy Avatar asked May 06 '21 22:05

Tommy


People also ask

What is code artifact in AWS?

AWS CodeArtifact is a fully managed artifact repository service that makes it easy for organizations of any size to securely store, publish, and share software packages used in their software development process.

Does CodeBuild create Docker image?

For Image, choose aws/codebuild/standard:4.0. Because you use this build project to build a Docker image, select Privileged. By default, Docker containers do not allow access to any devices. Privileged mode grants a build project's Docker container access to all devices.

How to use AWS codebuild with Docker Hub?

Follow the steps in Run CodeBuild directly to create a build environment, run the build, and view related build information. Confirm that AWS CodeBuild successfully pushed the Docker image to the repository. Sign in to Docker Hub, go to the repository, and choose the Tags tab. The latest tag should contain a very recent Last Updated value.

How do I run a codebuild build in AWS codebuild?

Follow the steps in Run CodeBuild directly to create a build environment, run the build, and view related build information. Confirm that AWS CodeBuild successfully pushed the Docker image to the repository. Sign in to Docker Hub, go to the repository, and choose the Tags tab. The latest tag should contain a very recent Last Updated value.

What's new in AWS codeartifact?

Tasks such as initial configuration, ongoing maintenance, and scaling inefficiencies are the biggest pain points for developers and organizations. With its addition of NuGet package support, AWS CodeArtifact now provides easy-to-configure and scalable package management for .NET developers.

How to pass credentials from codebuild to a docker build?

And you need to pass those args as build-args! And that is one of the ways you can pass the credentials of a codebuild instance to a docker build! You can also read about another way were you use a specific role purely for your docker build on the premium support website of AWS.


Video Answer


2 Answers

You can use the environment variable: PIP_INDEX_URL[1].

Below is an AWS CodeBuild buildspec.yml file where we construct the PIP_INDEX_URL for CodeArtifact by using this example from the AWS documentation.

buildspec.yml

  pre_build:
    commands:
      - echo Getting CodeArtifact authorization...
      - export CODEARTIFACT_AUTH_TOKEN=$(aws codeartifact get-authorization-token --domain "${CODEARTIFACT_DOMAIN}" --domain-owner "${AWS_ACCOUNT_ID}" --query authorizationToken --output text)
      - export PIP_INDEX_URL="https://aws:${CODEARTIFACT_AUTH_TOKEN}@${CODEARTIFACT_DOMAIN}-${AWS_ACCOUNT_ID}.d.codeartifact.${AWS_DEFAULT_REGION}.amazonaws.com/pypi/${CODEARTIFACT_REPO}/simple/"

In your Dockerfile, add an ARG PIP_INDEX_URL line just above your RUN pip install -r requirements.txt so it can become an environment variable during the build process:

Dockerfile

# this needs to be added before your pip install line!
ARG PIP_INDEX_URL

RUN pip install -r requirements.txt

Finally, we build the image with the PIP_INDEX_URL build-arg.

buildspec.yml

  build:
    commands:
      - echo Building the Docker image...
      - docker build -t "${IMAGE_REPO_NAME}" --build-arg PIP_INDEX_URL .

As an aside, adding ARG PIP_INDEX_URL to your Dockerfile shouldn't break any existing CI or workflows. If --build-arg PIP_INDEX_URL is omitted when building an image, pip will still use the default PyPI index.

Specifying --build-arg PIP_INDEX_URL=${PIP_INDEX_URL} is valid, but unnecessary. Specifying the argument name with no value will make Docker take its value from the environment variable of the same name[2].

Security note: If someone runs docker history ${IMAGE_REPO_NAME}, they can see the value of ${PIP_INDEX_URL}[3] . The token is only good for a maximum of 12 hours though, and you can shorten it to as little as 15 minutes with the --duration-seconds parameter of aws codeartifact get-authorization-token[4], so maybe that's acceptable. If your Dockerfile is a multi-stage build, then it shouldn't be an issue if you're not using ARG PIP_INDEX_URL in your target stage. docker build --secret does not seem to be supported in CodeBuild at this time.

like image 150
Phistrom Avatar answered Oct 23 '22 21:10

Phistrom


So, here is how I solved this for now. Seems kinda hacky, but it works. (EDIT: we have since switched to @phistrom answer)

  1. In the prebuild, I run the command and copy ~/.config/pip/pip.conf to the current build directory:
pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      ...
      - echo Fetching pip.conf for PYPI
      - aws codeartifact --region us-east-1 login --tool pip --repository ....
      - cp ~/.config/pip/pip.conf .
  build:
    commands:
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
  1. Then in the Dockerfile, I COPY that file in, do the pip install, then rm it
COPY requirements.txt pkg/
COPY --chown=myuser:myuser pip.conf /home/myuser/.config/pip/pip.conf
RUN pip install -r ./pkg/requirements.txt
RUN pip install ./pkg
RUN rm /home/myuser/.config/pip/pip.conf

like image 21
Tommy Avatar answered Oct 23 '22 21:10

Tommy