I am trying to do a pip install from codeartifact from within a dockerbuild in aws codebuild.
This article does not quite solve my problem: https://docs.aws.amazon.com/codeartifact/latest/ug/using-python-packages-in-codebuild.html
The login to AWS CodeArtifct is in the prebuild; outside of the Docker context.
But my pip install
is inside my Dockerfile (we pull from a private pypi registry).
How do I do this, without doing something horrible like setting an env variable to the password derived from reading ~/.config/pip.conf/
after running the login command in prebuild?
AWS CodeArtifact is a fully managed artifact repository service that makes it easy for organizations of any size to securely store, publish, and share software packages used in their software development process.
For Image, choose aws/codebuild/standard:4.0. Because you use this build project to build a Docker image, select Privileged. By default, Docker containers do not allow access to any devices. Privileged mode grants a build project's Docker container access to all devices.
Follow the steps in Run CodeBuild directly to create a build environment, run the build, and view related build information. Confirm that AWS CodeBuild successfully pushed the Docker image to the repository. Sign in to Docker Hub, go to the repository, and choose the Tags tab. The latest tag should contain a very recent Last Updated value.
Follow the steps in Run CodeBuild directly to create a build environment, run the build, and view related build information. Confirm that AWS CodeBuild successfully pushed the Docker image to the repository. Sign in to Docker Hub, go to the repository, and choose the Tags tab. The latest tag should contain a very recent Last Updated value.
Tasks such as initial configuration, ongoing maintenance, and scaling inefficiencies are the biggest pain points for developers and organizations. With its addition of NuGet package support, AWS CodeArtifact now provides easy-to-configure and scalable package management for .NET developers.
And you need to pass those args as build-args! And that is one of the ways you can pass the credentials of a codebuild instance to a docker build! You can also read about another way were you use a specific role purely for your docker build on the premium support website of AWS.
You can use the environment
variable: PIP_INDEX_URL
[1].
Below is an AWS CodeBuild buildspec.yml
file where we construct the
PIP_INDEX_URL
for CodeArtifact by using
this example from the AWS documentation.
buildspec.yml
pre_build: commands: - echo Getting CodeArtifact authorization... - export CODEARTIFACT_AUTH_TOKEN=$(aws codeartifact get-authorization-token --domain "${CODEARTIFACT_DOMAIN}" --domain-owner "${AWS_ACCOUNT_ID}" --query authorizationToken --output text) - export PIP_INDEX_URL="https://aws:${CODEARTIFACT_AUTH_TOKEN}@${CODEARTIFACT_DOMAIN}-${AWS_ACCOUNT_ID}.d.codeartifact.${AWS_DEFAULT_REGION}.amazonaws.com/pypi/${CODEARTIFACT_REPO}/simple/"
In your Dockerfile, add an ARG PIP_INDEX_URL
line just above
your RUN pip install -r requirements.txt
so it can become an environment
variable during the build process:
Dockerfile
# this needs to be added before your pip install line! ARG PIP_INDEX_URL RUN pip install -r requirements.txt
Finally, we build the image with the PIP_INDEX_URL
build-arg.
buildspec.yml
build: commands: - echo Building the Docker image... - docker build -t "${IMAGE_REPO_NAME}" --build-arg PIP_INDEX_URL .
As an aside, adding ARG PIP_INDEX_URL
to your Dockerfile shouldn't break any
existing CI or workflows. If --build-arg PIP_INDEX_URL
is omitted when
building an image, pip will still use the default PyPI index.
Specifying --build-arg PIP_INDEX_URL=${PIP_INDEX_URL}
is valid, but
unnecessary. Specifying the argument name with no value will make Docker take
its value from the environment variable of the same
name[2].
Security note: If someone runs docker history ${IMAGE_REPO_NAME}
, they can
see the value
of ${PIP_INDEX_URL}
[3]
. The token is only good for a maximum of 12 hours though, and you can shorten
it to as little as 15 minutes with the --duration-seconds
parameter
of aws codeartifact get-authorization-token
[4],
so maybe that's acceptable. If your Dockerfile is a multi-stage build, then it
shouldn't be an issue if you're not using ARG PIP_INDEX_URL
in your target
stage. docker build --secret
does not seem to be supported in CodeBuild at this time.
So, here is how I solved this for now. Seems kinda hacky, but it works. (EDIT: we have since switched to @phistrom answer)
~/.config/pip/pip.conf
to the current build directory:pre_build:
commands:
- echo Logging in to Amazon ECR...
...
- echo Fetching pip.conf for PYPI
- aws codeartifact --region us-east-1 login --tool pip --repository ....
- cp ~/.config/pip/pip.conf .
build:
commands:
- docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
- docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
COPY
that file in, do the pip install
, then rm it
COPY requirements.txt pkg/
COPY --chown=myuser:myuser pip.conf /home/myuser/.config/pip/pip.conf
RUN pip install -r ./pkg/requirements.txt
RUN pip install ./pkg
RUN rm /home/myuser/.config/pip/pip.conf
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With