How do you specify GitHub access token with CodeBuild from CloudFormation

Question

I have been wasting a lot of time trying to get a GitHub web hook setup with CloudFormation. The docs for this process are beyond useless, for example: https://docs.aws.amazon.com/codebuild/latest/userguide/sample-access-tokens.html

$ aws codebuild import-source-credentials --generate-cli-skeleton
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument operation: Invalid choice, valid choices are:

batch-delete-builds                      | batch-get-builds
batch-get-projects                       | create-project
create-webhook                           | delete-project
delete-webhook                           | invalidate-project-cache
list-builds                              | list-builds-for-project
list-curated-environment-images          | list-projects
start-build                              | stop-build
update-project                           | help

My issue is that I cannot find a way to specify a GitHub Access Token for CodeBuild with CloudFormation. I am simply trying to setup a web hook for a github repository to run a simple test suite when a pull request is created, updated, etc. As previously mentioned, I have found a lot of half-baked documentation like https://docs.aws.amazon.com/codebuild/latest/userguide/sample-github-pull-request.html that outlines how to setup a web hook with github, however, when I try and follow these guides I get a no GitHub token error in CloudFormation. Other documents say you need to set the access token from the UI or from the CLI, but the CLI is clearly broken and why would I create the CodeBuild resource in the management console when I am trying to setup cloud formation? I see no where in the example CloudFormation template to include the personal access token from GitHub and the documentation for the Source > Auth element is defined in terms of itself. "The resource value that applies to the specified authorization type" tells me nothing about what this "resource" is. Is this the GitHub personal access token that I have spent the last 8 hours looking for in the docs? I have no idea. I did try sticking my personal access token in that field and I get the same result. "No Access token found, please visit AWS CodeBuild console to connect to GitHub"

The following is my CloudFormation template:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "AWS CodeBuild Template",
    "Parameters": {
    },
    "Resources": {
        "CodeBuildProject": {
            "Type": "AWS::CodeBuild::Project",
            "Properties": {
                "Name": "TestingCodeBuild",
                "Description": "A description about my project",
                "ServiceRole": { "Fn::GetAtt": [ "CodeBuildServiceRole", "Arn" ] },
                "Artifacts": {
                    "Type": "no_artifacts"
                },
                "Environment": {
                    "Type": "LINUX_CONTAINER",
                    "ComputeType": "BUILD_GENERAL1_SMALL",
                    "Image": "ubuntu:bionic",
                    "EnvironmentVariables": [
                      {
                        "Name": "varName",
                        "Value": "varValue"
                      }
                    ]
                },
                "Source": {
                    "Auth" : {
                        "Resource": "WTF IS THIS VALUE, Docs say a resource is a resource for use with the type."
                        "Type" : "OAUTH"
                    },
                    "BuildSpec" : "buildspec.yml",
                    "GitCloneDepth" : 1,
                    "ReportBuildStatus" : true,
                    "Location" : "https://github.com/user/repo.git",
                    "Type" : "GITHUB"
                },
                "Triggers": {
                    "FilterGroups": [
                        [
                            {
                                "Pattern" : "PULL_REQUEST_CREATED, PULL_REQUEST_UPDATED, PULL_REQUEST_REOPENED",
                                "Type" : "EVENT"
                            }
                        ]
                    ],
                    "Webhook" : true
                },
                "TimeoutInMinutes": 10,
                "Tags": [
                    {
                      "Key": "Key1",
                      "Value": "Value1"
                    },
                    {
                      "Key": "Key2",
                      "Value": "Value2"
                    }
                ]
            }
        },
        "CodeBuildServiceRole": {
            "Type": "AWS::IAM::Role",
            "Properties": {
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": [
                                    "codebuild.amazonaws.com"
                                ]
                            },
                            "Action": "sts:AssumeRole"
                        }
                    ]
                },
                "Path": "/",
                "Policies": [
                    {
                        "PolicyName": "CodeBuildAccessPolicies",
                        "PolicyDocument": {
                            "Version": "2012-10-17",
                            "Statement": [
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "codecommit:CancelUploadArchive",
                                        "codecommit:GetBranch",
                                        "codecommit:GetCommit",
                                        "codecommit:GetUploadArchiveStatus",
                                        "codecommit:UploadArchive"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "codedeploy:CreateDeployment",
                                        "codedeploy:GetApplicationRevision",
                                        "codedeploy:GetDeployment",
                                        "codedeploy:GetDeploymentConfig",
                                        "codedeploy:RegisterApplicationRevision"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "codebuild:BatchGetBuilds",
                                        "codebuild:StartBuild"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "devicefarm:ListProjects",
                                        "devicefarm:ListDevicePools",
                                        "devicefarm:GetRun",
                                        "devicefarm:GetUpload",
                                        "devicefarm:CreateUpload",
                                        "devicefarm:ScheduleRun"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "iam:PassRole"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "elasticbeanstalk:*",
                                        "ec2:*",
                                        "elasticloadbalancing:*",
                                        "autoscaling:*",
                                        "cloudwatch:*",
                                        "s3:*",
                                        "sns:*",
                                        "cloudformation:*",
                                        "rds:*",
                                        "sqs:*",
                                        "ecs:*"
                                    ],
                                    "Resource": "*"
                                }
                            ]
                        }
                    }
                ]
            }
        }
    }
}

Update So I managed to get it to connect to GitHub by manually creating an unrelated CodeDeploy project called "TempProj" and connecting that to GitHub. Now it can automagically connect to GitHub when you create stacks in Cloud Formation. You can even delete the instance and it will just continue to work.

Answer 1

AWS::CodeBuild::SourceCredential is a new AWS resource, appeared in CloudFormation Resource Specification v5.1.0, that lets you connect CodeBuild with Github using Github's Personal Access Token (if you do not know how to create it, check out this quick guide).

Quick Example:

AWSTemplateFormatVersion: "2010-09-09"

Resources:
  # This resource allows to connect CodeBuild with Github using Personal Access Token.
  CodeBuildSourceCredential:
    Type: AWS::CodeBuild::SourceCredential
    Properties:
      AuthType: PERSONAL_ACCESS_TOKEN
      ServerType: GITHUB
      Token: "<YOUR-PERSONAL-GITHUB-ACCESS-TOKEN>"

  # CodeBuild resource.
  CodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Source:
        Auth:
          Resource: !Ref CodeBuildSourceCredential
          Type: OAUTH

Tip: Store personal access token in AWS Secrets Manager and fetch it Using Dynamic References to Specify Template Values.

Answer 2

You can use AWS Secrets Manager to securely store your GitHub OAuth Token, then you can use a dynamic reference in your CloudFormation template which will resolve to the stored value.

Here's a link to the docs: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager

When creating a secret with Secrets Manager, there are 3 parts to one secret:

• Secret Name (the label for this secret e.g. GitHubToken)
• Secret Key e.g. OAuthToken
• Secret Value (the actual thing you want to store)

The above example would be referenced in your CloudFormation template as:

'{{resolve:secretsmanager:GitHubToken:SecretString:OAuthToken}}'

A fuller snippet from a CloudFormation template for a CodePipeline that will trigger the pipeline to run after every git push to the specified branch of your repo is shown below:

...
MyPipeline:
  Type: AWS::CodePipeline::Pipeline
  Properties:
    Stages:
      -
        Name: GetSource
        Actions:
          -
            Name: GetMyGithubRepoSourceOnPush
            ActionTypeId:
              Category: Source
              Owner: ThirdParty
              Version: 1
              Provider: GitHub
            OutputArtifacts:
              - Name: NameOfArtifactForNextStages
            Configuration:
              Owner: MyGithubUsername
              Repo: MyGithubRepoName
              Branch: MyRepoBranchName
              OAuthToken: '{{resolve:secretsmanager:NameOfSecret:SecretString:KeyOfSecret}}'

I hope that helps.