Upload entire Bitbucket repo to S3 using Bitbucket Pipeline

Question

I'm using Bitbuckets Pipeline. I want it to push the entire contents of my repo (very small) to S3. I don't want to have to zip it up, push to S3 and then unzip things. I just want it to take the existing file/folder structure in my Bitbucket repo and push that to S3.

What should the yaml file and .py file look like to accomplish this?

Here is the current yaml file:

image: python:3.5.1

pipelines:
  branches:
    master:
      - step:
          script:
            # - apt-get update # required to install zip
            # - apt-get install -y zip # required if you want to zip repository objects
            - pip install boto3==1.3.0 # required for s3_upload.py
            # the first argument is the name of the existing S3 bucket to upload the artefact to
            # the second argument is the artefact to be uploaded
            # the third argument is the the bucket key
            # html files
            - python s3_upload.py my-bucket-name html/index_template.html html/index_template.html # run the deployment script
            # Example command line parameters. Replace with your values
            #- python s3_upload.py bb-s3-upload SampleApp_Linux.zip SampleApp_Linux # run the deployment script

And here is my current python:

from __future__ import print_function
import os
import sys
import argparse
import boto3
from botocore.exceptions import ClientError

def upload_to_s3(bucket, artefact, bucket_key):
    """
    Uploads an artefact to Amazon S3
    """
    try:
        client = boto3.client('s3')
    except ClientError as err:
        print("Failed to create boto3 client.\n" + str(err))
        return False
    try:
        client.put_object(
            Body=open(artefact, 'rb'),
            Bucket=bucket,
            Key=bucket_key
        )
    except ClientError as err:
        print("Failed to upload artefact to S3.\n" + str(err))
        return False
    except IOError as err:
        print("Failed to access artefact in this directory.\n" + str(err))
        return False
    return True


def main():

    parser = argparse.ArgumentParser()
    parser.add_argument("bucket", help="Name of the existing S3 bucket")
    parser.add_argument("artefact", help="Name of the artefact to be uploaded to S3")
    parser.add_argument("bucket_key", help="Name of the S3 Bucket key")
    args = parser.parse_args()

    if not upload_to_s3(args.bucket, args.artefact, args.bucket_key):
        sys.exit(1)

if __name__ == "__main__":
    main()

This requires me to list every single file in the repo in the yaml file as another command. I just want it to grab everything and upload it to S3.

Answer 1

You can change to use docker https://hub.docker.com/r/abesiyo/s3/

It runs quite well

bitbucket-pipelines.yml

image: abesiyo/s3

pipelines:
    default:
       - step:
          script:
             - s3 --region "us-east-1" rm s3://<bucket name>
             - s3 --region "us-east-1" sync . s3://<bucket name> 

Please also setup environment variables on bitbucket pipelines AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

Answer 2

Figured it out myself. Here is the python file, 's3_upload.py'

from __future__ import print_function
import os
import sys
import argparse
import boto3
#import zipfile
from botocore.exceptions import ClientError

def upload_to_s3(bucket, artefact, is_folder, bucket_key):
    try:
        client = boto3.client('s3')
    except ClientError as err:
        print("Failed to create boto3 client.\n" + str(err))
        return False
    if is_folder == 'true':
        for root, dirs, files in os.walk(artefact, topdown=False):
            print('Walking it')
            for file in files:
                #add a check like this if you just want certain file types uploaded
                #if file.endswith('.js'):
                try:
                    print(file)
                    client.upload_file(os.path.join(root, file), bucket, os.path.join(root, file))
                except ClientError as err:
                    print("Failed to upload artefact to S3.\n" + str(err))
                    return False
                except IOError as err:
                    print("Failed to access artefact in this directory.\n" + str(err))
                    return False
                #else:
                #    print('Skipping file:' + file)
    else:
        print('Uploading file ' + artefact)
        client.upload_file(artefact, bucket, bucket_key)
    return True


def main():

    parser = argparse.ArgumentParser()
    parser.add_argument("bucket", help="Name of the existing S3 bucket")
    parser.add_argument("artefact", help="Name of the artefact to be uploaded to S3")
    parser.add_argument("is_folder", help="True if its the name of a folder")
    parser.add_argument("bucket_key", help="Name of file in bucket")
    args = parser.parse_args()

    if not upload_to_s3(args.bucket, args.artefact, args.is_folder, args.bucket_key):
        sys.exit(1)

if __name__ == "__main__":
    main()

and here is they bitbucket-pipelines.yml file:

---
image: python:3.5.1

pipelines:
  branches:
    dev:
      - step:
          script:
            - pip install boto3==1.4.1 # required for s3_upload.py
            - pip install requests
            # the first argument is the name of the existing S3 bucket to upload the artefact to
            # the second argument is the artefact to be uploaded
            # the third argument is if the artefact is a folder
            # the fourth argument is the bucket_key to use
            - python s3_emptyBucket.py dev-slz-processor-repo
            - python s3_upload.py dev-slz-processor-repo lambda true lambda
            - python s3_upload.py dev-slz-processor-repo node_modules true node_modules
            - python s3_upload.py dev-slz-processor-repo config.dev.json false config.json
    stage:
      - step:
          script:
            - pip install boto3==1.3.0 # required for s3_upload.py
            - python s3_emptyBucket.py staging-slz-processor-repo
            - python s3_upload.py staging-slz-processor-repo lambda true lambda
            - python s3_upload.py staging-slz-processor-repo node_modules true node_modules
            - python s3_upload.py staging-slz-processor-repo config.staging.json false config.json
    master:
      - step:
          script:
            - pip install boto3==1.3.0 # required for s3_upload.py
            - python s3_emptyBucket.py prod-slz-processor-repo
            - python s3_upload.py prod-slz-processor-repo lambda true lambda
            - python s3_upload.py prod-slz-processor-repo node_modules true node_modules
            - python s3_upload.py prod-slz-processor-repo config.prod.json false config.json

As an example for the dev branch, it grabs everything in the "lambda" folder, walks the entire structure of that folder, and for each item it finds, it uploads it to the dev-slz-processor-repo bucket

Lastly, here is a little helpful function, 's3_emptyBucket', to remove all objects from the bucket before uploading the new ones:

from __future__ import print_function
import os
import sys
import argparse
import boto3
#import zipfile
from botocore.exceptions import ClientError

def empty_bucket(bucket):
    try:
        resource = boto3.resource('s3')
    except ClientError as err:
        print("Failed to create boto3 resource.\n" + str(err))
        return False
    print("Removing all objects from bucket: " + bucket)
    resource.Bucket(bucket).objects.delete()
    return True


def main():

    parser = argparse.ArgumentParser()
    parser.add_argument("bucket", help="Name of the existing S3 bucket to empty")
    args = parser.parse_args()

    if not empty_bucket(args.bucket):
        sys.exit(1)

if __name__ == "__main__":
    main()

Answer 3

Atlassian now offers "Pipes" to simplify configuration of some common tasks. There's one for S3 upload as well.

No need to specify a different image type:

image: node:8

pipelines:
  branches:
    master:
      - step:
          script:
            - pipe: atlassian/aws-s3-deploy:0.2.1
              variables:
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
                AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
                AWS_DEFAULT_REGION: "us-east-1"
                S3_BUCKET: "your.bucket.name"
                LOCAL_PATH: "dist"