AWS - Moving data from one S3 bucket to another with CloudFormation

Question

I'm trying to create a stack with CloudFormation. The stack needs to take some data files from a central S3 bucket and copy them to it's own "local" bucket.

I've written a lambda function to do this, and it works when I run it in the Lambda console with a test event (the test event uses the real central repository and successfully copies the file to a specified repo).

My current CloudFormation script does the following things:

1. Creates the "local" S3 bucket
2. Creates a role that the Lambda function can use to access the buckets
3. Defines the Lambda function to move the specified file to the "local" bucket
4. Defines some Custom resources to invoke the Lambda function.

It's at step 4 where it starts to go wrong - the Cloudformation execution seems to freeze here (CREATE_IN_PROGESS). Also, when I try to delete the stack, it seems to just get stuck on DELETE_IN_PROGRESS instead.

Here's how I'm invoking the Lambda function in the CloudFormation script:

"DataSync": {
    "Type": "Custom::S3DataSync",
    "Properties": {
        "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] },
        "InputFile": "data/provided-as-ip-v6.json",
        "OutputFile": "data/data.json"
    }
},
"KeySync1": {
    "Type": "Custom::S3DataSync",
    "Properties": {
        "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] },
        "InputFile": "keys/1/public_key.pem"
    }
},
"KeySync2": {
    "Type": "Custom::S3DataSync",
    "Properties": {
        "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] },
        "InputFile": "keys/2/public_key.pem"
    }
}

And the Lambda function itself:

exports.handler = function(event, context) {
    var buckets = {};
    buckets.in = {
        "Bucket":"central-data-repository",
        "Key":"sandbox" + "/" + event.ResourceProperties.InputFile
    };   
    buckets.out = {       
        "Bucket":"sandbox-data",       
        "Key":event.ResourceProperties.OutputFile || event.ResourceProperties.InputFile   
    };   

    var AWS = require('aws-sdk');   
    var S3 = new AWS.S3();   

    S3.getObject(buckets.in, function(err, data) {       
        if (err) {           
            console.log("Couldn't get file " + buckets.in.Key);           
            context.fail("Error getting file: " + err)       
        }       
        else {           
            buckets.out.Body = data.Body;           
            S3.putObject(buckets.out, function(err, data) {               
                if (err) {                   
                    console.log("Couln't write to S3 bucket " + buckets.out.Bucket);                   
                    context.fail("Error writing file: " + err);               
                }               
                else {                   
                    console.log("Successfully copied " + buckets.in.Key + " to " + buckets.out.Bucket + " at " + buckets.out.Key);                   
                    context.succeed();               
                }           
            });       
        }   
    });
}

Answer 1

Your Custom Resource function needs to send signals back to CloudFormation to indicate completion, status, and any returned values. You will see CREATE_IN_PROGRESS as the status in CloudFormation until you notify it that your function is complete.

The generic way of signaling CloudFormation is to post a response to a pre-signed S3 URL. But there is a cfn-response module to make this easier in Lambda functions. Interestingly, the two examples provided for Lambda-backed Custom Resources use different methods:

• Walkthrough: Refer to Resources in Another Stack - uses the cfn-response module
• Walkthrough: Looking Up Amazon Machine Image IDs - uses pre-signed URLs.