Async Lambda Function: Returning promise or sending responseURL does not terminate CloudFormation custom resource invocation

Question

I have a lambda function invoked as a custom resource via a CloudFormation template. It Creates/Deletes AWS Connect instances. The API calls work fine but I cannot seem to terminate the custom resource invocation, so the last CF block remains CREATE_IN_PROGRESS. No matter what I return from the async function it just won't terminate the CF execution with a success.

I'm able to use a non-async handler successfully as in https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-custom-resources-lambda-lookup-amiids.html but I need to make multiple API calls and await completions, hence the need for async handler.

Below is the code in it's simplest form, though I've tried just about everything, including using callback and context (ie exports.handler = async function(event, context, callback) {...}), both of which should be unnecessary with an async handler. I've tried using cfn-response to directly send a response which seems to be ignored with async handlers. I've tried returning directly the promises with and without the await before them, tried returning variables containing various responseStatus and responseData, nothing seems to work.

Transform: 'AWS::Serverless-2016-10-31'
Parameters:
  IdentityManagementType:
    Description: The type of identity management for your Amazon Connect users.
    Type: String
    AllowedValues: ["SAML", "CONNECT_MANAGED", "EXISTING_DIRECTORY"]
    Default: "SAML"
  InboundCallsEnabled:
    Description: Whether your contact center handles incoming contacts.
    Type: String
    AllowedValues: [true, false]
    Default: true
  InstanceAlias:
    Description: The name for your instance.
    Type: String
    MaxLength: 62
  OutboundCallsEnabled:
    Description: Whether your contact center allows outbound calls.
    Type: String
    AllowedValues: [true, false]
    Default: true
  DirectoryId:
    Description: Optional. The identifier for the directory, if using this type of Identity Management.
    Type: String
  ClientToken:
    Description: Optional. The idempotency token. Used for concurrent deployments
    Type: String
    MaxLength: 500
  Region:
    Description: Region to place the AWS Connect Instance
    Type: String
    Default: us-east-1
#Handler for optional values
Conditions:
  HasClientToken: !Not
    - !Equals
      - ""
      - !Ref ClientToken
  HasDirectoryId: !Not
    - !Equals
      - ""
      - !Ref DirectoryId

Resources:
  CreateConnectInstance:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub "${AWS::StackName}-AWSConnectInstance"
      Handler: index.handler
      Runtime: nodejs12.x
      Description: Invoke a function to create an AWS Connect instance.
      MemorySize: 128
      Timeout: 30
      Role: !GetAtt LambdaExecutionRole.Arn
      Layers:
        - !Sub "arn:aws:lambda:us-east-1:${AWS::AccountId}:layer:node_sdk:1"
      Environment:
        Variables:
          IdentityManagementType:
            Ref: IdentityManagementType
          InboundCallsEnabled:
            Ref: InboundCallsEnabled
          InstanceAlias:
            Ref: InstanceAlias
          OutboundCallsEnabled:
            Ref: OutboundCallsEnabled
          Region:
            Ref: Region
          #Optional Values
          ClientToken: !If
            - HasClientToken
            - !Ref ClientToken
            - !Ref "AWS::NoValue"
          DirectoryId: !If
            - HasClientToken
            - !Ref ClientToken
            - !Ref "AWS::NoValue"
      InlineCode: |
        var aws = require("aws-sdk");
        exports.handler = async function(event) {
            console.log("REQUEST RECEIVED:\n" + JSON.stringify(event));
            var connect = new aws.Connect({region: event.ResourceProperties.Region});
            var isInboundCallsEnabled = (process.env.InboundCallsEnabled == 'true');
            var isOutboundCallsEnabled = (process.env.OutboundCallsEnabled == 'true');
            var createInstanceParams = {
                InboundCallsEnabled: isInboundCallsEnabled,
                OutboundCallsEnabled: isOutboundCallsEnabled,
                IdentityManagementType: process.env.IdentityManagementType,
                ClientToken: process.env.ClientToken,
                DirectoryId: process.env.DirectoryId,
                InstanceAlias: process.env.InstanceAlias
            };

            // Create AWS Connect instance using specified parameters
            if (event.RequestType == "Create") {
                return await connect.createInstance(createInstanceParams).promise();
                // I can store this in a variable and read the contents fine, but...
                // returning the promise does not terminate execution
            }
        };


  InvokeCreateConnectInstance:
    Type: Custom::CreateConnectInstance
    Properties:
      ServiceToken: !GetAtt CreateConnectInstance.Arn
      Region: !Ref "AWS::Region"

The documentaiton at https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html explicitly states that you should be able to return await apiCall.promise() directly from any async function, exactly what I'm trying to do, such as

const s3 = new AWS.S3()

exports.handler = async function(event) {
  return s3.listBuckets().promise()
}

Why can't I return from my async function? Again the API calls are working, the Connect instances are created and deleted (though I've omitted the delete code for brevity), but CF just hangs hours and hours until eventually saying "Custom Resource failed to stabilize in expected time"

Here's the inline code by itself for readability:

        exports.handler = async function(event) {
            console.log("REQUEST RECEIVED:\n" + JSON.stringify(event));
            var connect = new aws.Connect({region: event.ResourceProperties.Region});
            var isInboundCallsEnabled = (process.env.InboundCallsEnabled == 'true');
            var isOutboundCallsEnabled = (process.env.OutboundCallsEnabled == 'true');
            var createInstanceParams = {
                InboundCallsEnabled: isInboundCallsEnabled,
                OutboundCallsEnabled: isOutboundCallsEnabled,
                IdentityManagementType: process.env.IdentityManagementType,
                ClientToken: process.env.ClientToken,
                DirectoryId: process.env.DirectoryId,
                InstanceAlias: process.env.InstanceAlias
            };

            // Create AWS Connect instance using specified parameters
            if (event.RequestType == "Create") {
                return await connect.createInstance(createInstanceParams).promise();
                // I can store this in a variable and read the contents fine, but...
                // returning the promise does not terminate CF execution
            }
          };

UPDATE: I've implemented the sendResponse method exactly as shown in the AMI lookup example (the first link) and am sending exactly the correct structure for the response, it even includes the newly created connect instance ID in the data field:

{
    "Status": "SUCCESS",
    "Reason": "See the details in CloudWatch Log Stream: 2020/12/23/[$LATEST]6fef3553870b4fba90479a37b4360cee",
    "PhysicalResourceId": "2020/12/23/[$LATEST]6fef3553870b4fba90479a37b4360cee",
    "StackId": "arn:aws:cloudformation:us-east-1:642608065726:stack/cr12/1105a290-4534-11eb-a6de-0a8534d05dcd",
    "RequestId": "2f7c3d9e-941f-402c-b739-d2d965288cfe",
    "LogicalResourceId": "InvokeCreateConnectInstance",
    "Data": {
        "InstanceId": "2ca7aa49-9b20-4feb-8073-5f23d63e4cbc"
    }
}

And STILL the custom resource will just not close in CloudFormation. I just don't understand why this is happening when I am returning the above to the event.responseURL. It's like specifying an async handler completely breaks the custom resource handler and prevents it from closing.

UPDATE: When I manually CURL the above response directly to the event.responseUrl the CF resource registers a success! WTF... I'm sending the exact same response as the lambda function is sending, and it accepts it from the CURL but not from my lambda function.

UPDATE: latest code including sendResponse, etc

var aws = require("aws-sdk");
exports.handler = async function(event, context, callback) {
    console.log("REQUEST RECEIVED:\n" + JSON.stringify(event));
    var connect = new aws.Connect({region: event.ResourceProperties.Region});
    var isInboundCallsEnabled = (process.env.InboundCallsEnabled == 'true');
    var isOutboundCallsEnabled = (process.env.OutboundCallsEnabled == 'true');
    var createInstanceParams = {
        InboundCallsEnabled: isInboundCallsEnabled,
        OutboundCallsEnabled: isOutboundCallsEnabled,
        IdentityManagementType: process.env.IdentityManagementType,
        ClientToken: process.env.ClientToken,
        DirectoryId: process.env.DirectoryId,
        InstanceAlias: process.env.InstanceAlias
    };
    var responseStatus;
    var responseData = {};

    // Create Connect instance
    if (event.RequestType == "Create") {
        try {
            var createInstanceRequest = await connect.createInstance(createInstanceParams).promise();
            responseStatus = "SUCCESS";
            responseData = {"InstanceId": createInstanceRequest.Id};
        } catch (err) {
            responseStatus = "FAILED";
            responseData = {Error: "CreateInstance failed"};
            console.log(responseData.Error + ":\n", err);
        }
        sendResponse(event, context, responseStatus, responseData);
        return;
    }

    // Look up the ID and call deleteInstance.
    if (event.RequestType == "Delete") {
        var instanceId;
        var listInstanceRequest = await connect.listInstances({}).promise();
        listInstanceRequest.InstanceSummaryList.forEach(instance => {
            if (instance.InstanceAlias == createInstanceParams.InstanceAlias) {
                instanceId = instance.Id;
            }
        });
        if (instanceId !== undefined) {
            try {
                var deleteInstanceRequest = await connect.deleteInstance({"InstanceId": instanceId}).promise();
                responseStatus = "SUCCESS";
                responseData = {"InstanceId": instanceId};
            } catch (err) {
                responseStatus = "FAILED";
                responseData = {Error: "DeleteInstance call failed"};
                console.log(responseData.Error + ":\n", err);
            }
        } else {
            responseStatus = "FAILED";
            responseData = {Error: "DeleteInstance failed; no match found"};
            console.log(responseData.Error);
        }
        sendResponse(event, context, responseStatus, responseData);
        return;
    }
};

// Send response to the pre-signed S3 URL 
function sendResponse(event, context, responseStatus, responseData) {
    var responseBody = JSON.stringify({
        Status: responseStatus,
        Reason: "CloudWatch Log Stream: " + context.logStreamName,
        PhysicalResourceId: context.logStreamName,
        StackId: event.StackId,
        RequestId: event.RequestId,
        LogicalResourceId: event.LogicalResourceId,
        Data: responseData
    });
    console.log("RESPONSE BODY:\n", responseBody);
    var https = require("https");
    var url = require("url");
    var parsedUrl = url.parse(event.ResponseURL);
    var options = {
        hostname: parsedUrl.hostname,
        port: 443,
        path: parsedUrl.path,
        method: "PUT",
        headers: {
            "content-type": "",
            "content-length": responseBody.length
        }
    };
    console.log("SENDING RESPONSE...\n");
    var request = https.request(options, function(response) {
        console.log("STATUS: " + response.statusCode);
        console.log("HEADERS: " + JSON.stringify(response.headers));
        // Tell AWS Lambda that the function execution is done  
        context.done();
    });
    request.on("error", function(error) {
        console.log("sendResponse Error:" + error);
        // Tell AWS Lambda that the function execution is done  
        context.done();
    });
    // write data to request body
    request.write(responseBody);
    request.end();
}

Been at this for two days now :(

PS in the logs the "RESPONSE BODY" is shown as expected like I copied above, and log shows the "SENDING RESPONSE" but does not get to the the "STATUS: " and "HEADERS: " portion of the request.https() call, which makes me think something with async interferes with this call... IDK

Answer 1

This one was really tricky but finally have everything figured out. I had to make the sendResponse function asynchronous by adding a promise to it, awaiting that promise and returning it. This allowed me to ultimately call "return await sendResponse(event, context, responseStatus, responseData);" and finally everything is working, both create and delete operations are successful and the CloudFormation custom resource completes as expected. Phew. Posting code here in hopes that others will benefit from it.

var aws = require("aws-sdk");
exports.handler = async function(event, context, callback) {
    console.log("REQUEST RECEIVED:\n" + JSON.stringify(event));
    var connect = new aws.Connect({region: event.ResourceProperties.Region});
    var isInboundCallsEnabled = (process.env.InboundCallsEnabled == 'true');
    var isOutboundCallsEnabled = (process.env.OutboundCallsEnabled == 'true');
    var createInstanceParams = {
        InboundCallsEnabled: isInboundCallsEnabled,
        OutboundCallsEnabled: isOutboundCallsEnabled,
        IdentityManagementType: process.env.IdentityManagementType,
        ClientToken: process.env.ClientToken,
        DirectoryId: process.env.DirectoryId,
        InstanceAlias: process.env.InstanceAlias
    };
    var responseStatus;
    var responseData = {};
    if (event.RequestType == "Create") {
        try {
            var createInstanceRequest = await connect.createInstance(createInstanceParams).promise();
            responseStatus = "SUCCESS";
            responseData = {"InstanceId": createInstanceRequest.Id};
        } catch (err) {
            responseStatus = "FAILED";
            responseData = {Error: "CreateInstance failed"};
            console.log(responseData.Error + ":\n", err);
        }
        return await sendResponse(event, context, responseStatus, responseData);
    }

    if (event.RequestType == "Delete") {
        var instanceId;
        var listInstanceRequest = await connect.listInstances({}).promise();
        listInstanceRequest.InstanceSummaryList.forEach(instance => {
            if (instance.InstanceAlias == createInstanceParams.InstanceAlias) {
                instanceId = instance.Id;
            }
        });
        if (instanceId !== undefined) {
            try {
                var deleteInstanceRequest = await connect.deleteInstance({"InstanceId": instanceId}).promise();
                responseStatus = "SUCCESS";
                responseData = {"InstanceId": instanceId};
            } catch (err) {
                responseStatus = "FAILED";
                responseData = {Error: "DeleteInstance call failed"};
                console.log(responseData.Error + ":\n", err);
            }
        } else {
            responseStatus = "FAILED";
            responseData = {Error: "DeleteInstance failed; no match found"};
            console.log(responseData.Error);
        }
        return await sendResponse(event, context, responseStatus, responseData);
    }
};

async function sendResponse(event, context, responseStatus, responseData) {
    let responsePromise = new Promise((resolve, reject) => {
        var responseBody = JSON.stringify({
            Status: responseStatus,
            Reason: "CloudWatch Log Stream: " + context.logStreamName,
            PhysicalResourceId: context.logStreamName,
            StackId: event.StackId,
            RequestId: event.RequestId,
            LogicalResourceId: event.LogicalResourceId,
            Data: responseData
        });
        console.log("RESPONSE BODY:\n", responseBody);
        var https = require("https");
        var url = require("url");
        var parsedUrl = url.parse(event.ResponseURL);
        var options = {
            hostname: parsedUrl.hostname,
            port: 443,
            path: parsedUrl.path,
            method: "PUT",
            headers: {
                "content-type": "",
                "content-length": responseBody.length
            }
        };
        console.log("SENDING RESPONSE...\n");
        var request = https.request(options, function(response) {
            console.log("STATUS: " + response.statusCode);
            console.log("HEADERS: " + JSON.stringify(response.headers));
            resolve(JSON.parse(responseBody));
            context.done();
        });
        request.on("error", function(error) {
            console.log("sendResponse Error:" + error);
            reject(error);
            context.done();
        });
        request.write(responseBody);
        request.end();
    });
    return await responsePromise;
}

Answer 2

This answer is a variant on the OP's answer for those using the "ZipFile" option in the "Code" property of an AWS::Lambda::Function resource in CloudFormation. The advantage of the ZipFile approach is that in addition permitting Lambda code inlined into the CF template, it also automatically bundles a "cfn-response.js" function very similar to the "async function sendResponse" in the OP's answer. With the insight gained from the OP's answer regarding a promised response (thank you, I was stuck and perplexed), this is how I incorporated the cfn-response function as an await-able Promise to signal CF after my asynchronous AWS API calls (omitted for brevity) were complete:

CreateSnapshotFunction:
    Type: AWS::Lambda::Function
    Properties:
        Runtime: nodejs12.x
        Handler: index.handler
        Timeout: 900 # 15 mins
        Code:
            ZipFile: !Sub |
                const resp = require('cfn-response');
                const aws = require('aws-sdk');
                const cf = new aws.CloudFormation({apiVersion: '2010-05-15'});
                const rds = new aws.RDS({apiVersion: '2014-10-31'});

                exports.handler = async function(evt, ctx) {
                    if (evt.RequestType == "Create") {
                        try {
                            // Query the given CF stack, determine its database
                            // identifier, create a snapshot of the database,
                            // and await an "available" status for the snapshot
                            let stack = await getStack(stackNameSrc);
                            let srcSnap = await createSnapshot(stack);
                            let pollFn = () => describeSnapshot(srcSnap.DBSnapshot.DBSnapshotIdentifier);
                            let continueFn = snap => snap.DBSnapshots[0].Status !== 'available';
                            await poll(pollFn, continueFn, 10, 89); // timeout after 14 min, 50 sec

                            // Send response to CF
                            await send(evt, ctx, resp.SUCCESS, {
                                SnapshotId: srcSnap.DBSnapshot.DBSnapshotIdentifier,
                                UpgradeRequired: upgradeRequired
                            });
                        } catch(err) {
                            await send(evt, ctx, resp.FAILED, { ErrorMessage: err } );
                        }
                    } else {
                        // Send success to CF for delete and update requests
                        await send(evt, ctx, resp.SUCCESS, {});
                    }
                };

                function send(evt, ctx, status, data) {
                    return new Promise(() => { resp.send(evt, ctx, status, data) });
                }