AWS SNS is bypassing API Gateway and calling Lamba functions directly

Question

I have stumbled upon a rather strange and, to my knowledge, undocumented behaviour of Amazon SNS. I am looking for a solution or settings to fix it.

SUMMARY

I have a SNS Topic, with an HTTPS subscription pointing to an Amazon API Gateway REST endpoint, backed with a Node.js Lambda function for executing the request.

Now, if I use SNS and Publish on the topic , the whole API Gateway mapping template gets ignored/short-circuited. The Lambda function ends-up receiving ONLY the original SNS JSON object.

However, if I use a web browser (or curl) to access the endpoint the API Gateway Mapping Translation gets called and the proper JSON data gets passed onto the Lambda function.

THE API Gateway Endpoint

The API Gateway (aka TheApi hereafter) is created with a sms resource under which a "path parameters" {phone}. Therefore, you can query https://TheApi/sms/111-222-3333 with either a POST or GET method.

Both methods have a generic Mapping Template which grabs all paths parameters, all headers parameters, all query parameters and the whole request body and translate that into one LARGE request body JSON object. Here is what the template looks like:

{
    "resource-path" : "$context.resourcePath",
    "http-method" : "$context.httpMethod",
    "headers": {
    #foreach($param in $input.params().header.keySet())
      "$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
    #end
    },
    "query": {
    #foreach($param in $input.params().querystring.keySet())
      "$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
    #end
    },
    "paths": {
    #foreach($param in $input.params().path.keySet())
      "$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end

    #end
    },
   "body" : $input.json('$')
}

This resulting object is then fed to the Lambda function as the event on which the lambda function operates. Here is the result of a simple API Gateway "Test":

Tue Feb 09 00:54:13 UTC 2016 : Endpoint request body after transformations:
{
    "resource-path" : "/sms/{phone}",
    "http-method" : "POST",
    "headers": {
        },
    "query": {
        },
    "paths": {
          "phone": "111-222-3333" 
        },
   "body" : {"foo":"bar","Alice":"Bob"}
}

This endpoint and the called Lambda function work flawlessly when called from a Web Browser (or a curl call ). AWS Cloud Watch logs show that everything is good under the sun, the Lambda event received is the same as above, therefore the Mapping translation is called.

THE Problem

Now, if I use SNS and Publish on the topic (the one with the HTTPS subscription on the API Gateway endpoint listed at the top), the whole API Gateway mapping template gets ignored/short-circuited.

The Lambda function ends-up receiving ONLY the original SNS JSON object and none of that custom mapping I wrote. The Lambda function doesn't receive any information about the calling agent, the requested url, the headers.... nada!! Here what the Lambda event looks like, as shown in CloudWatch:

{
    "Type": "Notification",
    "MessageId": "d38077e1-406a-5122-8a57-38cecfc635fd",
    "TopicArn": "arn:aws:sns:us-east-1:...:...",
    "Subject": "Ceci est un test",
    "Message": "Ceci est un message de test.",
    "Timestamp": "2016-02-06T06:06:36.649Z",
    "SignatureVersion": "1",
    "Signature": "...",
    "SigningCertURL": "...",
    "MessageAttributes": {
        "AWS.SNS.MOBILE.MPNS.Type": {
            "Type": "String",
            "Value": "token"
        },
        "AWS.SNS.MOBILE.MPNS.NotificationClass": {
            "Type": "String",
            "Value": "realtime"
        },
        "AWS.SNS.MOBILE.WNS.Type": {
            "Type": "String",
            "Value": "wns/badge"
        }
    }
}

As it can be seen, this JSON object is completely different.

FOOD for thoughts

1. Some may wonder: "Why I go to the trouble of making the API Gateway when I could forward SNS events directly to the Lambda functions?". The reason is quite simple, I need to attach additional information with the SNS message, in this case the phone number to send the message to. Using API Gateway I can create as many subscriptions to as many phone numbers without duplicating any code.

2. Others may wonder: "Why not using the SMS subscription built into SNS instead of making my own?". For one thing, I'm in Canada and Amazon SMS subscriptions no longer works in Canada. Secondly, I may wish to use another SMS service that Amazon's.

3. As it turns out, SNS topics can call Lambda functions directly. In which case, the SNS JSON object is exactly the same. Thus, it is as though AWS is detecting the HTTPS endpoint domain, resolving the underlying Lambda function and routing the call directly to the Lambda function without passing through the API Gateway services.

4. In fact, when I build another REST endpoint on another domain I control, I indeed receive the POST request with the SNS JSON body, which I can forward to the API Gateway endpoint and it gets translated just well.

Just like this:

{
    "resource-path": "/sms/{phone}",
    "http-method": "POST",
    "headers": {
        "Accept": "*/*",
        "CloudFront-Forwarded-Proto": "https",
        "CloudFront-Is-Desktop-Viewer": "true",
        "CloudFront-Is-Mobile-Viewer": "false",
        "CloudFront-Is-SmartTV-Viewer": "false",
        "CloudFront-Is-Tablet-Viewer": "false",
        "CloudFront-Viewer-Country": "US",
        "Content-Type": "application/json",
        "Via": "1.1 c903e93e57c533ecd52152e4407a295e.cloudfront.net (CloudFront)",
        "X-Amz-Cf-Id": "Fy_dCf5yJbW1GOZWJMVJqhbz1qt6sLfNO0N33FqAtf56X1tB4py8Ig==",
        "X-Forwarded-For": "69.65.27.156, 54.182.212.5",
        "X-Forwarded-Port": "443",
        "X-Forwarded-Proto": "https"
    },
    "query": {},
    "paths": {
        "phone": "14184901585"
    },
    "body": {
        "Type": "Notification",
        "MessageId": "d38077e1-406a-5122-8a57-38cecfc635fd",
        "TopicArn": "arn:aws:sns:us-east-1:...:...",
        "Subject": "Ceci est un test",
        "Message": "Ceci est un message de test.",
        "Timestamp": "2016-02-06T06:06:36.649Z",
        "SignatureVersion": "1",
        "Signature": "...",
        "SigningCertURL": "...",
        "UnsubscribeURL": "...",
        "MessageAttributes": {
            "AWS.SNS.MOBILE.MPNS.Type": {
                "Type": "String",
                "Value": "token"
            },
            "AWS.SNS.MOBILE.MPNS.NotificationClass": {
                "Type": "String",
                "Value": "realtime"
            },
            "AWS.SNS.MOBILE.WNS.Type": {
                "Type": "String",
                "Value": "wns/badge"
            }
        }
    }
}

CALL for help

Are there any hidden settings somewhere when I can make this SNS -> API Gateway -> Lambda work with the proper Mapping Translation?

Answer 1

The mapping template is applied based on the request's content type. If there is no content type specified in the request, it defaults to 'application/json'.

Based on your description, I would assume that your mapping template is set up with the content type 'application/json'. This works fine as long as the client doesn't specify a different content type in its request (which is the case for browsers for example).

Since SNS sends the request with a header 'Content-type: text/plain' (SNS Send Message Over HTTP), it doesn't match your mapping template's content type and hence will ignore it. To get working you could either change the content type in your current mapping or add another one which matches 'text/plain'.

For some more details you could also have a look here in the AWS forums: Default Content-Type for Mapping Template

Best,

Jurgen