AWS SES S3 process inbound email

Question

I'm working on a publish by email system based on AWS SES. For all incoming emails I've set routing to save messages in an S3 bucket so I can asynchronously process them. The problem I have is that the messages are saved in the S3 bucket in a raw format: headers, email body, etc + the encrypted attachment (a huge string) - all in a single file.

Is there a way to break the email message apart form the attachment and save both in separate files at AWS SES level? I'm trying to get the data in the format I need straight from AWS and avoid adding another processing step to the process.

If AWS SES doesn't provide such a feature, what would be the proper way to process these messages to obtain the result described above?

Answer 1

For anyone coming back later on to this question, this is the link to the JSON structure that you get when you invoke a Lambda function from SES.

http://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-notifications-examples.html

It took some searching to arrive at that page ;-)

From the link, a Lambda notification would look like this,

{
"notificationType": "Received",
"receipt": {
    "timestamp": "2015-09-11T20:32:33.936Z",
    "processingTimeMillis": 406,
    "recipients": [
        "[email protected]"
    ],
    "spamVerdict": {
        "status": "PASS"
    },
    "virusVerdict": {
        "status": "PASS"
    },
    "spfVerdict": {
        "status": "PASS"
    },
    "dkimVerdict": {
        "status": "PASS"
    },
    "action": {
        "type": "S3",
        "topicArn": "arn:aws:sns:us-east-1:012345678912:example-topic",
        "bucketName": "my-S3-bucket",
        "objectKey": "\email"
    }
},
"mail": {
    "timestamp": "2015-09-11T20:32:33.936Z",
    "source": "0000014fbe1c09cf-7cb9f704-7531-4e53-89a1-5fa9744f5eb6-000000@amazonses.com",
    "messageId": "d6iitobk75ur44p8kdnnp7g2n800",
    "destination": [
        "[email protected]"
    ],
    "headersTruncated": false,
    "headers": [
        {
            "name": "Return-Path",
            "value": "<0000014fbe1c09cf-7cb9f704-7531-4e53-89a1-5fa9744f5eb6-000000@amazonses.com>"
        },
        {
            "name": "Received",
            "value": "from a9-183.smtp-out.amazonses.com (a9-183.smtp-out.amazonses.com [54.240.9.183]) by inbound-smtp.us-east-1.amazonaws.com with SMTP id d6iitobk75ur44p8kdnnp7g2n800 for [email protected]; Fri, 11 Sep 2015 20:32:33 +0000 (UTC)"
        },
        {
            "name": "DKIM-Signature",
            "value": "v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1442003552; h=From:To:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Date:Message-ID:Feedback-ID; bh=DWr3IOmYWoXCA9ARqGC/UaODfghffiwFNRIb2Mckyt4=; b=p4ukUDSFqhqiub+zPR0DW1kp7oJZakrzupr6LBe6sUuvqpBkig56UzUwc29rFbJF hlX3Ov7DeYVNoN38stqwsF8ivcajXpQsXRC1cW9z8x875J041rClAjV7EGbLmudVpPX 4hHst1XPyX5wmgdHIhmUuh8oZKpVqGi6bHGzzf7g="
        },
        {
            "name": "From",
            "value": "[email protected]"
        },
        {
            "name": "To",
            "value": "[email protected]"
        },
        {
            "name": "Subject",
            "value": "Example subject"
        },
        {
            "name": "MIME-Version",
            "value": "1.0"
        },
        {
            "name": "Content-Type",
            "value": "text/plain; charset=UTF-8"
        },
        {
            "name": "Content-Transfer-Encoding",
            "value": "7bit"
        },
        {
            "name": "Date",
            "value": "Fri, 11 Sep 2015 20:32:32 +0000"
        },
        {
            "name": "Message-ID",
            "value": "<[email protected]>"
        },
        {
            "name": "X-SES-Outgoing",
            "value": "2015.09.11-54.240.9.183"
        },
        {
            "name": "Feedback-ID",
            "value": "1.us-east-1.Krv2FKpFdWV+KUYw3Qd6wcpPJ4Sv/pOPpEPSHn2u2o4=:AmazonSES"
        }
    ],
    "commonHeaders": {
        "returnPath": "0000014fbe1c09cf-7cb9f704-7531-4e53-89a1-5fa9744f5eb6-000000@amazonses.com",
        "from": [
            "[email protected]"
        ],
        "date": "Fri, 11 Sep 2015 20:32:32 +0000",
        "to": [
            "[email protected]"
        ],
        "messageId": "<[email protected]>",
        "subject": "Example subject"
    }
}
}

Answer 2

It doesn't look possible to have SES automatically split up the email for you. As per the documentation here:

Amazon SES provides you the raw, unmodified email, which is typically in Multipurpose Internet Mail Extensions (MIME) format.

I would use S3 or SNS to trigger a Lambda function whenever SES puts a new email file to S3. The Lambda function could split the file however you wish, then write those new files to another S3 bucket.

Answer 3

Regarding the question on how to write a Lambda. Here is a portion of our Lambda. The main thing to take out of it is the parseEvent function. and data.event.Records[0] which will give you details

exports.handler = function(event, context, callback) {

    var AWS = require('aws-sdk');

    // Validate characteristics of a SES event record.
    if (!event ||
      !event.hasOwnProperty('Records') ||
      event.Records.length !== 1 ||
      event.Records[0].hasOwnProperty('eventSource') ||
      event.Records[0].eventSource !== 'aws:ses' ||
      event.Records[0].eventVersion !== '1.0') {
      callback(null, {'disposition':'STOP_RULE_SET'});      
    }

    email = data.event.Records[0].ses.mail;
    subjectLine = event.Records[0].ses.mail.commonHeaders.subject;
}

The key is the event.Record[0].ses.mail. Unfortunately, I can't find the structure of it via a Google search, I am sure I had seen it before.