AWS put-bucket-notification-configuration for SQS throws "Unable to validate the following destination configurations"

Question

I want to send s3:CreateObject:* events to a SQS Queue. But setting up the notification configuration results in A client error (InvalidArgument) occurred when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations

This is the how I created the bucket:

aws s3api create-bucket --profile default --bucket my-bucket --create-bucket-configuration LocationConstraint=eu-west-1

This is the how I created the SQS Queue

aws sqs create-queue --profile default --queue-name my-queue --attributes file://attributes.json

with the attributes.json file

{
  "DelaySeconds":"0",
  "MessageRetentionPeriod":"3600",
  "Policy":"{\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":[\"sqs:SendMessage\",\"sqs:ReceiveMessage\"],\"Condition\":{\"ArnLike\": {\"aws:SourceArn\": \"arn:aws:s3:*:*:my-bucket\"}}}]}"
}

And finally the try to set up the notification which throws the error message I listed above:

aws s3api put-bucket-notification-configuration --profile default --bucket my-bucket --notification-configuration file://notification.json`

with the notification.json file

{
  "TopicConfigurations": [
  ],
  "QueueConfigurations": [
    {
      "QueueArn": "arn:aws:sqs:eu-west-1:123456789012:my-queue",
      "Events": [
        "s3:ObjectCreated:*"
      ],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "my-filter"
            }
          ]
        }
      }
    }
  ],
  "LambdaFunctionConfigurations": [
  ]
}

I really have no clue where the error could be. Thanks for any help!

Answer 1

It looks like your SQS policy is not working. Try to add Id to your policy and Resource to your statement. Something like this:

{ "DelaySeconds":"0", "MessageRetentionPeriod":"3600", "Policy":"{\"Id\":\"someid\",\"Statement\":[{\"Effect\":\"Allow\",\"Resource\": \"arn:aws:sqs:eu-west-1:123456789012:my-queue\",\"Principal\":\"*\",\"Action\":[\"sqs:SendMessage\",\"sqs:ReceiveMessage\"],\"Condition\":{\"ArnLike\": {\"aws:SourceArn\": \"arn:aws:s3:*:*:my-bucket\"}}}]}" }

Here is more information:

http://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html#step1-create-sqs-queue-for-notification

Also when calling API from command line, you can use --debug parameter. You would see full error message:

aws --debug s3api ...

Answer 2

I got a script that works. I post it here for whomever else might be puzzling over this :-)

#!/usr/bin/env python

import boto3
import json

bucket_name='spike-bucket-000'
queue_name='spike_queue_000'
region='eu-west-1'

s3 = boto3.client('s3', region)
sqs = boto3.client('sqs', region)

def check_if_bucket_exists(name):
    s3.head_bucket(Bucket=bucket_name)


try:
    check_if_bucket_exists(bucket_name)
    print('Bucket {} exists'.format(bucket_name))
except botocore.exceptions.ClientError:
    print('Creating bucket {}'.format(bucket_name))
    s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': region})

print('Ensuring queue {} exists'.format(queue_name))

response = sqs.create_queue(QueueName=queue_name)
queue_url = response['QueueUrl']
response = sqs.get_queue_attributes(QueueUrl=queue_url, AttributeNames=['QueueArn'])
queue_arn = response['Attributes']['QueueArn']

print('Granting bucket permission to post messages to queue')

queue_policy={
         "Version": "2008-10-17",
         "Id": "example-ID",
         "Statement": [
          {
           "Sid": "example-statement-ID",
           "Effect": "Allow",
           "Principal": {
            "AWS":"*"
           },
           "Action": [
            "SQS:SendMessage"
           ],
           "Resource": queue_arn,
           "Condition": {
              "ArnLike": {
              "aws:SourceArn": "arn:aws:s3:*:*:" + bucket_name
            }
           }
          }
         ]
        }

sqs.set_queue_attributes(QueueUrl=queue_url, Attributes={'Policy': json.dumps(queue_policy)})

print('Configuring bucket to notify object creation to queue')

response = s3.put_bucket_notification_configuration(
    Bucket=bucket_name,
    NotificationConfiguration={
        'QueueConfigurations': [
            {
                'Id': 'Notify-ObjectCreated-To-Queue',
                'QueueArn': queue_arn,
                'Events': [
                    's3:ObjectCreated:*',
                ]
#               ,
#                 'Filter': {
#                     'Key': {
#                         'FilterRules': [
#                             {
#                                 'Name': 'prefix'|'suffix',
#                                 'Value': 'string'
#                             },
#                         ]
#                     }
#}
            },
        ]
    }
)