Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is it possible to POST to the SQS url using the post body

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue/
?Action=SendMessage
&MessageBody=This+is+a+test+message

This works fine, however is it possible to send the payload using the post body?

I have an external service sending json payloads in the post body (at a very high throughput). It would be ideal if I can simply feed them the SQS url directly, and avoid having to create a AWS api gateway --> lambda --> SQS stack.

I'm open to alternative solutions.

like image 767
Andrew Wei Avatar asked Mar 29 '18 16:03

Andrew Wei


People also ask

How do I send data to SQS queue?

After you create your queue, you can send a message to it. From the left navigation pane, choose Queues. From the queue list, select the queue that you created. From Actions, choose Send and receive messages.

What is an SQS URL?

An Amazon SQS GET request is structured as a URL which consists of the following: Endpoint – The resource that the request is acting on (the queue name and URL), for example: https://sqs.us-east-2.amazonaws.com/ 123456789012 /MyQueue. Action – The action that you want to perform on the endpoint.

Is SQS URL public?

SQS Queue is not publicly accessible.


1 Answers

This may not have been exactly what the designers of API Gateway had in mind, but given...

  • the way the AWS Query APIs (as used by the SDKs) work on the wire for services like SQS -- url-encoded key/value pairs in the standard application/x-www-form-urlencoded format, and

  • the fact that the built-in body mapping template language in API Gateway (VTL) exposes $util.urlEncode(), and

  • the fact that API Gateway can transparently sign requests when sending them to a backend service

...this means you can simply construct a valid API request -- a form post -- by hand, as were, using a VTL template, that encapsulates the original message body -- urlencoded, and that's all we need. We don't actually care that it's JSON, as long as it is valid character data.

With this solution, a Lambda function is avoided, and the client does not need to know anything about how SQS expects messages to be formatted.

The entire incoming request body becomes the Message in SQS.


In the API Gateway console:

Create the resource (e.g. /) and the method (e.g. POST).

In the Integration Request settings:

Integration type: AWS Service
AWS Region: [select your region]
AWS Service: Simple Queue Service (SQS)
AWS Subdomain [leave blank]
HTTP Method: POST
Action Type: Use path override
Path override: /
Execution Role: [your role ARN, needs to be able to send a message to the queue]
Credentials cache: Do not add caller credentials to cache key
Content Handling: Passthrough

Under HTTP Headers, add one header, Content-Type. This value should be specified as being "mapped from" 'application/x-www-form-urlencoded' -- note that this is a single-quoted string.

Under Body Mapping Templates, choose Never.

Add a Content-Type of application/json and use the following mapping template:

Action=SendMessage##
&QueueUrl=$util.urlEncode('https://sqs.us-east-2.amazonaws.com/000000000000/my-queue-name')##
&MessageBody=$util.urlEncode($input.body)##

And you have an API that transforms the a raw JSON input body into an SQS SendMessage API request.

The ## at the end of each line are for readability -- VTL is a text templating language, so whitespace and newlines are preserved. Placing ## at the end of each line causes the newline to be stripped, which is necessary for building a correct web form. Otherwise, the entire body mapping template would need to be on a single line.

Deploy and then test:

$ curl -X POST https://xxxxxxxxxx.execute-api.us-east-2.amazonaws.com/v1 --data '{"works": true}' -H 'Content-Type: application/json'

Response:

{"SendMessageResponse":{"ResponseMetadata":{"RequestId":"bbbbbbbb-aaaa-5555-8888-334ed25bb6b3"},"SendMessageResult":{"MD5OfMessageAttributes":null,"MD5OfMessageBody":"81f2ecc3cb027268138bdfe7af0f8a3f","MessageId":"cccccccc-dddd-4444-1111-542e08bb39af","SequenceNumber":null}}}

For extra credit, a body mapping template in the Integration Response can be reused to customize the response, as well.

like image 142
Michael - sqlbot Avatar answered Oct 02 '22 21:10

Michael - sqlbot