Sending compressed text over Amazon SQS from PHP to NodeJS

Question

I seem to be stuck at sending the compressed messages from PHP to NodeJS over Amazon SQS.

Over on the PHP side I have:

$SQS->sendMessage(Array(
    'QueueUrl'    => $queueUrl,
    'MessageBody' => 'article',
    'MessageAttributes' => Array(
        'json' => Array(
            'BinaryValue' => bzcompress(json_encode(Array('type'=>'article','data'=>$vijest))),
            'DataType' => 'Binary'
        )
    )
));

NOTE 1: I also tried putting compressed data directly in the message, but the library gave me an error with some invalid byte data

On the Node side, I have:

body = decodeBzip(message.MessageAttributes.json.BinaryValue);

Where message is from sqs.receiveMessage() call and that part works since it worked for raw (uncompressed messages)

What I am getting is TypeError: improper format

I also tried using:

PHP - NODE

gzcompress() - zlib.inflateraw()

gzdeflate() - zlib.inflate()

gzencode() - zlib.gunzip()

And each of those pairs gave me their version of the same error (essentially, input data is wrong)

Given all that I started to suspect that an error is somewhere in message transmission

What am I doing wrong?

EDIT 1: It seems that the error is somewhere in transmission, since bin2hex() in php and .toString('hex') in Node return totally different values. It seems that Amazon SQS API in PHP transfers BinaryAttribute using base64 but Node fails to decode it. I managed to partially decode it by turning off automatic conversion in amazon aws config file and then manually decoding base64 in node but it still was not able to decode it.

EDIT 2: I managed to accomplish the same thing by using base64_encode() on the php side, and sending the base64 as a messageBody (not using MessageAttributes). On the node side I used new Buffer(messageBody,'base64') and then decodeBzip on that. It all works but I would still like to know why MessageAttribute is not working as it should. Current base64 adds overhead and I like to use the services as they are intended, not by work arounds.

Answer 1

This is what all the SQS libraries do under the hood. You can get the php source code of the SQS library and see for yourself. Binary data will always be base64 encoded (when using MessageAttributes or not, does not matter) as a way to satisfy the API requirement of having form-url-encoded messages.

I do not know how long the data in your $vijest is, but I am willing to bet that after zipping and then base64 encoding it will be bigger than before.

So my answer to you would be two parts (plus a third if you are really stubborn):

• When looking at the underlying raw API it is absolutely clear that not using MessageAttributes does NOT add additional overhead from base64. Instead, using MessageAttributes adds some slight additional overhead because of the structure of the data enforced by the SQS php library. So not using MessageAttributes is clearly NOT a workaround and you should do it if you want to zip the data yourself and you got it to work that way.
• Because of the nature of a http POST request it is a very bad idea to compress your data inside your application. Base64 overhead will likely nullify the compression advantage and you are probably better off sending plain text.
• If you absolutely do not believe me or the API spec or the HTTP spec and want to proceed, then I would advise to send a simple short string 'teststring' in the BinaryValue parameter and compare what you sent with what you got. That will make it very easy to understand the transformations the SQS library is doing on the BinaryValue parameter.