AWS SQS practical code PHP

Question

i had a question here
but i'm still missing the point understanding how to use SQS, could anyone help out with some code please.

question: what would one place inside a SQS?
i've read through amazons tutorial and the concept seems lovely but im missing the practical side of things.
for instance, this diagram is great: http://awsmedia.s3.amazonaws.com/catalog/images/159-for-madhu.png
what would go inside the SQS? I understand how to upload to S3, but still grey about the SQS part

require_once('sdk-1.5.6.2/sdk.class.php');
//random strings
$AWS_KEY = "MY_ACCESS_KEY";
$AWS_SECRET_KEY = "MY_SECRET_KEY";


//create a new SQS queue and grab the queue URL
$sqs = new AmazonSQS(array( "key" => $AWS_KEY, "secret" => $AWS_SECRET_KEY ));
$response = $sqs->create_queue('test-topic-queue');
$queue_url = (string) $response->body->CreateQueueResult->QueueUrl;

$queue_arn = 'arn:aws:sqs:us-east-1:ENCQ8gqrAcXv:test-topic-queue';
$topic_arn = 'arn:aws:sns:us-east-1:ENCQ8gqrAcXv:test-topic';
$response = $sns->subscribe($topic_arn, 'sqs', $queue_arn);
$subscription_arn = (string) $response->body->SubscribeResult->SubscriptionArn;




/*  *   * * * * * * * * * * * * * * * *     

THIS IS THE BIG GREY AREA, WHAT HAPPENS HERE ??? 

*   * * * * * * * * * * * * * * * * */



// delete SQS queue
$queue_url = 'https://sqs.us-east-1.amazonaws.com/ENCQ8gqrAcXv/test-topic-queue'; 
$response = $sqs->delete_queue($queue_url);

Answer 1

You should have two processes, one that inserts messages into the queue and your worker threads. A typical worker thread will look something like this:

while(true) {
    $res = $client->receiveMessage(array(
        'QueueUrl'          => $url,
        'WaitTimeSeconds'   => 1
    ));
    if ($res->getPath('Messages')) {

        foreach ($res->getPath('Messages') as $msg) {
            echo "Received Msg: ".$msg['Body'];

            // Do something useful with $msg['Body'] here

            $res = $client->deleteMessage(array(
                'QueueUrl'      => $url,
                'ReceiptHandle' => $msg['ReceiptHandle']
            ));
        }
    }
}

The WaitTimeSeconds parameters means to do "long polling" and has various benefits (see http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html).

You should use supervise or monit to make sure your worker threads stay running.

UPDATE: for api v3 use get() instead of getPath() - getPath is now deprecated

Answer 2

No code.. but an example (explaining the diagram basically).

You have a website where people can upload video. You want to convert every video to .mp4 (or .flv if you're that kinda guy).

Transcoding takes (for the sake of the argument) 1 minute per video. You don't want to do this in the same HTTP / PHP request as the upload, as it will take 1 minute for the video to return, and it means that many videos may be transcoding on the webserver on any given moment.

What you need instead, are separate machines / instances 100% responsible for transcoding. They get 'jobs' and process them in order (a queue). We call them workers.

So how are you going to communicate from your webserver that a worker should start trancoding. 1 way would be to set a database record that the workers check every x seconds.

This is not a great design. Generally whenever you must resort to polling, you're developing a workaround for something that can be more efficient.

In the PHP world, one way to do this is with Gearman. Alternatively, a ZeroMQ, RabbitMQ, but Gearman is quite the popular PHP solution. In the AWS world they provide SQS. Responsible for one thing:

Pushing in messages on 1 end. In this example these are the jobs coming from the webserver. Waiting for new messages on the other end. In this example these are the workers waiting for stuff to do.

The nice thing about SQS is that you can just add workers on 1 end, and add webservers on the other end. SQS makes sure only 1 worker gets the message. So it meets the horizontal scaling paradigm (provided SQS itself scales horizontally).