Amazon Kinesis NodeJS - putRecord action expires

Question

I'm using NodeJS and Express as proxy to handle HTTP GET requests, do some small adapting of the data, and submit data to Amazone Kinesis. Here is the extract from my code:

var express = require('express');
var app = express();


app.get('/proxy-test', function(req, res){
    var data = req.query;

    // perform some light data processing

    // send results to kinesis
    kinesis.putRecord({
        StreamName : MY_STREAM_NAME,
        Data : data,
        PartitionKey : MY_PARTITION_KEY
    }, function(err, data) {
        if (err) {
            console.log(err, err.stack); // an error occurred
        } else {
            console.log(data);
        }
    });

    res.send(200);
});

After running JMeter tests for 200 concurrent users (loop 100 times), after 5 mins, I'm getting this error:

{ [InvalidSignatureException: Signature expired: 20140409T152855Z is now earlier than 20140409T153211Z (20140409T153711Z - 5 min.)] message: 'Signature expired: 20140409T152855Z is now earlier than 20140409T153211Z (20140409T153711Z - 5 min.)', code: 'InvalidSignatureException', time: Wed Apr 09 2014 17:37:11 GMT+0200 (CEST), statusCode: 400, retryable: false, _willRetry: false } 'InvalidSignatureException: Signature expired: 20140409T152855Z is now earlier than 20140409T153211Z (20140409T153711Z - 5 min.)\n at Request.extractError (/Users/me/proxy/node_modules/aws-sdk/lib/service_interface/json.js:43:33)\n at Request.callListeners (/Users/me/proxy/node_modules/aws-sdk/lib/sequential_executor.js:114:20)\n at Request.emit (/Users/me/proxy/node_modules/aws-sdk/lib/sequential_executor.js:81:10)\n at Request.emit (/Users/me/proxy/node_modules/aws-sdk/lib/request.js:578:14)\n at Request.transition (/Users/me/proxy/node_modules/aws-sdk/lib/request.js:12:12)\n at AcceptorStateMachine.runTo (/Users/me/proxy/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /Users/me/proxy/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request. (/Users/me/proxy/node_modules/aws-sdk/lib/request.js:28:9)\n at Request. (/Users/me/proxy/node_modules/aws-sdk/lib/request.js:580:12)\n at Request.callListeners (/Users/me/proxy/node_modules/aws-sdk/lib/sequential_executor.js:90:20)'

Is there something I can do (configuration or code change) in order to ensure all Kinesis records are sent and saved?

Answer 1

It is possible you're experiencing clock drift on your Kinesis producer instance (the instance you're running NodeJS on). We used to see this quite often when querying the AWS API of various services (not just Kinesis).

It is pretty common to install and configure ntp to use the AWS nodes in the NTP pool:

sudo apt-get -qy install ntp
sudo sed -i -r 's/^(server\s[0-3]\.)ubuntu([a-z\.]+)/\1amazon\2 iburst/' /etc/ntp.conf
sudo sed -i -r 's/^(server\sntp\.ubuntu\.com)/#\1\nserver time.nist.gov/' /etc/ntp.conf
sudo service ntp restart

Here is a link to more documentation if you're interested:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html