Recommended way to write data to Amazon Kinesis

Question

I am trying to figure out the managed service which can write data to kinesis. with this way i need my message should get at least one delivery to kinesis stream. is it recommended or good idea to use SQS to write to Kinesis. I am looking the solution which can scale horizontally.

Answer 1

There are multiple options to write to Kinesis, based on your experience and environment.

The most straightforward method is to call the put-record API directly. You can write one record at the time with put-record or batch them together with put-records. These API calls are supported by the various SDKs (Java, .NET, php, ruby, javascript, python…).

Since one of the most useful use cases of Kinesis is to gather information from millions of users on their mobile devices or browsers, there are dedicated mobile SDKs for iOS, Android and JavaScript in the browser. See here: http://aws.amazon.com/mobile/sdk/. You can use these SDKs to remove the need for an ingestion infrastructure beyond Kinesis.

Some other options are to use tools like FluentD that is very popular in shipping logs. See here a connector that can make your life easier: https://github.com/awslabs/aws-fluent-plugin-kinesis

Another recent option is to use Kinesis Producer Library (KPL) that is adding the ability to aggregate many events together into a single event to optimise the shard capacity to the limit. It is also allowing asynchronous writing to Kinesis by the producers, and the KPL will handle the blocking, retires and other latency causing methods.

The log4j appenders are also a popular way to write log events directly to kinesis with minimal effort from the side of the developers. See here for more details: https://github.com/awslabs/kinesis-log4j-appender

Answer 2

If you are getting your resources from HTTP calls, try Amazon API Gateway: http://aws.amazon.com/api-gateway/

Here is a nice post about capabilities: https://aws.amazon.com/blogs/aws/amazon-api-gateway-build-and-run-scalable-application-backends/