How to deploy and Run Amazon Kinesis Application on Amazon Kinesis service

Question

I am trying to understand how to deploy an Amazon Kinesis Client application that was built using the Kinesis client library (KCL).

I found this but it only states

You can follow your own best practices for deploying code to an Amazon EC2 instance when you deploy a Amazon Kinesis application. For example, you can add your Amazon Kinesis application to one of your Amazon EC2 AMIs.

which is not giving a broader picture to me.

These examples use an Ant script to run Java program. Is this the best practice to follow?

Also, I understand even before running the EC2 instances I need to make sure

1. The developed code JAR/WAR or any other format needs to be on the EC2 instance
2. The EC2 instance needs to have all the required environment like Ant setup in place already to execute the program.

Could someone please add some more detail on this?

Answer 1

Amazon Kinesis will be responsible for ingesting data, not running your application. You can run your application anywhere, but it is a good idea to run it in EC2, as you are probably going to use other AWS Services, such as S3 or DynamoDB (Kinesis Client Library uses DynamoDB for sharding, for example).

To understand Kinesis better, I'd recommend that you launch the Kinesis Data Visualization Sample. When you launch this app, use the provided CloudFormation template. It will create a stack with the Kinesis stream and an EC2 instance with the application, that uses Kinesis Client Library and is a fully working example to start from.

Answer 2

The best way I have found to host a consumer program is using EMR, but not as a hadoop cluster. Package your program as a jar, and place it in s3. Launch an emr cluster and have it run your jar. Using the data pipeline you can schedule this job flow to run at regular intervals. You can also scale an emr cluster, or use a actual EMR job to process the stream if you choose to get the high tech.

Answer 3

You can also use Beanstalk. I believe this article is highly useful.