Menu
Can we have multiple destinations from single Kinesis Firehose? I saw this picture 
From this, it looks like it is possible to add s3, redshift and elastic search from single firehose. I exactly want to do this.
But when I do it from aws console, then it asks for single destination only. For elastic search, it asks for S3 also. So, I am able to add elastic search and s3 but still redhift is left. I am not sure how to do it from same kinesis. Please help.
217
A Kinesis data stream is a set of shards. There can be multiple consumer applications for one stream, and each application can consume data independently and concurrently.
Kafka requires more engineering hours for implementation and maintenance leading to a higher total cost of ownership (TCO). As an AWS cloud-native service, Kinesis supports a pay-as-you-go model leading to lower costs to achieve the same outcome.
You can register up to 20 consumers per data stream. A given consumer can only be registered with one data stream at a time. Only 5 consumers can be created simultaneously. In other words, you cannot have more than 5 consumers in a CREATING status at the same time.
The throughput of a Kinesis data stream is designed to scale without limits. The default shard quota is 500 shards per stream for the following AWS Regions: US East (N. Virginia), US West (Oregon), and Europe (Ireland). For all other Regions, the default shard quota is 200 shards per stream.
As you said it not available now but, using a recent added feature of firehose, you can write to 3 of them using one firehose.
But i am not sure this is the optimum way of doing this, you need to compare aws cost and development cost then decide.
If you want to try here is how you can do it.
Configure firehose to write to redshift, intermadiate files will be on s3, (possibly intermediate files are deleted) so you can store them in another bucket with bucket replication or triggering new file notification and move it to another bucket using a lambda function. At the same time to write every single record to Elasticsearch, you will use firehose lambda connection, it is in firehose console, called, data transformation. (link below) Write a simple elasticsearch http post write request for every record firehose has got, so you will have data on 3 destination.
https://aws.amazon.com/blogs/compute/amazon-kinesis-firehose-data-transformation-with-aws-lambda/
As i said before this can cost more, and having 2 firehoses doing redshift write and ES+S3 write is more easy.
165
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
