Menu
I am passing some json data to firehose delivery stream which in the end is getting saved into Redshift table. For my use case, I want the data to be stored in different tables.
Do I create different delivery stream for different tables? If I create it that way, there will be data duplication in S3 as the data must go through S3 in order to push data to Redshift using Firehose delivery stream.
507
Support for multiple data destinations You can specify the destination Amazon S3 bucket, the Amazon Redshift table, the Amazon OpenSearch Service domain, generic HTTP endpoints, or a service provider where the data should be loaded.
Kinesis Data Firehose delivers your data to your S3 bucket first and then issues an Amazon Redshift COPY command to load the data into your Amazon Redshift cluster. Specify an S3 bucket that you own where the streaming data should be delivered. Create a new S3 bucket, or choose an existing bucket that you own.
Kinesis Data Firehose first delivers data to Amazon S3 and then uses COPY syntax to load data into an Amazon Redshift table.
A single delivery stream can only deliver data to one Amazon S3 bucket currently. If you want to have data delivered to multiple S3 buckets, you can create multiple delivery streams.
From the Kinesis Firehose FAQ:
Q: Can a single delivery stream deliver data to multiple Amazon Redshift clusters or tables?
A single delivery stream can only deliver data to one Amazon Redshift cluster and one table currently. If you want to have data delivered to multiple Redshift clusters or tables, you can create multiple delivery streams.
You will need multiple streams.
125
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
