Menu
I am setting up a kinesis firehose stream and everything works well with the files getting created on s3 which are delimited. But i was wondering if there is a way to specify an extension to this file since the consumer of this file require it to be either a csv or txt. Is there any way of doing this?
Thanks
806
Amazon Kinesis Data Firehose can convert the format of your input data from JSON to Apache Parquet or Apache ORC before storing the data in Amazon S3. Parquet and ORC are columnar data formats that save space and enable faster queries compared to row-oriented formats like JSON.
You can integrate Amazon S3 with Amazon Kinesis Firehose by implementing the following steps: Step 1: Creating & Configuring a Delivery Stream. Step 2: Configuring Data Transformations. Step 3: Choosing the Destination.
S3 compressions and encryption Snappy, Zip, and Hadoop-Compatible Snappy compression is not available for delivery streams with Amazon Redshift as the destination.
Firehose can scale to gigabytes of streaming data per second, and allows for batching, encrypting and compressing of data. It should be noted that Firehose will automatically scale to meet demand, which is in contrast to Kinesis Streams, for which you must manually provision enough capacity to meet anticipated needs.
You can create an s3 trigger to lambda and rename from there.
159
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
