Menu
The AWS docs makes it look as if you can specify DataFormatConversionConfiguration for a AWS::KinesisFirehose::DeliveryStream in cloudformation, but is not documentation on where the property is supposed to go. Tried adding it under Properties, ExtendedS3DestinationConfiguration, ProcessingConfiguration, and one of the Processors. Each time, CF complains with-
The following resource(s) failed to update: [EventFirehose]. 12:24:32 UTC-0500
UPDATE_FAILED AWS::KinesisFirehose::DeliveryStream EventFirehose Encountered unsupported property DataFormatConversionConfiguration
There own docs say-
If you want Kinesis Data Firehose to convert the format of your input data from JSON to Parquet or ORC, specify the optional DataFormatConversionConfiguration element in ExtendedS3DestinationConfiguration or in ExtendedS3DestinationUpdate.
What am I doing wrong?
422
Amazon Kinesis Data Firehose can convert the format of your input data from JSON to Apache Parquet or Apache ORC before storing the data in Amazon S3. Parquet and ORC are columnar data formats that save space and enable faster queries compared to row-oriented formats like JSON.
You can author AWS CloudFormation templates in JSON or YAML formats. We support all AWS CloudFormation features and functions for both formats, including in AWS CloudFormation Designer.
AWS CloudFormation supports the following AWS-specific types: AWS::EC2::AvailabilityZone::Name. An Availability Zone, such as us-west-2a . AWS::EC2::Image::Id.
The end point URL for CloudFormation is region based and AFAIK there isn't a place whether you can specify an region specific (diff region) information.
The template format version can change independently of the API and WSDL versions. The value for the template format version declaration must be a literal string. You can't use a parameter or function to specify the template format version. If you don't specify a value, AWS CloudFormation assumes the latest template format version.
Why do we need AWS Cloudformation? Just imagine that you have to develop an application that uses various AWS resources. When it comes to creating and managing those resources, it can be highly time-consuming and challenging.
JSON is a text-based format that represents structured data on the basis of JavaScript object syntax. It carries the AWS resources details in the structured format according to which AWS infrastructure is created. Format version: It defines the version of a template.
Step 1: Go to the Cloudformation dashboard on the AWS management console. Click on create the stack. Step 2: You will be redirected to this webpage. We will be using a sample template of Lamp Stack in this. Select the option: Use a sample template. Select the Lamp Stack template. Click on View in Designer to view the design of the template.
As per SDK documentation, it should be inside ExtendedS3DestinationConfiguration or ExtendedS3DestinationUpdate. However, the cloudformation is currently NOT supporting this property as per this docs. This is a very common discrepancy between cloudformation and other AWS services. One similar issue is mentioned here AWS ECS: Severe bug in ECS Service Cloudformation template (which got resolved recently).
For the time being, you can update via SDK or wait for some time for cloudformation to catch up.
If you want Kinesis Data Firehose to convert the format of your input data from JSON to Parquet or ORC, specify the optional DataFormatConversionConfiguration element in ExtendedS3DestinationConfiguration or in ExtendedS3DestinationUpdate
Edit
As of June 2019, the DataFormatConversionConfiguration property is in available in CloudFormation. See change log: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html
173
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
