With the Avro Java API, I can make a simple record schema like:
Schema schemaWithTimestamp = SchemaBuilder
.record("MyRecord").namespace("org.demo")
.fields()
.name("timestamp").type().longType().noDefault()
.endRecord();
How do I tag a schema field with a logical type, specifically: https://avro.apache.org/docs/1.8.1/api/java/org/apache/avro/LogicalTypes.TimestampMillis.html
Logical Types A logical type is an Avro primitive or complex type with extra attributes to represent a derived type.
Avro in HDF is 1.7. 7 and timestamp was only introduced in Avro 1.8. x. I would suggest to treat the timestamp field as string.
Avro is an open source data serialization system that helps with data exchange between systems, programming languages, and processing frameworks. Avro helps define a binary format for your data, as well as map it to the programming language of your choice.
Avro is a language independent, schema-based data serialization library. It uses a schema to perform serialization and deserialization. Moreover, Avro uses a JSON format to specify the data structure which makes it more powerful.
Thanks for the first solution, now for nullable logicalType like:
{
"name":"maturityDate",
"type":["null", {
"type":"long","logicalType":"timestamp-millis"
}]
},
I figure the following:
Schema timestampMilliType = LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
Schema clientIdentifier = SchemaBuilder.record("ClientIdentifier")
.namespace("com.baeldung.avro")
.fields()
.requiredString("hostName")
.requiredString("ipAddress")
.name("maturityDate")
.type()
.unionOf()
.nullType()
.and()
.type(timestampMilliType)
.endUnion()
.noDefault()
.endRecord();
Thanks to DontPanic:
Schema timestampMilliType = LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
Schema schemaWithTimestamp = SchemaBuilder
.record("MyRecord").namespace("org.demo")
.fields()
.name("timestamp_with_logical_type").type(timestampMilliType).noDefault()
.name("timestamp_no_logical_type").type().longType().noDefault()
.endRecord();
System.out.println(schemaWithTimestamp.toString(true));
This results in:
{
"type" : "record",
"name" : "MyRecord",
"namespace" : "org.demo",
"fields" : [ {
"name" : "timestamp_with_logical_type",
"type" : {
"type" : "long",
"logicalType" : "timestamp-millis"
}
}, {
"name" : "timestamp_no_logical_type",
"type" : "long"
} ]
}
I think you may create schema manually:
List<Schema.Field> fields = new ArrayList<>();
Schema timeStampField = Schema.create(Schema.Type.LONG);
fields.add(new Schema.Field("timestamp", LogicalTypes.timestampMillis().addToSchema(timeStampField), null, null));
Schema resultSchema = Schema.createRecord("MyRecord", null, "org.demo", false, fields);
System.out.println(resultSchema);
your schema:
{"type":"record","name":"MyRecord","namespace":"org.demo","fields":[{"name":"timestamp","type":"long"}]}
resultSchema with timestampMillis:
{"type":"record","name":"MyRecodr","namespace":"org.demo","fields":[{"name":"timestamp","type":{"type":"long","logicalType":"timestamp-millis"}}]}
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With