Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Avro Java API Timestamp Logical Type?

Tags:

java

time

avro

With the Avro Java API, I can make a simple record schema like:

    Schema schemaWithTimestamp = SchemaBuilder
            .record("MyRecord").namespace("org.demo")
            .fields()
            .name("timestamp").type().longType().noDefault()
            .endRecord();

How do I tag a schema field with a logical type, specifically: https://avro.apache.org/docs/1.8.1/api/java/org/apache/avro/LogicalTypes.TimestampMillis.html

like image 461
clay Avatar asked Mar 28 '17 22:03

clay


People also ask

What is logical type in Avro?

Logical Types A logical type is an Avro primitive or complex type with extra attributes to represent a derived type.

Does Avro support timestamp?

Avro in HDF is 1.7. 7 and timestamp was only introduced in Avro 1.8. x. I would suggest to treat the timestamp field as string.

What is an Avro record?

Avro is an open source data serialization system that helps with data exchange between systems, programming languages, and processing frameworks. Avro helps define a binary format for your data, as well as map it to the programming language of your choice.

What is Avro Java?

Avro is a language independent, schema-based data serialization library. It uses a schema to perform serialization and deserialization. Moreover, Avro uses a JSON format to specify the data structure which makes it more powerful.


3 Answers

Thanks for the first solution, now for nullable logicalType like:

{ 
"name":"maturityDate",
  "type":["null", {
    "type":"long","logicalType":"timestamp-millis"
    }]
},

I figure the following:

        Schema timestampMilliType = LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));

        Schema clientIdentifier = SchemaBuilder.record("ClientIdentifier")
                .namespace("com.baeldung.avro")
                .fields()
                .requiredString("hostName")
                .requiredString("ipAddress")
                .name("maturityDate")
                .type()
                .unionOf()
                .nullType()
                .and()
                .type(timestampMilliType)
                .endUnion()
                .noDefault()
                .endRecord();
like image 76
Edmond Wong Avatar answered Oct 17 '22 02:10

Edmond Wong


Thanks to DontPanic:

    Schema timestampMilliType = LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));

    Schema schemaWithTimestamp = SchemaBuilder
            .record("MyRecord").namespace("org.demo")
            .fields()
            .name("timestamp_with_logical_type").type(timestampMilliType).noDefault()
            .name("timestamp_no_logical_type").type().longType().noDefault()
            .endRecord();

    System.out.println(schemaWithTimestamp.toString(true));

This results in:

{
  "type" : "record",
  "name" : "MyRecord",
  "namespace" : "org.demo",
  "fields" : [ {
    "name" : "timestamp_with_logical_type",
    "type" : {
      "type" : "long",
      "logicalType" : "timestamp-millis"
    }
  }, {
    "name" : "timestamp_no_logical_type",
    "type" : "long"
  } ]
}
like image 21
clay Avatar answered Oct 17 '22 02:10

clay


I think you may create schema manually:

List<Schema.Field> fields = new ArrayList<>();
Schema timeStampField = Schema.create(Schema.Type.LONG);
fields.add(new Schema.Field("timestamp", LogicalTypes.timestampMillis().addToSchema(timeStampField), null, null));
Schema resultSchema = Schema.createRecord("MyRecord", null, "org.demo", false, fields);
System.out.println(resultSchema);

your schema:

{"type":"record","name":"MyRecord","namespace":"org.demo","fields":[{"name":"timestamp","type":"long"}]}

resultSchema with timestampMillis:

{"type":"record","name":"MyRecodr","namespace":"org.demo","fields":[{"name":"timestamp","type":{"type":"long","logicalType":"timestamp-millis"}}]}
like image 2
DontPanic Avatar answered Oct 17 '22 02:10

DontPanic