Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do you update an existing avro schema using apache avro SchemaBuilder?

Tags:

java

apache

avro

I am testing a new schema registry which loads and retrieves different kinds of avro schemas. In the process of testing, I need to create a bunch of different types of avro schemas. As it involves a lot of permutations, I decided to create the schema programmatically. I am using the apache avro SchemaBuilder to do so.

I created the avro using :

Schema oldSchema = SchemaBuilder
      .record("abc")
      .aliases("records")
      .fields()
      .name("field_null")
      .type("null")
      .noDefault()
      .endRecord();

This worked. The avro created looks like :

{
 "type" : "record",
 "name" : "abc",
 "fields" : [ {
   "name" : "field_null",
   "type" : "null"
  } ],
  "aliases" : [ "records" ]
}

Now I want to create a new version of the schema using the apache avro libraries like :

{
 "type" : "record",
 "name" : "abc",
 "fields" : [ {
   "name" : "field_null",
   "type" : "null"
  },
  {
   "name" : "new_field",
   "type" : "int",
   "default" : 10
  }
 ],
 "aliases" : [ "records" ]
}

For this, I tried :

Schema.Field field = new Schema.Field("new_field", SchemaBuilder.builder().intType(),
    "NewField", 10);

List<Schema.Field> fields = new ArrayList<>();
fields.add(field);
fields.addAll(oldSchema.getFields());

Schema record = Schema.createRecord(oldSchema.getName(),
    "Changes",
    oldSchema.getNamespace(),
    false,
    fields);

I get :

org.apache.avro.AvroRuntimeException: Field already used: field_null type:NULL pos:0

at org.apache.avro.Schema$RecordSchema.setFields(Schema.java:647)
at org.apache.avro.Schema$RecordSchema.<init>(Schema.java:618)
at org.apache.avro.Schema.createRecord(Schema.java:167)

My problem is :

  1. How do I add new versions of the schema using existing libraries?
  2. Should I use avro schemaBuilder to create the schema or rather create my own POJOs to build the schema/save the avsc files in a data directory.
like image 634
Shreya Bhat Avatar asked Aug 22 '17 12:08

Shreya Bhat


People also ask

What is Logicaltype in Avro schema?

Logical types specify a way of representing a high-level type as a base Avro type. For example, a date is specified as the number of days after the unix epoch (or before using a negative value). This enables extensions to Avro's type system without breaking binary compatibility.

How do you make an Avro schema file?

Creating Avro Schemastype − This field comes under the document as well as the under the field named fields. In case of document, it shows the type of the document, generally a record because there are multiple fields. When it is field, the type describes data type.


1 Answers

You can try this to create fields, maybe it's clumsy :

Schema.Field field = new Schema.Field("new_field",SchemaBuilder.builder().intType(),
    "NewField", 10);

List<Schema.Field> fields = new ArrayList<>();

for (Schema.Field f : oldSchema.getFields()) {

   Schema.Field _field = new Schema.Field(f.name(), f.schema(), f.doc(), f.defaultValue());
  fields.add(_field);

}
like image 107
xiping xing Avatar answered Oct 26 '22 04:10

xiping xing