Extend Avro schema via Java API by adding one field

Question

I use the Java API for Avro from Scala and wonder if there is an easy programmatical way to add a field to an existing record schema using the Avro GenericRecord / SchemaBuilder API?

Answer 1

There's no easy way - but I know exactly what you are trying to do.

Here's an example extending an existing schema (eg SchemaBuilder) dynamically.

    Schema schema = SchemaBuilder
            .record("schema_base").namespace("com.namespace.test")
            .fields()
            .name("longField").type().longType().noDefault()
            .name("stringField").type().stringType().noDefault()
            .name("booleanField").type().booleanType().noDefault()
            .name("optionalStringColumn").type().optional().stringType()
            .endRecord();



    List<Schema.Field> field_list = schema.getFields();

    ArrayList<Schema.Field> new_list = new ArrayList();

    //create a new "empty" schema
    //public static Schema createRecord(String name, String doc, String namespace, boolean isError) {

    Schema s2 = Schema.createRecord("new_schema", "info", "com.namespace.test", false);

    //add existing fields
    for(Schema.Field f : field_list) {

        //f.schema() here is really type "schema" like long or string, not a link back to a custom schema
        Schema.Field ff = new Schema.Field(f.name(), f.schema(), f.doc(), f.defaultVal());
        new_list.add(ff);
    }

    //this here is just to show how to create an optional string, its a union of null and string types
    ArrayList<Schema> optionalString = new ArrayList<>();
    optionalString.add(Schema.create(Schema.Type.NULL));
    optionalString.add(Schema.create(Schema.Type.STRING));

    //add the new 3 test fields in as optional string types
    //default value here appears arbitrary, when you write the record if its not optional it doesn't //pick up default value

    String[] sArray = {"test", "test2", "test3"};

    for(String s : sArray) {

        Schema.Field f = new Schema.Field( s, Schema.createUnion(optionalString), s, "null");
        new_list.add(f);
    }

    s2.setFields(new_list);

You can't just setFields on the existing schema because once they exist, the schema is locked.

Note: be careful with the default value - if there's a type mismatch everything will write out fine, but you won't be able to read the avro file!

Answer 2

Update

Alternatively, you can use SAvro.

libraryDependencies += "ca.dataedu" %% "savro" % "0.3.0"

and then

schema.addField("newField1", SchemaBuilder.builder().stringType())

More examples you can find in the README.

This is same answer but a different format of coding

@tmx has provided a complete answer. Once a schema is created, everything is locked. The only way is to implement a copy method. Here is a more compact version:

    // Start with a base schema 
    Schema base = ...;
    // Get a copy of base schema's fields.
    // Once a field is used in a schema, it gets a position.
    // We can't recycle a field and it will throw an exception.
    // Hence, we need a fresh field from each field of the old schema
    List<Schema.Field> baseFields = base.getFields().stream()
                .map(field -> new Schema.Field(field.name(), field.schema(), field.doc(), field.defaultVal()))
                .collect(Collectors.toList());
    // Add your field
    baseFields.add(new Schema.Field("Name", newFieldSchema));
    Schema newSchema = Schema.createRecord(
        base.getName(), 
        "New schema by adding a new field", 
        "com.my.name.space", 
        false, 
        baseFields);

having baseFields, you could do any modification you'd like, add/delete/modify.