Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to create schema containing list of objects using Avro?

Tags:

java

schema

avro

Does anyone knows how to create Avro schema which contains list of objects of some class?

I want my generated classes to look like below :

class Child {     String name; }  class Parent {     list<Child> children; } 

For this, I have written part of schema file but do not know how to tell Avro to create list of objects of type Children?

My schema file looks like below :

{     "name": "Parent",     "type":"record",     "fields":[         {             "name":"children",             "type":{                 "name":"Child",                 "type":"record",                 "fields":[                     {"name":"name", "type":"string"}                 ]             }         }     ]  } 

Now problem is that I can mark field children as either Child type or array but do not know how to mark it as a array of objects of type Child class?

Can anyone please help?

like image 302
Shekhar Avatar asked Aug 01 '14 09:08

Shekhar


People also ask

How do you write an Avro schema?

Creating Avro Schemastype − This field comes under the document as well as the under the field named fields. In case of document, it shows the type of the document, generally a record because there are multiple fields. When it is field, the type describes data type.

Does Avro contain schema?

Avro is used to define the data schema for a record's value. This schema describes the fields allowed in the value, along with their data types. You apply a schema to the value portion of an Oracle NoSQL Database record using Avro bindings.

What is Avro schema used for?

Avro is an open source data serialization system that helps with data exchange between systems, programming languages, and processing frameworks. Avro helps define a binary format for your data, as well as map it to the programming language of your choice.

How would you define map in Avro schema?

If I use map as top-level type, e.g., {"type": "map", "values": "string"} , then all fields have to be string type, if there are different types of fields, then map is helpless. You can define your map value type to be a union or named record type containing an union. Avro is quite flexible in this regard.


2 Answers

You need to use array type for creating the list. Following is the updated schema that handles your usecase.

{     "name": "Parent",     "type":"record",     "fields":[         {             "name":"children",             "type":{                 "type": "array",                   "items":{                     "name":"Child",                     "type":"record",                     "fields":[                         {"name":"name", "type":"string"}                     ]                 }             }         }     ]  } 
like image 181
Kapil Balagi Avatar answered Oct 14 '22 11:10

Kapil Balagi


I had following class and avro maven plugin generated two classes accordingly :

public class Employees{     String accountNumber;     String address;     List<Account> accountList;     }  public class Account {     String accountNumber;     String id; } 

Avro file format :

{     "type": "record",     "namespace": "com.mypackage",     "name": "AccountEvent",     "fields": [         {             "name": "accountNumber",             "type": "string"         },         {             "name": "address",             "type": "string"         },         {             "name": "accountList",             "type": {                 "type": "array",                 "items":{                     "name": "Account",                     "type": "record",                     "fields":[                         {   "name": "accountNumber",                             "type": "string"                         },                         {   "name": "id",                             "type": "string"                         }                     ]                 }             }         }     ] } 
like image 38
Smart Coder Avatar answered Oct 14 '22 11:10

Smart Coder