How to use apache avro GenericRecord for dynamic data?

Question

I want to use Apache Avro to serialize my data, my client is written in C++, and my server is written in Java.

1. My server java code looks like this:

   Schema scm = new Schema.Parser().parse("....shcema String.....");
   ByteArrayInputStream inputStream = new ByteArrayInputStream(record.array());
   Decoder coder = new DecoderFactory().directBinaryDecoder(inputStream, null);
   GenericDatumReader<GenericRecord> reDatumReader = new GenericDatumReader<GenericRecord>(scm);
   try {
       GenericRecord result = (GenericRecord)reDatumReader.read(null, coder);
               //here! the result "name", "num_groups" is empty!
       System.out.println(result.get("name")+"   "+result.get("num_groups"));
   } catch (IOException e) {
       // TODO Auto-generated catch block
       e.printStackTrace();
   } 
   

2. And my client code:

   std::string schemaDescript ="....shcema String.....";
   
   std::stringstream rsStream(schemaDescript);
   avro::ValidSchema rSchema;
   avro::compileJsonSchema(rsStream, rSchema);
   avro::EncoderPtr encoder = avro::binaryEncoder();
   std::auto_ptr<avro::OutputStream> oStream = avro::memoryOutputStream();
   encoder->init(*oStream);   
   avro::GenericDatum rData(rSchema);
   avro::GenericRecord sReord = rData.value<avro::GenericRecord>();
   sReord.setFieldAt(0, avro::GenericDatum("i am nice"));
   sReord.setFieldAt(1, avro::GenericDatum(1));
   sReord.setFieldAt(2, avro::GenericDatum(12));
   sReord.setFieldAt(3, avro::GenericDatum(13));
   
   avro::GenericWriter gwriter(rSchema, encoder);
   gwriter.write(rData);
   oStream->flush();
   
   std::auto_ptr<avro::InputStream> inSt = avro::memoryInputStream(*oStream);
   avro::StreamReader instReader(*inSt);
   
   size_t outputLen = oStream->byteCount();
   uint8_t* theByteData = new uint8_t[outputLen];
   instReader.hasMore();
   instReader.readBytes(theByteData, outputLen);
   

I send the theByteData to the server, the code works (no exception), but the result is empty, can anyone tell me what's wrong?

And why in Java we get value with key: result.get("name"); but in C++ we get value with index: record.fieldAt(0).value<string>(). If I can't get value with the string key, how to match the index to string key?

Answer 1

I had the same problem this morning and i found a solution in the Avro Test Cpp file("DataFileTests.cc") with the "testWriteGeneric" function.

For example:

My Schema file(cpx.json):

{
  "type": "record",
  "name": "cpx",
  "fields" : [
  {"name": "re", "type": "double"},
  {"name": "im", "type" : "int"}
  ]
 }

My Cpp file:

typedef std::pair<avro::ValidSchema, avro::GenericDatum> Pair;

int main(int ac, char **av)
{

    // encode
    std::ifstream ifs(cpx.json);
    avro::ValidSchema schema;
    avro::compileJsonSchema(ifs, schema);

    // I create a pair of validSchema and GenericDatum
    Pair p(schema, avro::GenericDatum());

    avro::GenericDatum &Data = p.second;
    Data = avro::GenericDatum(schema);
    avro::GenericRecord &sReord = Data.value<avro::GenericRecord>();

    // I set my values
    sReord.setFieldAt(sReord.fieldIndex("re"), avro::GenericDatum(42.5));
    sReord.setFieldAt(sReord.fieldIndex("im"), avro::GenericDatum(24));


    // I create a DataFileWriter and i write my pair of ValidSchema and GenericValue
    avro::DataFileWriter<Pair> dataFileWriter("test.bin", schema);
    dataFileWriter.write(p);
    dataFileWriter.close();
}