Cannot have circular references in bean class, but got the circular reference of class class org.apache.avro.Schema

Question

I want to convert javaRDD which contain Avro objects(eg. objects of MyAvroClsass) to data frame in java spark. I am getting below ERROR

Cannot have circular references in bean class, but got the circular reference of class class org.apache.avro.Schema

Code:

JavaRDD<Row> test; 
Dataset<Row> outputDF = sparksession.createDataFrame(test.rdd(),<MyAvroClsass>.class);

Answer 1

This is related to: Infinite recursion in createDataFrame for avro types

There is work being done in the spark-avro project to address this issue see: https://github.com/databricks/spark-avro/pull/217 and https://github.com/databricks/spark-avro/pull/216

Once this is merged, there should be a function to convert an RDD of Avro objects into a DataSet (a DataSet of Rows is equivalent to a DataFrame), without the circular reference issue with the getSchema() function in the generated class.