Menu
I have an RDD of type Row i.e, RDD[Row] and avro schema object .I need to create a dataframe with this info.
I need toconvert avro schema object into StructType for creating DataFrame.
Can you please help .
675
StructType – Defines the structure of the Dataframe Spark provides spark. sql. types. StructType class to define the structure of the DataFrame and It is a collection or list on StructField objects. By calling Spark DataFrame printSchema() print the schema on console where StructType columns are represented as struct .
com.databricks.spark.avro has a class to help you with this
StructType requiredType = (StructType) SchemaConverters.toSqlType(AvroClass.getClassSchema()).dataType();
Please go through this specific example : http://bytepadding.com/big-data/spark/read-write-parquet-files-using-spark/
66
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
