Why does Spark Sql Exclude the Inet Data Type?

Question

I have a postgres database (9.2) that uses the inet type in the schema. I am attempting to write to the postgres with Spark (1.6).

df.write.mode(mode).jdbc(db, table, props)

I am applying the schema to the df with;

context.createDataFrame(rdd, schema())

and getting all BatchUpdateException errors described in SPARK-13286 due to the schema mismatch. The schema is a StructType of StructFields that require a Spark Sql DataType. Is there a way to make this work with any of the existing Spark Sql Data Types?

Answer 1

Why does Spark Sql Exclude the Inet Data Type?

Realistically Spark cannot support all the custom types which are used by different JDBC sources (not only RDBMS) out there.

way to make this work with any of the existing Spark Sql Data Types?

You can use query to cast to the type, which is consumable by Spark (not tested):

spark.read.jdbc(url, "(SELECT CAST(inet_col AS TEXT) FROM table) AS t")