ArrowTypeError: Did not pass numpy.dtype object', 'Conversion failed for column X with type int32

Question

Problem

I am trying to save a data frame as a parquet file on Databricks, getting the ArrowTypeError.

Databricks Runtime Version: 7.6 ML (includes Apache Spark 3.0.1, Scala 2.12)

Log Trace

ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column inv_yr with type int32')

Answer 1

The issue you are facing originates from the fact that you are using an old pyarrow wheel with the latest numpy 1.20 release. You are running into the bug "PyArray_DescrCheck doesn't work anymore if the consumer library was compiled with an older NumPy version ". Either update your pyarrow version or downgrade to numpy<1.20.