Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a way to get the column data type in pyspark?

Has been discussed that the way to find the column datatype in pyspark is using df.dtypes get datatype of column using pyspark. The problem with this is that for datatypes like an array or struct you get something like array<string> or array<integer>.

Question: Is there a native way to get the pyspark data type? Like ArrayType(StringType,true)

like image 616
argenisleon Avatar asked Aug 01 '18 23:08

argenisleon


1 Answers

Just use schema:

df.schema[column_name].dataType
like image 56
user10168341 Avatar answered Oct 15 '22 08:10

user10168341