Get list of data types from schema in Apache Spark

Question

I have the following code in Spark-Python to get the list of names from the schema of a DataFrame, which works fine, but how can I get the list of the data types?

columnNames = df.schema.names 

For example, something like:

columnTypes = df.schema.types 

Is there any way to get a separate list of the data types contained in a DataFrame schema?

Answer 1

Here's a suggestion:

df = sqlContext.createDataFrame([('a', 1)])  types = [f.dataType for f in df.schema.fields]  types > [StringType, LongType] 

Reference:

• pyspark.sql.types.StructType
• pyspark.sql.types.StructField

Answer 2

Since the question title is not python-specific, I'll add scala version here:

val types = df.schema.fields.map(f => f.dataType) 

It will result in an array of org.apache.spark.sql.types.DataType.