Pyspark error on creating dataframe: 'StructField' object has no attribute 'encode'

Question

I'm facing a little issue when creating a dataframe:

from pyspark.sql import SparkSession, types

spark = SparkSession.builder.appName('test').getOrCreate()

df_test = spark.createDataFrame(
    ['a string', 1],
    schema = [
        types.StructField('col1', types.StringType(), True),
        types.StructField('col2', types.IntegerType(), True)
    ]
)

## AttributeError: 'StructField' object has no attribute 'encode'

I don't see anything wrong with my code (it's so simple I feel really dumb). But I can't get this to work. Can you point me in the right direction?

Answer 1

You were most of the way there!

When you call createDataFrame specifying a schema, the schema needs to be a StructType. An ordinary list isn't enough.

1. Create an RDD of tuples or lists from the original RDD;
2. Create the schema represented by a StructType matching the structure of tuples or lists in the RDD created in the step 1.
3. Apply the schema to the RDD via createDataFrame method provided by SparkSession.

Also, the first field in createDataFrame is a list of rows, not a list of values for one row. So a single one-dimensional list will cause errors. Wrapping it in a dict that explicitly identifies which columns hold which values is one solution, but there might be others.

The result should look something like:

df_test = spark.createDataFrame(
    [{'col1': 'a string', 'col2': 1}],
    schema = types.StructType([
        types.StructField('col1', types.StringType(), True),
        types.StructField('col2', types.IntegerType(), True)
    ])
)