Add empty column to dataframe in Spark with python

Question

I have a dataframe that I want to make a unionAll with another dataframe. The problem is that the second dataframe has three more columns than the first one. Is there a way for me to add three columns with only empty cells in my first dataframe?

Answer 1

Maybe this will help

To add string type column:

from pyspark.sql.types import StringType

df.withColumn("COL_NAME", lit(None).cast(StringType()))

To Add integer type

from pyspark.sql.types import IntegerType

df.withColumn("COL_NAME", lit(0).cast(IntegerType()))