How to create dataframe from list in Spark SQL?

Question

Spark version : 2.1

For example, in pyspark, i create a list

test_list = [['Hello', 'world'], ['I', 'am', 'fine']]

then how to create a dataframe form the test_list, where the dataframe's type is like below:

DataFrame[words: array<string>]

Answer 1

here is how -

from pyspark.sql.types import *  cSchema = StructType([StructField("WordList", ArrayType(StringType()))])  # notice extra square brackets around each element of list  test_list = [['Hello', 'world']], [['I', 'am', 'fine']]  df = spark.createDataFrame(test_list,schema=cSchema)  

Answer 2

i had to work with multiple columns and types - the example below has one string column and one integer column. A slight adjustment to Pushkr's code (above) gives:

from pyspark.sql.types import *  cSchema = StructType([StructField("Words", StringType())\                       ,StructField("total", IntegerType())])  test_list = [['Hello', 1], ['I am fine', 3]]  df = spark.createDataFrame(test_list,schema=cSchema)  

output:

 df.show()  +---------+-----+ |    Words|total| +---------+-----+ |    Hello|    1| |I am fine|    3| +---------+-----+