How to create a sample single-column Spark DataFrame in Python?

Question

I want to create a sample single-column DataFrame, but the following code is not working:

df = spark.createDataFrame(["10","11","13"], ("age"))  ## ValueError ## ... ## ValueError: Could not parse datatype: age 

The expected result:

age 10 11 13 

Answer 1

the following code is not working

With single element you need a schema as type

spark.createDataFrame(["10","11","13"], "string").toDF("age") 

or DataType:

from pyspark.sql.types import StringType  spark.createDataFrame(["10","11","13"], StringType()).toDF("age") 

With name elements should be tuples and schema as sequence:

spark.createDataFrame([("10", ), ("11", ), ("13",  )], ["age"]) 

Answer 2

Well .. There is some pretty easy method for creating sample dataframe in PySpark

>>> df = sc.parallelize([[1,2,3], [2,3,4]]).toDF() >>> df.show() +---+---+---+ | _1| _2| _3| +---+---+---+ |  1|  2|  3| |  2|  3|  4| +---+---+---+ 

to create with some column names

>>> df1 = sc.parallelize([[1,2,3], [2,3,4]]).toDF(("a", "b", "c")) >>> df1.show() +---+---+---+ |  a|  b|  c| +---+---+---+ |  1|  2|  3| |  2|  3|  4| +---+---+---+ 

In this way, no need to define schema too.Hope this is the simplest way