PySpark Dataframe from Python Dictionary without Pandas

Question

I am trying to convert the following Python dict into PySpark DataFrame but I am not getting expected output.

dict_lst = {'letters': ['a', 'b', 'c'], 
             'numbers': [10, 20, 30]}
df_dict = sc.parallelize([dict_lst]).toDF()  # Result not as expected
df_dict.show()

Is there a way to do this without using Pandas?

Answer 1

Quoting myself:

I find it's useful to think of the argument to createDataFrame() as a list of tuples where each entry in the list corresponds to a row in the DataFrame and each element of the tuple corresponds to a column.

So the easiest thing is to convert your dictionary into this format. You can easily do this using zip():

column_names, data = zip(*dict_lst.items())
spark.createDataFrame(zip(*data), column_names).show()
#+-------+-------+
#|letters|numbers|
#+-------+-------+
#|      a|     10|
#|      b|     20|
#|      c|     30|
#+-------+-------+

The above assumes that all of the lists are the same length. If this is not the case, you would have to use itertools.izip_longest (python2) or itertools.zip_longest (python3).

from itertools import izip_longest as zip_longest # use this for python2
#from itertools import zip_longest # use this for python3

dict_lst = {'letters': ['a', 'b', 'c'], 
             'numbers': [10, 20, 30, 40]}

column_names, data = zip(*dict_lst.items())

spark.createDataFrame(zip_longest(*data), column_names).show()
#+-------+-------+
#|letters|numbers|
#+-------+-------+
#|      a|     10|
#|      b|     20|
#|      c|     30|
#|   null|     40|
#+-------+-------+

Answer 2

Your dict_lst is not really the format you want to adopt to create a dataframe. It would be better if you had a list of dict instead of a dict of list.

This code creates a DataFrame from you dict of list :

from pyspark.sql import SQLContext, Row

sqlContext = SQLContext(sc)

dict_lst = {'letters': ['a', 'b', 'c'], 
             'numbers': [10, 20, 30]}

values_lst = dict_lst.values()
nb_rows = [len(lst) for lst in values_lst]
assert min(nb_rows)==max(nb_rows) #We must have the same nb of elem for each key

row_lst = []
columns = dict_lst.keys()

for i in range(nb_rows[0]):
    row_values = [lst[i] for lst in values_lst]
    row_dict = {column: value for column, value in zip(columns, row_values)}
    row = Row(**row_dict)
    row_lst.append(row)

df = sqlContext.createDataFrame(row_lst)