Convert list of dicts to CSV

Question

I'm trying to convert this list of dicts to csv, first of all my data is contained in a Counter dictionnary, but since it's like a dict, I figured it would be the same thing as a dict when it comes to using it. So this is what I do :

My counter dict looks like this :

 counterdict =  {1:Counter({u'a':1, u'b':1, u'c':3}),2:Counter({u'd':1, u'f':4, u'e':2})...}

I convert it to a list of dicts like this :

new_dict = [dict(d,Month=k) for k, d in counterdict.iteritems()]

to get :

new_dict = [{Month :1, u'a':1, u'b':1, u'c':3}, {Month : 2,u'd':1, u'f':4, u'e':2}...]

Then I want to convert new_dict data to csv :

out_path= "outfile.csv"
out_file = open(out_path, 'wb')
new_dict = [dict(d,Month=k) for k, d in counterdict.iteritems()]
writer = DictWriter(out_file, fieldnames=allFields, dialect='excel')
for row in new_dict:
    writer.writerow(row)
out_file.close()

But I get the error :

Traceback (most recent call last):
 File "C:/wamp/www/metrics1/script/cgi/translate_parameters.py", line 333, in <module>
writer.writerow(row)
File "C:\Python27\lib\csv.py", line 148, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
 File "C:\Python27\lib\csv.py", line 141, in _dict_to_list
wrong_fields = [k for k in rowdict if k not in self.fieldnames]
TypeError: argument of type 'NoneType' is not iterable

Help please!!

Edit:

allFields comes from counterdict values like this :

allFields = list(set().union(*counterdict.values()))

So this gives me a list of all fields of counterdict.

Answer 1

You set the fieldnames argument to None; you need to make sure allFields is an ordered sequence of strings instead.

Demo illustrating the problem:

>>> from cStringIO import StringIO
>>> import csv
>>> w = csv.DictWriter(StringIO(), fieldnames=None)
>>> w.writerow({'foo':'bar'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/lib/python2.7/csv.py", line 148, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
  File "/opt/lib/python2.7/csv.py", line 141, in _dict_to_list
    wrong_fields = [k for k in rowdict if k not in self.fieldnames]
TypeError: argument of type 'NoneType' is not iterable
>>> w = csv.DictWriter(StringIO(), fieldnames=['foo'])
>>> w.writerow({'foo':'bar'})

Answer 2

This task is a nice opportunity to use a library, such as pandas, which is built to work automatically with lists of dicts. Do:

import pandas as pd
df = pd.DataFrame(list_of_dicts)
df = df.set_index("first_column")
df.to_csv("filename")