Python DictReader - Skipping rows with missing columns?

Question

I have a Excel .CSV file I'm attempting to read in with DictReader.

All seems to be well, except it seems to omit rows, specifically those with missing columns.

Our input looks like:

mail,givenName,sn,lorem,ipsum,dolor,telephoneNumber
[email protected],ian,bay,3424,8403,2535,+65(2)34523534545
[email protected],mike,gibson,3424,8403,2535,+65(2)34523534545
[email protected],ross,martin,,,,+65(2)34523534545
[email protected],david,connor,,,,+65(2)34523534545
[email protected],chris,call,3424,8403,2535,+65(2)34523534545

So some of the rows have missing lorem/ipsum/dolor columns, and it's just a string of commas for those.

We're reading it in with:

def read_gd_dump(input_file="blah 20100423.csv"):
    gd_extract = csv.DictReader(open('blah 20100423.csv'), restval='missing', dialect='excel')
    return dict([(row['something'], row) for row in gd_extract])

And I checked that "something" (the key for our dict) isn't one of the missing columns, I had originally suspected it might be that. It's one of the columns after that.

However, DictReader seems to completely skip over the rows. I tried setting restval to something, didn't seem to make any difference. I can't seem to find anything in Python's CSV docs (http://docs.python.org/library/csv.html) that would explain this behaviour, but I may have misread something.

Answer 1

Can't reproduce your problem -- when I save that data and then assign list(gd_extract), I see:

[{'telephoneNumber': '+65(2)34523534545', 'ipsum': '8403', 'sn': 'bay', 'dolor': '2535', 'mail': '[email protected]', 'givenName': 'ian', 'lorem': '3424'}, {'telephoneNumber': '+65(2)34523534545', 'ipsum': '8403', 'sn': 'gibson', 'dolor': '2535', 'mail': '[email protected]', 'givenName': 'mike', 'lorem': '3424'}, {'telephoneNumber': '+65(2)34523534545', 'ipsum': '', 'sn': 'martin', 'dolor': '', 'mail': '[email protected]', 'givenName': 'ross', 'lorem': ''}, {'telephoneNumber': '+65(2)34523534545', 'ipsum': '', 'sn': 'connor', 'dolor': '', 'mail': '[email protected]', 'givenName': 'david', 'lorem': ''}, {'telephoneNumber': '+65(2)34523534545', 'ipsum': '8403', 'sn': 'call', 'dolor': '2535', 'mail': '[email protected]', 'givenName': 'chris', 'lorem': '3424'}]

five dicts, including those with missing ipsum etc. I fear that in your laudable attempt at simplifying the problem you've simplified it excessively, so that your bug has gone away.

If you have duplicates in column something (can't check, since you don't have that column in your sample data) that would of course explain the "apparently missing" rows -- they're not missing from the csv reader's returned stream, they get "overwritten" in the dict you're returning. Could that be the issue?