Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python 2 CSV writer produces wrong line terminator on Windows

According to the its documentation csv.writer should use '\r\n' as lineterminator by default.

import csv

with open("test.csv", "w") as f:
    writer = csv.writer(f)

    rows = [(0,1,2,3,4),
           (-0,-1,-2,-3,-4),
           ("a","b","c","d","e"),
           ("A","B","C","D","E")]           

    print writer.dialect.lineterminator.replace("\r", "\\r").replace("\n", "\\n")
    writer.writerows(rows)
    print writer.dialect.lineterminator.replace("\r", "\\r").replace("\n", "\\n")

This prints

\r\n
\r\n

as expected. But, the created csv-file uses the lineterminator '\r\r\n'

0,1,2,3,4

0,-1,-2,-3,-4

a,b,c,d,e

A,B,C,D,E

Is this a bug or is there something wrong in my usage of csv.writer?

Python version:

ActivePython 2.6.2.2 (ActiveState Software Inc.) based on Python 2.6.2 (r262:71600, Apr 21 2009, 15:05:37) [MSC v.1500 32 bit (Intel)] on win32

on Windows Vista

like image 500
wierob Avatar asked Jul 23 '09 07:07

wierob


People also ask

Why is csv Writer adding newline?

This function in csv module returns a writer object that converts data into a delimited string and stores in a file object. The function needs a file object created with open() function and with write permission as a parameter. Every row written in the file issues a newline character by default.

How do you use line terminator in Python?

A lineterminator is a string used to terminate lines produced by writer objects. The default value is \r\n . You can change its value by passing any string as a lineterminator parameter. However, the reader object only recognizes \n or \r as lineterminator values.

What is newline in Python csv?

Including the newline parameter allows the csv module to handle the line endings itself - replicating the format as defined in your csv. Follow this answer to receive notifications.


2 Answers

In Python 2.x, always open your file in binary mode, as documented. csv writes \r\n as you expected, but then the underlying Windows text file mechanism cuts in and changes that \n to \r\n ... total effect: \r\r\n

From the csv.writer documentation:

If csvfile is a file object, it must be opened with the 'b' flag on platforms where that makes a difference.

There seems to be some reticence about actually uttering the name of the main culprit :-)

Edit: As mentioned by @jebob in the comments to this answer and based on @Dave Burton's answer, to handle this case in both Python 2 and 3, you should do the following:

if sys.version_info >= (3,0,0):     f = open(filename, 'w', newline='') else:     f = open(filename, 'wb') 
like image 196
John Machin Avatar answered Oct 08 '22 12:10

John Machin


Unfortunately, it's a bit different with the csv module for Python 3, but this code will work on both Python 2 and Python 3:

if sys.version_info >= (3,0,0):
    f = open(filename, 'w', newline='')
else:
    f = open(filename, 'wb')
like image 25
Dave Burton Avatar answered Oct 08 '22 12:10

Dave Burton