How can I use io.StringIO() with the csv module?

Question

I tried to backport a Python 3 program to 2.7, and I'm stuck with a strange problem:

>>> import io
>>> import csv
>>> output = io.StringIO()
>>> output.write("Hello!")            # Fail: io.StringIO expects Unicode
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unicode argument expected, got 'str'
>>> output.write(u"Hello!")           # This works as expected.
6L
>>> writer = csv.writer(output)       # Now let's try this with the csv module:
>>> csvdata = [u"Hello", u"Goodbye"]  # Look ma, all Unicode! (?)
>>> writer.writerow(csvdata)          # Sadly, no.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unicode argument expected, got 'str'

According to the docs, io.StringIO() returns an in-memory stream for Unicode text. It works correctly when I try and feed it a Unicode string manually. Why does it fail in conjunction with the csv module, even if all the strings being written are Unicode strings? Where does the str come from that causes the Exception?

_{(I do know that I can use StringIO.StringIO() instead, but I'm wondering what's wrong with io.StringIO() in this scenario)}

Answer 1

The Python 2.7 csv module doesn't support Unicode input: see the note at the beginning of the documentation.

It seems that you'll have to encode the Unicode strings to byte strings, and use io.BytesIO, instead of io.StringIO.

The examples section of the documentation includes examples for a UnicodeReader and UnicodeWriter wrapper classes (thanks @AlexeyKachayev for the pointer).

Answer 2

Please use StringIO.StringIO().

http://docs.python.org/library/io.html#io.StringIO

http://docs.python.org/library/stringio.html

io.StringIO is a class. It handles Unicode. It reflects the preferred Python 3 library structure.

StringIO.StringIO is a class. It handles strings. It reflects the legacy Python 2 library structure.

Answer 3

I found this when I tried to serve a CSV file via Flask directly without creating the CSV file on the file system. This works:

import io
import csv

data = [[u'cell one', u'cell two'], [u'cell three', u'cell four']]

output = io.BytesIO()
writer = csv.writer(output, delimiter=',')
writer.writerows(data)
your_csv_string = output.getvalue()

See also

• More about CSV
• The Flask part
• A few notes about Strings / Unicode