iterator should return strings, not bytes (did you open the file in text mode?) while uploading csv in django

Question

I want to upload csv file and store in database. when I click the submit button I get the following error

Environment:


Request Method: POST
Request URL: http://localhost:8000/csv

Django Version: 1.9
Python Version: 3.4.3
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.gis',
 'django.contrib.admindocs',
 'world',
 'pft',
 'wms',
 'djgeojson',
 'html5',
 'geoexplorer']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "C:\Python34\lib\site-packages\django\core\handlers\base.py" in get_response
  149.                     response = self.process_exception_by_middleware(e, request)

File "C:\Python34\lib\site-packages\django\core\handlers\base.py" in get_response
  147.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "C:\Python34\lib\site-packages\django\contrib\auth\decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)

File "C:\Python34\Lib\site-packages\django\bin\geodjango\pft\views.py" in add_multiple_accounts
  95.              csv_result, rows_error = utils.handle_uploaded_file(file, utils.account_valid_fields, utils.create_account_in_db)

File "C:\Python34\Lib\site-packages\django\bin\geodjango\pft\utils.py" in handle_uploaded_file
  24.     if not valid_fields_method(data.fieldnames):

File "C:\Python34\lib\csv.py" in fieldnames
  96.                 self._fieldnames = next(self.reader)

Exception Type: Error at /csv
Exception Value: iterator should return strings, not bytes (did you open the file in text mode?)

my handle file upload function is below-utils.py

def handle_uploaded_file(file, valid_fields_method, record_creation_function):
    file.seek(0)
    sniffdialect = csv.reader(codecs.iterdecode(file, 'utf-8')
    file.seek(0)
    data = csv.DictReader(file, dialect=sniffdialect)

    if not valid_fields_method(data.fieldnames):
        return False, -1

    result, rows_error = record_creation_function(data)

    return result, rows_error

I'm using django 1.9 and python 3.4. Please help ! I'm struggling for past 2 days.

Answer 1

TL;DR:

Change this:

file.seek(0)
sniffdialect = csv.reader(codecs.iterdecode(file, 'utf-8')
file.seek(0)
data = csv.DictReader(file, dialect=sniffdialect)

To this:

file.seek(0)
data = csv.DictReader(codecs.iterdecode(file, 'utf-8'))

(By the way, I'd check if seek(0) is really needed.)

---

Explanation:

You have understood that you need to decode the contents from your file, which is correct. However, to do so, you are using dialect, which is wrong.

dialect is not meant for this kind of stuff. Also, it should be a name registered via csv.register_dialect(), not an arbitrary iterator. But anyhow, dialects can't help you here.

Instead, you should pass the decoded stream directly to DictReader.