Django Filefield UnicodeDecodeError on model.save

Question

None of the similar questions on StackOverflow seem to have the same problem as me, I've only found http://qnundrum.com/question/766895, which was never answered. I'm running Python 3.3 and Django 1.6, so usually Unicode stuff is automatically taken care of. Appreciate any help I can get.

I'm trying to save automatically generated ebooks to my database for later retrieval. Some of the books contain non-ASCII characters. I've got the generation down, .mobi and .epub's work as intended. Here's models.py:

class StoryDownload(models.Model):
    text = models.OneToOneField('stories.Story', primary_key=True, related_name='downloads')
    epub = models.FileField(upload_to='epub/', blank=True, null=True)
    mobi = models.FileField(upload_to='mobi/', blank=True, null=True)

    def update_downloads(self):
        #code to generate epub and mobi files from text
        ...
        self.epub = File(open('filename.epub'), 'r'))
        self.mobi = File(open('filename.mobi'), 'r'))
        self.save()
        ...

The error comes on self.save() is what confuses me; if the files are accepted as Django File objects, then why can't I save them?

Traceback:

File "C:\Users\Chris\Envs\stories\lib\site-packages\django\core\handlers\base.py" in get_response
  114.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\views\generic\base.py" in view
  69.             return self.dispatch(request, *args, **kwargs)
File "C:\Users\Chris\Envs\stories\lib\site-packages\braces\views\_access.py" in dispatch
  64.             request, *args, **kwargs)
File "C:\Users\Chris\Envs\stories\lib\site-packages\guardian\mixins.py" in dispatch
  190.             **kwargs)
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\views\generic\base.py" in dispatch
  87.         return handler(request, *args, **kwargs)
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\views\generic\edit.py" in post
  228.         return super(BaseUpdateView, self).post(request, *args, **kwargs)
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\views\generic\edit.py" in post
  171.             return self.form_valid(form)
File "C:\Users\Chris\Envs\stories\dev\akrito\chapters\views.py" in form_valid
  69.         self.chapter.story.save()
File "C:\Users\Chris\Envs\stories\dev\akrito\stories\models.py" in save
  87.             self.downloads.update_downloads()
File "C:\Users\Chris\Envs\stories\dev\akrito\stories\models.py" in update_downloads
  135.         self.save()
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\db\models\base.py" in save
  545.                        force_update=force_update, update_fields=update_fields)
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\db\models\base.py" in save_base
  573.             updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\db\models\base.py" in _save_table
  632.                       for f in non_pks]
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\db\models\base.py" in <listcomp>
  632.                       for f in non_pks]
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\db\models\fields\files.py" in pre_save
  252.             file.save(file.name, file, save=False)
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\db\models\fields\files.py" in save
  86.         self.name = self.storage.save(name, content)
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\core\files\storage.py" in save
  49.         name = self._save(name, content)
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\core\files\storage.py" in _save
  203.                         for chunk in content.chunks():
File "C:\Users\Chris\Envs\stories\lib\site-packages\django\core\files\base.py" in chunks
  76.             data = self.read(chunk_size)
File "C:\Users\Chris\Envs\stories\lib\encodings\cp1252.py" in decode
  23.         return codecs.charmap_decode(input,self.errors,decoding_table)[0]

Exception Type: UnicodeDecodeError at /stories/2/1/e/
Exception Value: 'charmap' codec can't decode byte 0x81 in position 123: character maps to <undefined>

Answer 1

For a text file you need to call open with the desired encoding. The default encoding is locale.getpreferredencoding(False), which is why the traceback shows it attempting to decode using the Windows 1252 codepage.

That said, MOBI and EPUB files are zipped archives that should be opened in binary mode, e.g. open('filename.epub', 'rb').