Why do Python unicode strings require special treatment for UTF-8 BOM?

Question

For some reason, Python seems to be having issues with BOM when reading unicode strings from a UTF-8 file. Consider the following:

with open('test.py') as f:
   for line in f:
      print unicode(line, 'utf-8')

Seems straightforward, doesn't it?

That's what I thought until I ran it from command line and got:

UnicodeEncodeError: 'charmap' codec can't encode character u'\ufeff' in position 0: character maps to <undefined>

A brief visitation to Google revealed that BOM has to be cleared manually:

import codecs
with open('test.py') as f:
   for line in f:
      print unicode(line.replace(codecs.BOM_UTF8, ''), 'utf-8')

This one runs fine. However I'm struggling to see any merit in this.

Is there a rationale behind above-described behavior? In contrast, UTF-16 works seamlessly.

Answer 1

The 'utf-8-sig' encoding will consume the BOM signature on your behalf.