How to correctly parse utf-8 xml with ElementTree?

Question

I need help to understand why parsing my xml file* with xml.etree.ElementTree produces the following errors.

*My test xml file contains arabic characters.

Task: Open and parse utf8_file.xml file.

My first try:

import xml.etree.ElementTree as etree
with codecs.open('utf8_file.xml', 'r', encoding='utf-8') as utf8_file:
    xml_tree = etree.parse(utf8_file)

Result 1:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 236-238: ordinal not in range(128)

My second try:

import xml.etree.ElementTree as etree
with codecs.open('utf8_file.xml', 'r', encoding='utf-8') as utf8_file:
    xml_string = etree.tostring(utf8_file, encoding='utf-8', method='xml')
    xml_tree  = etree.fromstring(xml_string)

Result 2:

AttributeError: 'file' object has no attribute 'getiterator'

Please explain the errors above and comment on the possible solution.

Answer 1

Leave decoding the bytes to the parser; do not decode first:

import xml.etree.ElementTree as etree
with open('utf8_file.xml', 'r') as xml_file:
    xml_tree = etree.parse(xml_file)

An XML file must contain enough information in the first line to handle decoding by the parser. If the header is missing, the parser must assume UTF-8 is used.

Because it is the XML header that holds this information, it is the responsibility of the parser to do all decoding.

Your first attempt failed because Python was trying to encode the Unicode values again so that the parser could handle byte strings as it expected. The second attempt failed because etree.tostring() expects a parsed tree as first argument, not a unicode string.