Iterating XML with lxml in Python: how to know how much of the input file has been read?

Question

I'm using lxml.etree.iterparse() to iterate through a large XML file.

I would like to know how far I've got in the parsing of the input file, so that I might get a progress indicator.

My first idea was to use os.stat( filename ).st_size to know how big is my XML file, then as I'm getting events from the parser, retrieve the current position in the file. But I can't figure out how lxml.etree could give me access to its internal position. iterparse() is taking a filename as its source argument, so I can't open myself the file and call its tell() method to known how many bytes have been read so far.

Are you aware of any lxml.etree built-in indicator for current parser progression ? Or do you have an idea of integrating such a progression ?

Answer 1

You could pass a file object to iterparse, and then call f.tell(). This will give you the approximate position of the Element in the file.

import lxml.etree as ET
import os

filename = 'data.xml'
total_size = os.path.getsize(filename)
with open(filename, 'r') as f:
    context = ET.iterparse(f, events=('end', ), tag='Record')
    for event, elem in context:
        print(event, elem, float(f.tell())/total_size)

will yield something like

(u'end', <Element Record at 0xb743e2d4>, 0.09652665470688218)
(u'end', <Element Record at 0xb743e2fc>, 0.09652665470688218)
(u'end', <Element Record at 0xb743e324>, 0.09652665470688218)
...
(u'end', <Element Record at 0xb744739c>, 1.0)
(u'end', <Element Record at 0xb74473c4>, 1.0)
(u'end', <Element Record at 0xb74473ec>, 1.0)