Should memory usage increase when using ElementTree.iterparse() when clear()ing trees?

Question

import os
import xml.etree.ElementTree as et

for ev, el in et.iterparse(os.sys.stdin):
    el.clear()

Running the above on the ODP structure RDF dump results in always increasing memory. Why is that? I understand ElementTree still builds a parse tree, albeit with the child nodes clear()ed. If that is the cause of this memory usage pattern, is there a way around it?

Answer 1

You are clearing each element but references to them remain in the root document. So the individual elements still cannot be garbage collected.

The solution is to clear references in the root, like so:

import xml.etree.ElementTree as ET

# get iterator
context = ET.iterparse(source, events=("start", "end"))

# get the root element
event, root = next(context)

for event, elem in context:
    if event == "end" and elem.tag == "record":
        # process record elements here...
        root.clear()

Another thing to remember about memory usage, which may not be affecting your situation, is that once the VM allocates memory for heap storage from the system, it generally never gives that memory back. Most Java VMs work this way too. So you should not expect the size of the interpreter in top or ps to ever decrease, even if that heap memory is unused.

update :

Code changed in order to work in Python 3+.

Answer 2

As mentioned in the answer by Kevin Guerra, the "root.clear()" strategy in the ElementTree documentation only removes fully parsed children of the root. If those children are anchoring huge branches, it's not very helpful.

He touched on the ideal solution, but didn't post any code, so here is an example:

element_stack = []
context = ET.iterparse(stream, events=('start', 'end'))
for event, elem in context:
    if event == 'start':
        element_stack.append(elem)
    elif event == 'end':
        element_stack.pop()
        # see if elem is one of interest and do something with it here
        if element_stack:
            element_stack[-1].remove(elem)
del context

The element of interest will not have subelements; they'll have been removed as soon as their end tags were seen. This might be OK if all you need is the element's text or attributes.

If you want to query into the element's descendants, you need to let a full branch be built for it. For this, maintain a flag, implemented as a depth counter for those elements. Only call .remove() when the depth is zero:

element_stack = []
interesting_element_depth = 0
context = ET.iterparse(stream, events=('start', 'end'))
for event, elem in context:
    if event == 'start':
        element_stack.append(elem)
        if elem.tag == 'foo':
            interesting_element_depth += 1
    elif event == 'end':
        element_stack.pop()
        if elem.tag == 'foo':
            interesting_element_depth -= 1
            # do something with elem and its descendants here
        if element_stack and not interesting_element_depth:
            element_stack[-1].remove(elem)
del context