python lxml findall with multiple namespaces

Question

I'm trying to parse an XML document with multiple namespaces with lxml, and I'm stuck on getting the findall() method to return something.

My XML:

<MeasurementRecords xmlns="http://www.company.com/common/rsp/2012/07"
                    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         
                    xsi:schemaLocation="http://www.company.com/common/rsp/2012/07 RSP_EWS_V1.6.xsd">
    <HistoryRecords>
        <ValueItemId>100_0000100004_3788_Resource-0.customId_WSx Data Precip Type</ValueItemId>
            <List>
                <HistoryRecord>
                    <Value>60</Value>
                    <State>Valid</State>
                    <TimeStamp>2016-04-20T12:40:00Z</TimeStamp>
                </HistoryRecord>
            </List>
        </HistoryRecords>
    <HistoryRecords>
</MeasurementRecords>

My code:

from lxml import etree
from pprint import pprint

RSPxmlFile = '/home/user/Desktop/100_0000100004_3788_20160420144011263_records.xml'

with open (RSPxmlFile, 'rt') as f:
    tree = etree.parse(f)

root = tree.getroot()

for node in tree.findall('MeasurementRecords', root.nsmap):
    print node
    print "parameter = ", node.text

Gives:

ValueError: empty namespace prefix is not supported in ElementPath

Some experiments I've tried after reading this:

>>> root.nsmap
{'xsi': 'http://www.w3.org/2001/XMLSchema-instance', None: http://www.company.com/common/rsp/2012/07'}

>>> nsmap['foo']=nsmap[None]
>>> nsmap.pop(None)
'http://www.company.com/common/rsp/2012/07'
>>> nsmap
{'xsi': 'http://www.w3.org/2001/XMLSchema-instance', 'foo': 'http://www.company.com/common/rsp/2012/07'}
>>> tree.xpath("//MeasurementRecords", namespaces=nsmap)
[]
>>> tree.xpath('/foo:MeasurementRecords', namespaces=nsmap)
[<Element {http://www.company.com/common/rsp/2012/07}MeasurementRecords at 0x6ffffda5290>]
>>> tree.xpath('/foo:MeasurementRecords/HistoryRecords', namespaces=nsmap)
[]

But that didn't seem to help.

So, more experiments:

>>> tree.findall('//{http://www.company.com/common/rsp/2012/07}MeasurementRecords')
[]
>>> print root
<Element {http://www.company.com/common/rsp/2012/07}MeasurementRecords at 0x6ffffda5290>
>>> print tree
<lxml.etree._ElementTree object at 0x6ffffda5368>
>>> for node in tree.iter():
...     print node
...
<Element {http://www.company.com/common/rsp/2012/07}MeasurementRecords at 0x6ffffda5290>
<Element {http://www.company.com/common/rsp/2012/07}HistoryRecords at 0x6ffffda5cf8>
<Element {http://www.company.com/common/rsp/2012/07}ValueItemId at 0x6ffffda5f38>
...etc...
>>> tree.findall("//HistoryRecords", namespaces=nsmap)
[]
>>> tree.findall("//foo:MeasurementRecords/HistoryRecords", namespaces=nsmap)
[]

I'm stumped. I have no idea what's wrong.

Answer 1

If you start with this:

>>> tree = etree.parse(open('data.xml'))
>>> root = tree.getroot()
>>> 

This will fail to find any elements...

>>> root.findall('{http://www.company.com/common/rsp/2012/07}MeasurementRecords')
[]

...but that's because root is a MeasurementRecords element; it does not contain any MeasurementRecords elements. On the other hand, the following works just fine:

>>> root.findall('{http://www.company.com/common/rsp/2012/07}HistoryRecords')
[<Element {http://www.company.com/common/rsp/2012/07}HistoryRecords at 0x7fccd0332ef0>]
>>> 

Using the xpath method, you could do something like this:

>>> nsmap={'a': 'http://www.company.com/common/rsp/2012/07',
... 'b': 'http://www.w3.org/2001/XMLSchema-instance'}
>>> root.xpath('//a:HistoryRecords', namespaces=nsmap)
[<Element {http://www.company.com/common/rsp/2012/07}HistoryRecords at 0x7fccd0332ef0>]

So:

• The findall and find methods require {...namespace...}ElementName syntax.
• The xpath method requires namespace prefixes (ns:ElementName), which it looks up in the provided namespaces map. The prefix doesn't have to match the prefix used in the original document, but the namespace url must match.

So this works:

>>> root.find('{http://www.company.com/common/rsp/2012/07}HistoryRecords/{http://www.company.com/common/rsp/2012/07}ValueItemId')
<Element {http://www.company.com/common/rsp/2012/07}ValueItemId at 0x7fccd0332a70>

Or this works:

>>> root.xpath('/a:MeasurementRecords/a:HistoryRecords/a:ValueItemId',namespaces=nsmap)
[<Element {http://www.company.com/common/rsp/2012/07}ValueItemId at 0x7fccd0330830>]