Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why doesn't xpath work when processing an XHTML document with lxml (in python)?

Tags:

I am testing against the following test document:

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"                        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml">    <head>         <title>hi there</title>     </head>     <body>         <img class="foo" src="bar.png"/>     </body> </html> 

If I parse the document using lxml.html, I can get the IMG with an xpath just fine:

>>> root = lxml.html.fromstring(doc) >>> root.xpath("//img") [<Element img at 1879e30>] 

However, if I parse the document as XML and try to get the IMG tag, I get an empty result:

>>> tree = etree.parse(StringIO(doc)) >>> tree.getroot().xpath("//img") [] 

I can navigate to the element directly:

>>> tree.getroot().getchildren()[1].getchildren()[0] <Element {http://www.w3.org/1999/xhtml}img at f56810> 

But of course that doesn't help me process arbitrary documents. I would also expect to be able to query etree to get an xpath expression that will directly identify this element, which, technically I can do:

>>> tree.getpath(tree.getroot().getchildren()[1].getchildren()[0]) '/*/*[2]/*' >>> tree.getroot().xpath('/*/*[2]/*') [<Element {http://www.w3.org/1999/xhtml}img at fa1750>] 

But that xpath is, again, obviously not useful for parsing arbitrary documents.

Obviously I am missing some key issue here, but I don't know what it is. My best guess is that it has something to do with namespaces but the only namespace defined is the default and I don't know what else I might need to consider in regards to namespaces.

So, what am I missing?

like image 235
John Avatar asked Nov 17 '08 22:11

John


1 Answers

The problem is the namespaces. When parsed as XML, the img tag is in the http://www.w3.org/1999/xhtml namespace since that is the default namespace for the element. You are asking for the img tag in no namespace.

Try this:

>>> tree.getroot().xpath( ...     "//xhtml:img",  ...     namespaces={'xhtml':'http://www.w3.org/1999/xhtml'} ...     ) [<Element {http://www.w3.org/1999/xhtml}img at 11a29e0>] 
like image 61
Ned Batchelder Avatar answered Oct 02 '22 16:10

Ned Batchelder