Python lxml etree.tostring() returns empty string running on mod_wsgi

Question

I have Python 2.7.8 on CentOS 6.8 where my server is built upon Apache2 + WSGI. My application should handle the date received with http POST, then create an instruction based on XML template gotten from local .xml file. Finally, it must send xml instruction back with 200 OK response. The logic of the application is seems working Ok and I may see my updated xml tree with:

print etree.tostring(root, pretty_print=True, xml_declaration-True, encoding='UTF-8')

The issue appears to be happened at next row of my code where I'm doing the same operation but trying assign the output to variable:

xml_body = etree.tostring(root, pretty_print=True, xml_declaration-True, encoding='UTF-8')

print xml_body

The output is empty string, thus my application then returns nothing back to Apache.

My environment info may be helpful:

==For bug report ===
Python              : sys.version_info(major=2, minor=7, micro=8, releaselevel='final', serial=0)
lxml.etree          : (3, 6, 4, 0)
libxml used         : (2, 7, 6)
libxml compiled     : (2, 9, 4)
libxslt used        : (1, 1, 26)
libxslt compiled    : (1, 1, 29)

It looks similar to this bug report However, the author mentioned it doesn't work at all. There is another one request which contains similar issue, but it still isn't resolved. I've checked out I might successfully play the same scenario in Python cli:

Python 2.7.8 (default, May 15 2016, 12:46:09) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
>>> foo = etree.Element('foo')
>>> foo
<Element foo at 0x7f5097c4fc20>
>>> foo.tag
'foo'
>>> foo.text = 'barrisimo'

>>> xmlb = etree.tostring(foo, pretty_print=True, xml_declaration=True, encoding='UTF-8')
>>> print xmlb
<?xml version='1.0' encoding='UTF-8'?>
<foo>barrisimo</foo>1

Have somebody come across the same issue ever? I'm in dead end and I'd be appreciated for any help, ideas or helpful links.

Answer 1

Have the same issue.

My test file test.py:

from lxml import etree


def application(env, start_response):
    foo = etree.Element("foo")
    foo.text = "bar"
    out = etree.tostring(foo)

    print('-------------')
    print(foo)
    print('=============')
    print(out)
    print('-------------')

    if start_response:
        start_response('200 OK', [('Content-Type', 'text/html')])

    return [b"TEST"]


if __name__ == "__main__":
    application(None, None)

When run in cli mode:

python test.py

It returns:

-------------
<Element foo at 0x7f5b80671488>
=============
<foo>bar</foo>
-------------

But if I run in wsgi mode, it returns None:

uwsgi --http :9090 --wsgi-file test.py

Then receive

-------------
<Element foo at 0x7f8f84aa7638>
=============

-------------

Pip freeze:

lxml==3.7.2
uWSGI==2.0.14

Lib versions:

libxml used         : (2, 9, 3)
libxml compiled     : (2, 9, 3)
libxslt used        : (1, 1, 29)
libxslt compiled    : (1, 1, 29)

Core reason of issue is unknown, but problem was solved by removing package libxml2-dev and reinstalling uwsgi:

aptitude remove libxml2-dev
pip uninstall uwsgi
pip install --no-cache-dir uwsgi

So, somehow package libxml2-dev is not compatible with uwsgi.

Answer 2

In case the libxml version at runtime is different from the compiled version it means there are more than one libxml version installed in your system:

libxml used         : (2, 7, 8)
libxml compiled     : (2, 9, 3)

In my case mod_php5 was the problem. I have fixed this by re-compiling apache without PHP. It turns out PHP was using libxml version 2.7.8 and the python interpreter used in mod_wsgi was pointing to the same xmllib instead of using the python one.