How can I install lxml dependencies on Amazon EC2 linux?

Question

I tried:

sudo yum install libxml2-dev libxslt-dev python-dev

and I got:

Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                         | 2.1 kB     00:00     
amzn-updates/latest                                      | 2.3 kB     00:00     
No package libxml2-dev available.
No package libxslt-dev available.
No package python-dev available.

pip did not work either.

I got the package like this:

git clone --branch lxml-3.2.4 https://github.com/lxml/lxml

Then I tried without cython because sudo python setup.py install gave me RuntimeError: ERROR: Trying to build without Cython, but pre-generated 'src/lxml/lxml.etree.c' is not available

sudo python setup.py install --without-cython

Then I got:

sudo python setup.py install --without-cython
Building lxml version 3.2.4.
WARNING: Trying to build without Cython, but pre-generated 'src/lxml/lxml.etree.c' is not available.
WARNING: Trying to build without Cython, but pre-generated 'src/lxml/lxml.objectify.c' is not available.
Building without Cython.
ERROR: /bin/sh: xslt-config: command not found

** make sure the development packages of libxml2 and libxslt are installed **

Using build configuration of libxslt 
/usr/lib64/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'bugtrack_url'
  warnings.warn(msg)
running install
running bdist_egg
running egg_info
writing src/lxml.egg-info/PKG-INFO
writing top-level names to src/lxml.egg-info/top_level.txt
writing dependency_links to src/lxml.egg-info/dependency_links.txt
reading manifest file 'src/lxml.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'lxml.etree.c' under directory 'src/lxml'
warning: no files found matching 'lxml.objectify.c' under directory 'src/lxml'
warning: no files found matching 'lxml.etree.h' under directory 'src/lxml'
warning: no files found matching 'lxml.etree_api.h' under directory 'src/lxml'
warning: no files found matching '*.txt' under directory 'src/lxml/tests'
warning: no files found matching '*.html' under directory 'doc'
writing manifest file 'src/lxml.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-2.6/lxml/includes
running build_ext
building 'lxml.etree' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/ec2-user/lxml/src/lxml/includes -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o
gcc: error: src/lxml/lxml.etree.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4
[ec2-user@ip-172-31-14-83 lxml]$ 

It really wants me to make sure the development packages of libxml2 and libxslt are installed.

If I can do that I hope it will work. Any ideas? Thanks! It seems people have related issues to this, but most are on other systems.

---

UPDATE: Based on Vor's suggestion I used the -devel extension and that worked to install the dependencies.

sudo yum install libxml2-devel libxslt-devel python-devel

Unfortunately I get this:

$sudo python setup.py install --without-cython


Building lxml version 3.4.0.
WARNING: Trying to build without Cython, but pre-generated 'src/lxml/lxml.etree.c' is not available.
WARNING: Trying to build without Cython, but pre-generated 'src/lxml/lxml.objectify.c' is not available.
Building without Cython.
Using build configuration of libxslt 1.1.28
Building against libxml2/libxslt in the following directory: /usr/lib64
/usr/lib64/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'bugtrack_url'
  warnings.warn(msg)
running install
running bdist_egg
running egg_info
writing requirements to src/lxml.egg-info/requires.txt
writing src/lxml.egg-info/PKG-INFO
writing top-level names to src/lxml.egg-info/top_level.txt
writing dependency_links to src/lxml.egg-info/dependency_links.txt
reading manifest file 'src/lxml.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'lxml.etree.c' under directory 'src/lxml'
warning: no files found matching 'lxml.objectify.c' under directory 'src/lxml'
warning: no files found matching 'lxml.etree.h' under directory 'src/lxml'
warning: no files found matching 'lxml.etree_api.h' under directory 'src/lxml'
warning: no files found matching '*.html' under directory 'doc'
writing manifest file 'src/lxml.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-2.6/lxml/includes
running build_ext
building 'lxml.etree' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -I/home/ec2-user/lxml/src/lxml/includes -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o -w
gcc: error: src/lxml/lxml.etree.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4

Answer 1

I had the same problem. After this line:

sudo yum install libxml2-devel libxslt-devel python-devel

I tried again with

sudo yum install lxml

And it worked :)

Answer 2

The problem on EC2 t.micro is that you run out of memory to compile the lxml package.

When you run: dmesg | tail you will see the issue

[167588.470868] Out of memory: Kill process 9804 (cc1) score 499 or sacrifice child [167588.475046] Killed process 9804 (cc1) total-vm:693864kB, anon-rss:521696kB, file-rss:0kB

The only way is to install the python-lxml package via yum or other package manager.

Answer 3

sudo yum install libxml2-devel libxslt-devel python-devel

add

sudo yum install gcc

and this worked for me.