Repeatedly failing to install scrapy and lxml

Question

I'd previously used Anaconda to handle python, but I'm and start working with virtual environments.

I set up virtualenv and virtualenvwrapper, and have been trying to add modules, specifically scrapy and lxml, for a project I want to try.

Each time I pip install, I hit an error.

For scrapy:

File "/home/philip/Envs/venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1003, in run_setup
    raise DistutilsError("Setup script exited with %s" % (v.args[0],))

distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

---------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /home/philip/Envs/venv/build/cryptography
Storing debug log for failure in /home/philip/.pip/pip.log

For lxml:

In file included from src/lxml/lxml.etree.c:346:0:

/home/philip/Envs/venv/build/lxml/src/lxml/includes/etree_defs.h:9:31: fatal error: libxml/xmlversion.h: No such file or directory

 include "libxml/xmlversion.h"

                               ^

compilation terminated.

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

----------------------------------------
Cleaning up... Command /home/philip/Envs/venv/bin/python -c "import setuptools, tokenize;__file__='/home/philip/Envs/venv/build/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-zIsPdl-record/install-record.txt
--single-version-externally-managed --compile --install-headers /home/philip/Envs/venv/include/site/python2.7 failed with error code 1 in /home/philip/Envs/venv/build/lxml Storing debug log for failure in /home/philip/.pip/pip.log

I tried to install it following scrapy's documentation, but scrapy was still not listed when I called for python's installed modules.

Any ideas? Thanks--really appreciate it!

I'm on Ubuntu 13.10 if it matters. Other modules I've tried have installed fine (though I've only gone for a handful).

Answer 1

I had the same problem in Ubuntu 14.04. I've solved it with the instructions of the page linked by @jdigital and the openssl-dev library pointed by @user3115915. Just to help others:

sudo apt-get install libxslt1-dev libxslt1.1 libxml2-dev libxml2 libssl-dev
sudo pip install scrapy