scrapy requires Python 2.7 but already it is

Question

I installed Python 2.7 and then tried to install scrapy via pip. It didn't work, so I deleted scrapy and tried to install again via pip. When I do I get the following errors:

src/lxml/lxml.etree.c:192359: error: âxsltDocDefaultLoaderâ undeclared (first use in this function)

src/lxml/lxml.etree.c:192368: error: âxsltDocLoaderFuncâ undeclared (first use in this function)

src/lxml/lxml.etree.c:192368: error: expected â)â before â__pyx_f_4lxml_5etree__xslt_doc_loaderâ

error: command 'gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools;__file__='/tmp/pip_build_root/lxml/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-fiN0SO-record/install-record.txt --single-version-externally-managed failed with error code 1 in /tmp/pip_build_root/lxml
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.4.1', 'console_scripts', 'pip')()
  File "/usr/lib/python2.6/site-packages/pip-1.4.1-py2.6.egg/pip/__init__.py", line 148, in main
    return command.main(args[1:], options)
  File "/usr/lib/python2.6/site-packages/pip-1.4.1-py2.6.egg/pip/basecommand.py", line 169, in main
    text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 81: ordinal not in range(128)

I then try to run scrapy, but it complains that it requires Python 2.7. As you can see from the screenshot below, I am running version 2.7. Any ideas what is wrong?

Answer 1

The traceback shows that you are using pip from python 2.6.

How have you installed python 2.7? Depending on your system, you might have available a pip2.7 command. Otherwise you could try installing pip for your python 2.7:

$ wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
$ python get-pip.py

You might need to use sudo for the last command. Once you have pip installed, it's highly recommended to use virtualenv (pip install virtualenv) to install the new dependencies and not mess up with your system installation.

Alternatively, if you install the latest pip, you can use the option --user to install packages locally without requiring sudo: pip install --user scrapy.

Also, it seems you have problems installing lxml. What distribution of linux are you using? Have you installed the libraries libxml2-dev and libxslt-dev?

Verify that you have the prerequisites for scrapy and lxml and refer to your package manager in case of any missing library:

• http://doc.scrapy.org/en/latest/intro/install.html
• http://lxml.de/installation.html