how to install scrapy on ubuntu?

Question

I know that intall the scrapy should install the w3lib first,so I install the w3lib firstly,but when I import the scrapy in python ide,the program is crashed. the error:

creating Twisted.egg-info

writing requirements to Twisted.egg-info\requires.txt

writing Twisted.egg-info\PKG-INFO

writing top-level names to Twisted.egg-info\top_level.txt

writing dependency_links to Twisted.egg-info\dependency_links.txt

writing manifest file 'Twisted.egg-info\SOURCES.txt'

warning: manifest_maker: standard file '-c' not found



reading manifest file 'Twisted.egg-info\SOURCES.txt'

writing manifest file 'Twisted.egg-info\SOURCES.txt'

copying twisted\internet\_sigchld.c -> build\lib.win-amd64-2.7\twisted\internet

creating build\lib.win-amd64-2.7\twisted\internet\iocpreactor\iocpsupport

copying twisted\internet/iocpreactor/iocpsupport\iocpsupport.c -> build\lib.win-amd64-2.7\twisted\internet/iocpreactor/i
ocpsupport

copying twisted\internet/iocpreactor/iocpsupport\winsock_pointers.c -> build\lib.win-amd64-2.7\twisted\internet/iocpreac
tor/iocpsupport

copying twisted\python\_epoll.c -> build\lib.win-amd64-2.7\twisted\python

copying twisted\python\_initgroups.c -> build\lib.win-amd64-2.7\twisted\python

copying twisted\python\sendmsg.c -> build\lib.win-amd64-2.7\twisted\python

copying twisted\runner\portmap.c -> build\lib.win-amd64-2.7\twisted\runner

copying twisted\test\raiser.c -> build\lib.win-amd64-2.7\twisted\test

running build_ext

What's wrong?

Answer 1

This is how I installed scrapy on ubuntu:

sudo apt-get update
sudo apt-get install python-pip build-essential python-dev libxslt-dev libxml2-dev   
sudo -H pip install Scrapy
scrapy version

The important thing that solved my issues was sudo -H pip install Scrapy specifically the -H flag.

I also exited out of the terminal and started a new terminal to ensure the all the environment variables were set correctly

Answer 2

Make sure you had installed the Twisted, pyOpenSSL and pycrypto. These are my steps to install scrapy on ubuntu. 1.install gcc and lxml:

sudo apt-get install python-dev
sudo apt-get install libevent-dev
sudo apt-get install libxml2 libxml2-dev
apt-get install libxml2-dev libxslt-dev
apt-get install python-lxml 

2.install twisted:

sudo apt-get install python-twisted python-libxml2 python-simplejson

sudo apt-get install build-essential libssl-dev libffi-dev python-dev

3.install pyOpenSSL:

wget http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz
tar -zxvf pyOpenSSL-0.13.tar.gz
cd pyOpenSSL-0.13
sudo python setup.py install

4.install pycrypto

wget http://pypi.python.org/packages/source/p/pycrypto/pycrypto-2.5.tar.gz
tar -zxvf pycrypto-2.5.tar.gz
cd pycrypto-2.5
sudo python setup.py install

5.install easy_install:(if you don't have easy_install)

wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

6.install w3lib

sudo easy_install -U w3lib

7.install scrapy

sudo easy_install Scrapy

If you wanna know much,please goto my blog.