How Can I install Twisted + Scrapy on Python3.6 and CentOs

Question

I use the newest Python on Centos 7, and a dedicated virtualenv

(ENV) [luoc@study ~ ]$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.3.1611 (Core) 
Release:    7.3.1611
Codename:   Core

(ENV) [luoc@study ~ ]$ python --version
Python 3.6.0

When I install scrapy, the error

(ENV) [luoc@study ~ ]$ pip install scrapy
Collecting scrapy
  Using cached Scrapy-1.3.2-py2.py3-none-any.whl
Collecting cssselect>=0.9 (from scrapy)
  Using cached cssselect-1.0.1-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in ./ENV/lib/python3.6/site-packages (from scrapy)
Collecting Twisted>=13.1.0 (from scrapy)
  Could not find a version that satisfies the requirement Twisted>=13.1.0 (from scrapy) (from versions: )
No matching distribution found for Twisted>=13.1.0 (from scrapy)

and when I install twisted independence, the error

(ENV) [luoc@study ~ ]$ pip install Twisted
Collecting Twisted
  Could not find a version that satisfies the requirement Twisted (from versions: )
No matching distribution found for Twisted

(ENV) [luoc@study ~ ]$ pip install --verbose Twisted
Collecting Twisted
  1 location(s) to search for versions of Twisted:
  * https://pypi.python.org/simple/twisted/
  Getting page https://pypi.python.org/simple/twisted/
  Looking up "https://pypi.python.org/simple/twisted/" in the cache
  Current age based on date: 40208
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  The cached response is "stale" with no etag, purging
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/twisted/ HTTP/1.1" 200 10196
  Updating cache with response from "https://pypi.python.org/simple/twisted/"
  Caching b/c date exists and max-age > 0
  Analyzing links from page https://pypi.python.org/simple/twisted/
Skipping link ...
Could not find a version that satisfies the requirement Twisted (from versions: )
Cleaning up...
No matching distribution found for Twisted
Exception information:
Traceback (most recent call last):
  File "/home/luoc/ENV/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/luoc/ENV/lib/python3.6/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/home/luoc/ENV/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/home/luoc/ENV/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/home/luoc/ENV/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/home/luoc/ENV/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/luoc/ENV/lib/python3.6/site-packages/pip/index.py", line 514, in find_requirement
    'No matching distribution found for %s' % req
pip.exceptions.DistributionNotFound: No matching distribution found for Twisted

So why can't install Twisted on Python3.6? Does something wrong on my environment？

Kmike suggested me to ask twisted's developer

Answer 1

I know problem is arleady solved but I want to help future users.
I'm on Debian Strech and just installed Python 3.6,
If you want to install scrapy compiling Python 3.6 yourself, try run

sudo apt-get install bzip2 libbz2-dev

After that, just recompile Python 3.6 and it should works

Answer 2

Did you compile Python 3.6 by yourself?

It seems your python version was compiled without bzip2 support. Here is a past ticket for the same issue: https://twistedmatrix.com/trac/ticket/8177

I'd suggest to use Miniconda to have a Python 3.6 environment and follow this instructions to install Scrapy.