running Scrapy but it error: No module named _util

Question

I have installed Scrapy，and import it in python, every things look fine. But it cause error when i try a example in http://scrapy-chs.readthedocs.io/zh_CN/0.24/intro/tutorial.html.

I run scrapy crawl swspider, then i get:

> 2018-05-14 14:24:16 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: tutorial)
> 2018-05-14 14:24:16 [scrapy.utils.log] INFO: Versions: lxml 3.2.1.0,
> libxml2 2.9.1, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted
> 18.4.0, Python 2.7.5 (default, Nov 20 2015, 02:00:19) - [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)], pyOpenSSL 0.13.1 (OpenSSL 1.0.1e-fips 11
> Feb 2013), cryptography 0.8.2, Platform
> Linux-3.10.0-327.el7.x86_64-x86_64-with-centos-7.2.1511-Core
> 2018-05-14 14:24:16 [scrapy.crawler] INFO: Overridden settings:
> {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES':
> ['tutorial.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'tutorial'}
> Traceback (most recent call last):   File
> "/disk1/wulixin/install/bin/scrapy", line 11, in <module>
>     sys.exit(execute())   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/cmdline.py",
> line 150, in execute
>     _run_print_help(parser, _run_command, cmd, args, opts)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/cmdline.py",
> line 90, in _run_print_help
>     func(*a, **kw)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/cmdline.py",
> line 157, in _run_command
>     cmd.run(args, opts)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/commands/crawl.py",
> line 57, in run
>     self.crawler_process.crawl(spname, **opts.spargs)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/crawler.py",
> line 170, in crawl
>     crawler = self.create_crawler(crawler_or_spidercls)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/crawler.py",
> line 198, in create_crawler
>     return self._create_crawler(crawler_or_spidercls)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/crawler.py",
> line 203, in _create_crawler
>     return Crawler(spidercls, self.settings)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/crawler.py",
> line 55, in __init__
>     self.extensions = ExtensionManager.from_crawler(self)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/middleware.py",
> line 58, in from_crawler
>     return cls.from_settings(crawler.settings, crawler)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/middleware.py",
> line 34, in from_settings
>     mwcls = load_object(clspath)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/utils/misc.py",
> line 44, in load_object
>     mod = import_module(module)   File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in
> import_module
>     __import__(name)   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/extensions/memusage.py",
> line 16, in <module>
>     from scrapy.mail import MailSender   File "/disk1/wulixin/install/lib/python2.7/site-packages/scrapy/mail.py",
> line 25, in <module>
>     from twisted.internet import defer, reactor, ssl   File "/disk1/wulixin/install/lib64/python2.7/site-packages/twisted/internet/ssl.py",
> line 230, in <module>
>     from twisted.internet._sslverify import (   File "/disk1/wulixin/install/lib64/python2.7/site-packages/twisted/internet/_sslverify.py",
> line 15, in <module>
>     from OpenSSL._util import lib as pyOpenSSLlib ImportError: No module named _util

Answer 1

If you are using the newest version of Twisted. You can try downgrade Twisted:

pip install Twisted==16.4.1

Answer 2

you need upgrade pyopenssl

sudo pip install pyopenssl --user --upgrade