Running scrapy with PyCharm - Debug works but Run does not work

Question

I met a very strange issue, running Scrapy with PyCharm:
With the exact same configuration, Debug works, but Run with PyCharm does NOT work.

• Windows 10
• PyCharm 2016.3.3
• Scrapy 1.3.3
• Python 3.6.0

Configuration: PyCharm Edit Configuration Page

When I attempt to debug scrapy, it works perfectly. When I attempt to run scrapy, I get below error:

C:\Users\baib2\AppData\Local\Programs\Python\Python36\python.exe 
C:/Users/baib2/AppData/Local/Programs/Python/Python36/Lib/site-packages/scrapy/cmdline.py crawl scenelist_spider
Traceback (most recent call last):
  File "C:/Users/baib2/AppData/Local/Programs/Python/Python36/Lib/site-packages/scrapy/cmdline.py", line 8, in <module   
    import scrapy
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\__init__.py", line 27, in <module   
    from . import _monkeypatches
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\_monkeypatches.py", line 20, in <module   
    import twisted.persisted.styles  # NOQA
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\twisted\persisted\styles.py", line 21, in <module   
    from twisted.python.compat import _PY3, _PYPY
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\twisted\python\__init__.py", line 11, in <module   
    from .compat import unicode
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\twisted\python\compat.py", line 612, in <module   
    from http import cookiejar as cookielib
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\Lib\site-packages\scrapy\http\__init__.py", line 8, in <module   
    from scrapy.http.headers import Headers
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\http\__init__.py", line 10, in <module   
    from scrapy.http.request import Request
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\http\request\__init__.py", line 8, in <module   
    from w3lib.url import safe_url_string
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\w3lib\url.py", line 17, in <module   
    from six.moves.urllib.request import pathname2url, url2pathname
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 92, in __get__
    result = self._resolve()
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 160, in _resolve
    module = _import_module(self.mod)
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 82, in _import_module
    __import__(name)
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 88, in <module   
    import http.client
ModuleNotFoundError: No module named 'http.client'

Process finished with exit code 1

No need to attach any code because even running the command:

python3.exe C:\Users\baib2\AppData\Local\Programs\Python\Python36\Lib\site-packages\scrapy\cmdline.py

will produce the same error.

I've checked my sys.path, comparing with run, debug has 1 more path:

'C:\Program Files (x86)\JetBrains\PyCharm 2016.3.3\helpers\pydev'

And I really don't think this shall make any difference.

Hope someone can take a look, thanks!

Answer 1

Menu Run>Edit configurations enable checkbox "Show Command line afterwards" This helps for me

Answer 2

In case you are using a new version of PyCharm where Show Command line afterward is not available anymore, Run with Python console solves the issue.