Why am I getting this error in scrapy - python3.7 invalid syntax

Question

I've had a heck of a time installing scrapy. I have it installed on my mac but I am getting this error when running the tutorial:

Virtualenvs/scrapy_env/lib/python3.7/site-packages/twisted/conch/manhole.py", line 154
    def write(self, data, async=False):
                              ^
SyntaxError: invalid syntax

I'm on the latest versions of everything as far as I can tell. Getting this up and running has been a pain. sheesh. OS High Sierra 10.13.3 python 3.7 installed ipython I've updated about everything I can think of. terminal line is:

scrapy shell http://quotes.toscrape.com/random

or scrapy shell "http://quotes.toscrape.com/random" even tried single quotes. Any help would be great!

Here is the complete log:

(scrapy_env) XXX-MacBook-Pro:Virtualenvs ComputerName$ scrapy shell http://quotes.toscrape.com/random
2018-02-19 06:38:53 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-02-19 06:38:53 [scrapy.utils.log] INFO: Versions: lxml 3.6.2.0, libxml2 2.9.4, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 3.7.0a2 (v3.7.0a2:f7ac4fe52a, Oct 16 2017, 21:11:18) - [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)], pyOpenSSL 17.5.0 (OpenSSL 1.0.2n  7 Dec 2017), cryptography 2.1.4, Platform Darwin-17.4.0-x86_64-i386-64bit
2018-02-19 06:38:53 [scrapy.crawler] INFO: Overridden settings: {'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'LOGSTATS_INTERVAL': 0}
Traceback (most recent call last):
  File "/Users/ComputerName/Virtualenvs/scrapy_env/bin/scrapy", line 11, in <module>
    load_entry_point('Scrapy==1.5.0', 'console_scripts', 'scrapy')()
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/cmdline.py", line 150, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/cmdline.py", line 90, in _run_print_help
    func(*a, **kw)
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/cmdline.py", line 157, in _run_command
    cmd.run(args, opts)
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/commands/shell.py", line 65, in run
    crawler = self.crawler_process._create_crawler(spidercls)
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/crawler.py", line 203, in _create_crawler
    return Crawler(spidercls, self.settings)
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/crawler.py", line 55, in __init__
    self.extensions = ExtensionManager.from_crawler(self)
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/middleware.py", line 58, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/utils/misc.py", line 44, in load_object
    mod = import_module(module)
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 680, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/Scrapy-1.5.0-py3.7.egg/scrapy/extensions/telnet.py", line 12, in <module>
    from twisted.conch import manhole, telnet
  File "/Users/ComputerName/Virtualenvs/scrapy_env/lib/python3.7/site-packages/twisted/conch/manhole.py", line 154
    def write(self, data, async=False):
                              ^
SyntaxError: invalid syntax
(scrapy_env) XXX-MacBook-Pro:Virtualenvs ComputerName$

Answer 1

you can do

pip install git+https://github.com/twisted/twisted.git@trunk

because the 'trunk' branch had already merged the fix '9384-remove-async-param'

you can see in https://github.com/twisted/twisted/pull/966

Answer 2

Turns out that async and await are keywords in python 3.7. This is why syntax error. This is from user stranac. Installed new virtualenv targeting 2.7 and had no issues with installing scrapy or running bench/shell.