ModuleNotFoundError: No module named 'Cython' when installing pyarrow

Question

I am trying to install packages from requirements.txt in a virtualenv. However, when pip comes to installing pyarrow, it fails with the following:

Collecting pyarrow==0.9.0 (from -r requirements.txt (line 41))
  Using cached https://files.pythonhosted.org/packages/be/2d/11751c477e4e7f4bb07ac7584aafabe0d0608c170e4bff67246d695ebdbe/pyarrow-0.9.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-dgdvn2c1/pyarrow/setup.py", line 29, in <module>
        from Cython.Distutils import build_ext as _build_ext
    ModuleNotFoundError: No module named 'Cython'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-dgdvn2c1/pyarrow/  

I am using Python 3.6.3 and Pip 18.0

What I tried:

Most of the SO answers ask to check if python and pip refer to the venv python and I am sure it does.

$ which python
/Users/ME/repos/myproject/venv/bin/python
(venv) 

I also tried installing cython via pip and even then installation fails albeit with a different error.

How do I solve this?

Edit: Stacktrace after installing Cython:

Collecting pyarrow==0.9.0 (from -r requirements.txt (line 41))
  Using cached https://files.pythonhosted.org/packages/be/2d/11751c477e4e7f4bb07ac7584aafabe0d0608c170e4bff67246d695ebdbe/pyarrow-0.9.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/setup.py", line 508, in <module>
        url="https://arrow.apache.org/"
      File "/Users/ME/Work/myproject/venv/lib/python3.6/site-packages/setuptools/__init__.py", line 131, in setup
        return distutils.core.setup(**attrs)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/Users/ME/Work/myproject/venv/lib/python3.6/site-packages/setuptools/dist.py", line 370, in __init__
        k: v for k, v in attrs.items()
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py", line 281, in __init__
        self.finalize_options()
      File "/Users/ME/Work/myproject/venv/lib/python3.6/site-packages/setuptools/dist.py", line 529, in finalize_options
        ep.load()(self, ep.name, value)
      File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/integration.py", line 23, in version_keyword
        dist.metadata.version = get_version(**value)
      File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 135, in get_version
        parsed_version = _do_parse(config)
      File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 77, in _do_parse
        parse_result = _call_entrypoint_fn(config, config.parse)
      File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 40, in _call_entrypoint_fn
        return fn(config.absolute_root)
      File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/setup.py", line 462, in parse_version
        return version_from_scm(root)
      File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 28, in version_from_scm
        return _version_from_entrypoint(root, "setuptools_scm.parse_scm")
      File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 44, in _version_from_entrypoint
        for ep in iter_matching_entrypoints(config.absolute_root, entrypoint):
    AttributeError: 'str' object has no attribute 'absolute_root'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/

Answer 1

This is a known issue (#2163) with pyarrow that has been closed without a fix.

It's worth noting that there is no pyarrow 0.9.0 wheel available for MacOS in pypi. It was made available in a subsequent 0.9.0.post1 release.

The following should work with Python 3.6 on MacOS:

pip install pyarrow==0.9.0.post1