Error importing pyarrow in jupyter notebook after pip installation of pyarrow

Question

I am having an issue importing pyarrow into my jupyter notebook. To give some context, I created a virtual environment named rc_env. I've uninstalled and reinstalled pyarrow quite a few times but this the message I received when I recently install it.

pip install pyarrow
Requirement already satisfied: pyarrow in ./Environments/rc_env/lib/python3.8/site-packages (4.0.1)
Requirement already satisfied: numpy>=1.16.6 in ./Environments/rc_env/lib/python3.8/site-packages (from pyarrow) (1.21.0)

So the package is in my environment. And for another sanity check, I used the pip list command to see the python packages installed in my environment.

(rc_env) LTA00015JFK:~ adenner$ pip list
Package                       Version
----------------------------- ---------
appnope                       0.1.2
argon2-cffi                   20.1.0
async-generator               1.10
attrs                         21.2.0
backcall                      0.2.0
beautifulsoup4                4.9.3
bleach                        3.3.0
boto3                         1.17.103
botocore                      1.20.103
cachetools                    4.2.2
certifi                       2021.5.30
cffi                          1.14.5
chardet                       4.0.0
cycler                        0.10.0
debugpy                       1.3.0
decorator                     5.0.9
defusedxml                    0.7.1
dictor                        0.1.7
entrypoints                   0.3
et-xmlfile                    1.1.0
google                        3.0.0
google-api-core               1.30.0
google-api-python-client      2.11.0
google-auth                   1.32.1
google-auth-httplib2          0.1.0
google-auth-oauthlib          0.4.4
google-cloud                  0.34.0
google-cloud-bigquery         2.20.0
google-cloud-bigquery-storage 2.5.0
google-cloud-core             1.7.1
google-cloud-vision           2.3.2
google-crc32c                 1.1.2
google-resumable-media        1.3.1
googleapis-common-protos      1.53.0
grpcio                        1.38.1
httplib2                      0.19.1
idna                          2.10
ipykernel                     6.0.0
ipython                       7.25.0
ipython-genutils              0.2.0
ipywidgets                    7.6.3
jedi                          0.18.0
Jinja2                        3.0.1
jmespath                      0.10.0
jsonschema                    3.2.0
jupyter                       1.0.0
jupyter-client                6.1.12
jupyter-console               6.4.0
jupyter-core                  4.7.1
jupyterlab-pygments           0.1.2
jupyterlab-widgets            1.0.0
kiwisolver                    1.3.1
libcst                        0.3.19
MarkupSafe                    2.0.1
matplotlib                    3.4.2
matplotlib-inline             0.1.2
mistune                       0.8.4
mypy-extensions               0.4.3
nbclient                      0.5.3
nbconvert                     6.1.0
nbformat                      5.1.3
nest-asyncio                  1.5.1
notebook                      6.4.0
numpy                         1.21.0
oauthlib                      3.1.1
openpyxl                      3.0.7
packaging                     20.9
pandas                        1.2.5
pandas-gbq                    0.15.0
pandocfilters                 1.4.3
parso                         0.8.2
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        8.3.0
pip                           21.1.3
prometheus-client             0.11.0
prompt-toolkit                3.0.19
proto-plus                    1.19.0
protobuf                      3.17.3
ptyprocess                    0.7.0
pyarrow                       4.0.1
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.20
pydata-google-auth            1.2.0
Pygments                      2.9.0
pyparsing                     2.4.7
pyrsistent                    0.18.0
python-dateutil               2.8.1
pytz                          2021.1
PyYAML                        5.4.1
pyzmq                         22.1.0
qtconsole                     5.1.1
QtPy                          1.9.0
requests                      2.25.1
requests-oauthlib             1.3.0
rsa                           4.7.2
s3transfer                    0.4.2
scipy                         1.7.0
seaborn                       0.11.1
Send2Trash                    1.7.1
seshat                        0.8.5
setuptools                    57.0.0
six                           1.16.0
soupsieve                     2.2.1
terminado                     0.10.1
testpath                      0.5.0
titlecase                     2.2.0
tornado                       6.1
tqdm                          4.61.1
traitlets                     5.0.5
typing-extensions             3.10.0.0
typing-inspect                0.7.1
uritemplate                   3.0.1
urllib3                       1.26.6
uuid                          1.30
wcwidth                       0.2.5
webencodings                  0.5.1
wheel                         0.36.2
widgetsnbextension            3.5.1
XlsxWriter                    1.4.3

Now when I open up python and try to import the module, I receive the following error message:

(rc_env) LTA00015JFK:~ adenner$ python 
Python 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/adenner/Environments/rc_env/lib/python3.8/site-packages/pyarrow/__init__.py", line 63, in <module>
    import pyarrow.lib as _lib
ImportError: dlopen(/Users/adenner/Environments/rc_env/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so, 2): Symbol not found: ____chkstk_darwin
  Referenced from: /Users/adenner/Environments/rc_env/lib/python3.8/site-packages/pyarrow/libarrow.400.dylib
  Expected in: /usr/lib/libSystem.B.dylib
 in /Users/adenner/Environments/rc_env/lib/python3.8/site-packages/pyarrow/libarrow.400.dylib

I have tried using both anaconda3 and miniconda3 environments but unfortunately, I keep receiving the same error message. I am also using macOS High Serra version 10.13.6 Any suggestions?

Answer 1

This is a known bug with the latest pyarrow release: https://issues.apache.org/jira/browse/ARROW-13108. you will need macOS 10.15+ to run the wheel builds.

As you are already in an environment created by conda, you could instead use the pyarrow conda package. This will work on macOS 10.9+ and is even the preferred way to install pyarrow:

conda install -c conda-forge pyarrow

Alternatively using mamba:

mamba install -c conda-forge pyarrow

Answer 2

you can try the jupyter lab instead of using jupyter notebook. I got this problem when i use notebook, but pyarrow works well in jupyter lab