Installing Cryptography on an Apple Silicon M1 Mac

Question

Help! I'm trying to install cryptography on my m1. I know I can run terminal in rosetta mode, but I'm wondering if there is a way not to do that.

Output:

    ERROR: Command errored out with exit status 1:
     command: /opt/homebrew/opt/[email protected]/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/tmpl4sga84k
         cwd: /private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-install-jko4b562/cryptography_7b1bbc9ece2f481a8e8e9ea03b1a0030
    Complete output (55 lines):
    
        =============================DEBUG ASSISTANCE=============================
        If you are seeing a compilation error please try the following steps to
        successfully install cryptography:
        1) Upgrade to the latest pip and try again. This will fix errors for most
           users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
        2) Read https://cryptography.io/en/latest/installation.html for specific
           instructions for your platform.
        3) Check our frequently asked questions for more information:
           https://cryptography.io/en/latest/faq.html
        =============================DEBUG ASSISTANCE=============================
    
    Traceback (most recent call last):
      File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
        main()
      File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 145, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 44, in <module>
        setup(
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/opt/homebrew/Cellar/[email protected]/3.9.1_7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 432, in __init__
        _Distribution.__init__(self, {
      File "/opt/homebrew/Cellar/[email protected]/3.9.1_7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 292, in __init__
        self.finalize_options()
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 708, in finalize_options
        ep(self)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 715, in _finalize_setup_keywords
        ep.load()(self, ep.name, value)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
        add_cffi_module(dist, cffi_module)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
        execfile(build_file_name, mod_vars)
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 25, in execfile
        exec(code, glob, glob)
      File "src/_cffi_src/build_openssl.py", line 77, in <module>
        ffi = build_ffi_for_binding(
      File "src/_cffi_src/utils.py", line 54, in build_ffi_for_binding
        ffi = build_ffi(
      File "src/_cffi_src/utils.py", line 74, in build_ffi
        ffi = FFI()
      File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/cffi/api.py", line 48, in __init__
        import _cffi_backend as backend
    ImportError: dlopen(/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
      Referenced from: /private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
      Expected in: flat namespace
     in /private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-9bqzge_f/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so

I've tried to build and run like their instructions say in that code block to the same error. I've looked around and nobody has seemingly found the fix yet, but those things are two months old usually. What am I missing?

Answer 1

This issue is due to a mismatch between the libffi header version and the version of libffi the dynamic linker finds. In general it appears users encountering this problem have homebrew libffi installed and have a Python built against that in some fashion.

When this happens cffi (a cryptography dependency) compiles, but fails at runtime raising this error. This should be fixable by passing the right path as a linker argument. To reinstall cffi you should pip uninstall cffi followed by

LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all:

This is an ugly workaround, but will get you past this hurdle for now.

Update: I've uploaded arm64 wheels for macOS so the below compilation is no longer required if your pip is up-to-date. However, if, for some reason you wish to compile it yourself:

LDFLAGS="-L$(brew --prefix [email protected])/lib" CFLAGS="-I$(brew --prefix [email protected])/include" pip install cryptography

Answer 2

I wasn't able to previously install cffi, until I discovered an unrelated issue. I was at this for about two days, until I found this command:

python3 -m ensurepip --upgrade

Magically, everything started working for me. It came from an issue between Python and Pip coming from different sources.

Answer stolen from this question: using pip3: module "importlib._bootstrap" has no attribute "SourceFileLoader"

Edit: This may be a courtesy of the above poster, so could be unrelated. If so, thank you anonymous human!

Answer 3

Probably, you'll have a problem with more packages and each has it's own solution for Apple Silicon, it's exhausting.

I came to final solution: using x86_x64 Homebrew which installs x86 packages, including Python. Thus, all your requirements are installing as on the x86_x64 macs and there are no more problems with the compilation errors and so on.

Instructions:

1. Run iTerm2 (or default Terminal app) under Rosetta 2 (right click on the app icon -> Get info -> Open using rosetta).
2. Install homebrew as usual /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" or you can get this link from https://brew.sh/ for security reasons (never copy curl commands from stackoverflow without double-checking).
3. Add an alias in your ~/.zshrc (if you're using ZSH) or ~/.bash_profile (if you're bash user): alias brew='arch -x86_64 /usr/local/bin/brew'.
4. Turn off Open using rosetta in iTerm2 Get info.

Now, every time you'll print brew in terminal apps you'll run x86_x64 Homebrew. And when you install any package from brew, it'll work under Rosetta 2 automatically.

Answer 4

I'm using Macbook Pro M1 2020 model and faced the same issue. The issue was only with my cffi and pip versions maybe. Because these 4 steps helped me -

1. Uninstalling old cffi pip uninstall cffi
2. Upgrading pip python -m pip install --upgrade pip
3. Reinstalling cffi pip install cffi
4. Intalling cryptography pip install cryptography

Answer 5

A little late to the party, but the solutions above didn't work for me. Paul got me on the right track, but my problem was that pyenv used the mac libffi for its build and cffi used the homebrew version. I read this somewhere, can't claim this unique insight.

My solution was to ensure that my python (3.8.13) was built by pyenv using the homebrew libffi by ensuring correct headers libraries and package config:

export LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib -L$(brew --prefix [email protected])/lib -L$(brew --prefix libffi)/lib"
export CPPFLAGS="-I$(brew --prefix zlib)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix [email protected])/include -I$(brew --prefix libffi)/include"
export PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig:$(brew --prefix libffi)/lib/pkgconfig"

rebuilding python...

pyenv uninstall 3.8.13
pyenv install 3.8.13

killing the pip cache

pip cache purge

and, finally, reinstalling my dependencies using pipenv

pipenv --rm
pipenv sync --dev

After these steps, I was free from the dreaded

ImportError: dlopen(/private/var/folders/k7/z3mq67_532bdr_rcm2grml240000gn/T/pip-build-env-apk5b25z/overlay/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_ffi_prep_closure'

Answer 6

This answer here worked like a charm! @paveldroo

As an extension to the answer above, I went ahead and saved the alias in step 3 as alias ibrew='arch -x86_64 /usr/local/bin/brew' at ~/.zshrc

This means when I install anything with brew command, it gets installed for M1 architecture, and when I install with ibrew command it gets installed for -x86_64 architecture.

As a consequence, I installed two instances of python3 at my system one at /opt/homebrew/bin/python3 using brew and the other at /usr/local/bin/python3 using ibrew

The two versions adds some flexibility on creating the project virtual environments as needed. For example you could create virtual environments using:

1. /usr/local/bin/python3 -m venv venv for -x86_64 architecture
2. /opt/homebrew/bin/python3 -m venv venv for M1 architecture