airflow initdb: undefined symbol: Py_GetArgcArgv

Question

I followed the documentation to install Apache-airflow. https://airflow.apache.org/docs/stable/start.html

When I execute airflow initdb, an error occurs every time.

x-MacBook-Pro:~ x$ airflow initdb
 ......
import airflow.utils.dag_processing
  File "/Library/Python/3.7/site-packages/airflow/utils/dag_processing.py", line 40, in <module>
    from setproctitle import setproctitle
ImportError: dlopen(/Library/Python/3.7/site-packages/setproctitle.cpython-37m-darwin.so, 2): Symbol not found: _Py_GetArgcArgv
  Referenced from: /Library/Python/3.7/site-packages/setproctitle.cpython-37m-darwin.so
  Expected in: flat namespace
 in /Library/Python/3.7/site-packages/setproctitle.cpython-37m-darwin.so

One answer suggested that this is a problem with the binary package. But I still don't know how to solve that. This is the link https://github.com/psycopg/psycopg2/issues/807.

MacOSX 10.15.3

pip 20.0.2

Python 3.7.3

Answer 1

Of course I had the same issue. macOS 10.15.2.

I had python 3.8 globally installed. I did some research and I found out that the reason was the way that c library m-darwin.so was compiled.

This steps help me fixed the issue:

1. Installed pyenv

brew install pyenv

2. Downgraded to python 3.7.0

pyenv install 3.7.0

3. Set python 3.7.0 globally

pyenv global 3.7.0

4. Running python3 -V should give you:

Python 3.7.0

5. Recreated my project environment using virtualenv:

python3 -m virtualenv --python=python3.7 my awesome_env

6. Activated my env and installed all dependencies and it woorked.

source awesome_env/bin/activate
pip install -r requirements.txt

I think this whole process trigger the recompilation of the libraries.

Answer 2

I had the same problem, when I was using the system (OS) python3 interpreter, i.e. /usr/bin/python3

you can simply install python 3.7 or 3.8 (it works for both) with homebrew:

brew install [email protected]

Make sure python3 is now pointing to /usr/local/bin/python3 by running which python3.

Then install apache-airflow:

python3 -m venv .venv
pip install apache-airflow
airflow initdb

Answer 3

This error is explained more fully in the troubleshooting section here: https://airflow.apache.org/docs/apache-airflow/stable/installation.html

In my case, I had the problem with 3.8.2 installed via homebrew and installing 3.9.4 via pyenv did the trick.