ModuleNotFoundError: No module named '_lzma' when building python using pyenv on macos

Question

Trying to use pyenv to build python versions on macOS, but every install I make ends up with

❯ python
>>> import lzma

ModuleNotFoundError: No module named '_lzma'

I have installed all of the following packages:

brew install openssl readline sqlite3 xz zlib
xcode-select --install

Only caveat is that homebrew installs packages to ~/.brew.

Any input is appreciated.

Answer 1

None of the prior answers worked for me. The instructions in this gist worked for me.

In short: You may be missing the xz libraries, in which case you can install them with Homebrew and then reinstall/rebuild the Python version from pyenv:

$  brew install xz
$  pyenv uninstall <desired-python-version>
$  pyenv install <desired-python-version>

Note: I only had this problem with the Python installed by pyenv, but not the Mac system Python or the conda python. It might be best to use the brewed python (brew install python) unless you have a specific need for pyenv (like needing more control over the python version/updating).

Answer 2

Issue 39430?

I don't have an answer. But this symptom sounds like this Python bug ticket:

• Issue 39430_ tarfile.open(mode=r) race condition when importing lzma - Python tracker

Reported in 2020. Unfortunately still unfixed as of 2022.

Workaround for package "csvkit": Try python2

I experienced this issue when trying to use package csvkit on my Windows 10 running MobaXterm:

Package installs just fine but then the /bin/csv* tools don't run:

$ /bin/python3 -m pip install csvkit --quiet
✔

$ csvstat --version
Traceback (most recent call last):
  File "/bin/csvstat", line 5, in <module>
    from csvkit.utilities.csvstat import launch_new_instance
  File "/usr/lib/python3.6/site-packages/csvkit/utilities/csvstat.py", line 12, in <module>
    from csvkit.cli import CSVKitUtility, parse_column_identifiers
  File "/usr/lib/python3.6/site-packages/csvkit/cli.py", line 16, in <module>
    import lzma
  File "/usr/lib/python3.6/lzma.py", line 27, in <module>
    from _lzma import *
ModuleNotFoundError: No module named '_lzma'
✘

My workaround was to use python2. Not super happy with that.

First throw out the non-working python3 version like so:

$ /bin/python3 -m pip uninstall csvkit --quiet --yes
✔

$ hash -r
✔

$ csvstat --version
csvstat: command not found
✘

So now it's gone. Let's try again but with python2:

$ /bin/python2 -m pip list | grep -i csv
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
WARNING: You are using pip version 20.3.4; however, version 21.3.1 is available.
You should consider upgrading via the '/bin/python2 -m pip install --upgrade pip' command.
✘

$ /bin/python2 -m pip install csvkit --quiet
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
  WARNING: The scripts csvclean, csvcut, csvformat, csvgrep, csvjoin, csvjson, csvlook, csvpy, csvsort, csvsql, csvstack, csvstat, in2csv and sql2csv are installed in '/usr/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: You are using pip version 20.3.4; however, version 21.3.1 is available.
You should consider upgrading via the '/bin/python2 -m pip install --upgrade pip' command.
✔

$ csvstat --version
csvstat 1.0.7
✔

Result: csvkit is installed and /bin/csvstat.exe runs without the "_lzma" error. But unfortunately on python2 and not on python3. -- This workaround is kinda good enough for me, because I just wanted the /bin/csv* utilities but maybe useless for people who actually need to run things on python3.

Answer 3

Based on pyenv wiki, you should install the desired python version with --enable-framework flag. It did work for me.

PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.6.7

The reason is that the module you're trying to use is implemented in C, and CPython with shared libs will be built with the aforementioned flag.

Answer 4

Ended up figuring out. This issue only arose when moving the Homebrew directory from its default. This was not needed when Homebrew was installed normally.

I added this to my .zshrc (alternatively, your .bashrc or .bash_profile)

export LDFLAGS="-L/Users/pcosta/.brew/opt/xz/lib $LDFLAGS"
export CPPFLAGS="-I/Users/pcosta/.brew/opt/xz/include $CPPFLAGS"
export PKG_CONFIG_PATH="/Users/pcosta/.brew/opt/xz/lib/pkgconfig:$PKG_CONFIG_PATH"

Homebrew warns that you should do this for other installed packages, but not xz. Presumably, because it is not needed if Homebrew lives where it expects to.