Msys2 with python 3.8: ImportError: cannot import name 'open_code' from 'io'

Question

NOTE: There have been several EDITs to the question, as per comments. They are indicated below, and separated by lines. As of now, the only remaining issue seems to be that numpy cannot load, possibly (but not certainly) due to two alternative python 3.8 systems present.

I have updated my msys2 system a couple of months ago. That apparently included an update from python 3.7 to 3.8, but which left me with two broken pythons: I can start python when it is 3.7, but there are no associated packages, and I cannot start python when it is 3.8, which is the version holding packages. I do not know what went wrong with that, or what did I do wrong. I just noticed this now with the first time I mean to use python again after the upgrade.

I will describe here a sequence of steps I followed and what I found. I will post supporting code below for clarity.

1. I can start python, but pandas (e.g.) and many other packages are not found in python. Checking further, /mingw64/lib/python3.7/site-packages is essentially empty (surely emptied when upgrading to 3.8).

2. Looking for the pandas package, I found I have one version installed.

3. The pandas version is for python 3.8, surely upgraded from 3.7.

4. I redirected PYTHONPATH from 3.7 to 3.8

5. Now I cannot even start python. EDIT Now I can start python, with some misconfiguration issues (i.e., partially fixed).

Now the question is

How can I fix python3.8, which gives the error below?

ImportError: cannot import name 'open_code' from 'io' (unknown location)

How can I fix python3.8, which gives the problems below?

New problems:

5.1. I should have python pointing to 3.8, and also fix packages.

5.2. Some modules are not found, some other are.

Note: I don't know if Msys2 upgrade breaks python2-pyqt5 has anything to do with this.

Related:

https://github.com/tox-dev/tox/issues/1334

https://github.com/yan12125/python3-android/issues/19

https://python-forum.io/Thread-Fatal-Python-error-init-sys-streams-can-t-initialize-sys-standard-streams-Attribute

---

TL;DR: Supporting code

1. pandas not found

   $ python
   Python 3.7.4 (default, Jul 11 2019, 10:29:54)
   [GCC 9.1.0] on msys
   Type "help", "copyright", "credits" or "license" for more information.
   Reading /home/user1/.pythonrc
   readline is in /usr/lib/python3.7/lib-dynload/readline.cpython-37m.dll
   >>> import pandas
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   ModuleNotFoundError: No module named 'pandas'
   >>>
   

2. pandas is actually installed

   $ pacman -Sl | grep python | grep installed
   mingw64 mingw-w64-x86_64-python 3.8.2-1 [installed: 3.8.1-1]
   mingw64 mingw-w64-x86_64-python-apipkg 1.5-1 [installed]
   ...
   mingw64 mingw-w64-x86_64-python-pandas 1.0.3-1 [installed: 1.0.1-1]
   ...
   mingw64 mingw-w64-x86_64-python2-setuptools 44.1.0-1 [installed: 42.0.2-1]
   msys python 3.7.4-1 [installed]
   msys python2 2.7.17-1 [installed]
   

3. My pandas version is for python 3.8

   $ pacman -Ql mingw-w64-x86_64-python-pandas | head -5
   mingw-w64-x86_64-python-pandas /mingw64/
   mingw-w64-x86_64-python-pandas /mingw64/lib/
   mingw-w64-x86_64-python-pandas /mingw64/lib/python3.8/
   mingw-w64-x86_64-python-pandas /mingw64/lib/python3.8/site-packages/
   mingw-w64-x86_64-python-pandas /mingw64/lib/python3.8/site-packages/pandas-1.0.1-py3.8.egg-info/
   

4. I redirected PYTHONPATH from 3.7 to 3.8

   Changed

   export PYVERSION="3.7"
   export PYTHONDIR2="${MINGW_HOME}/lib/python${PYVERSION}"
   export PYTHONPATH="${PYTHONDIR2}:${PYTHONDIR2}/site-packages"
   

   to

   export PYVERSION="3.8"
   ...
   

5. Now I cannot even start python. EDIT: Old problem:

   $ python
   Fatal Python error: init_sys_streams: can't initialize sys standard streams
   Traceback (most recent call last):
     File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/io.py", line 54, in <module>
   ImportError: cannot import name 'open_code' from 'io' (unknown location)
   Aborted (core dumped)
   

New problems:

$ python --version
Python 3.7.4
$ type python
python is hashed (/usr/bin/python)
$ ls /usr/bin/python
/usr/bin/python
$ python3.8
Python 3.8.2 (default, Apr  9 2020, 13:17:39)  [GCC 9.3.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Reading C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/home/user1/.pythonrc
Module readline not available.
Traceback (most recent call last):
  File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/home/user1/.pythonrc", line 42, in <module>
    del os, atexit, readline, rlcompleter, save_history, historyPath
NameError: name 'readline' is not defined
>>> import readline
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
>>> import zipfile
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pandas/__init__.py", line 16, in <module>
    raise ImportError(
ImportError: Unable to import required dependencies:
numpy: DLL load failed while importing _ctypes: No se puede encontrar el módulo especificado.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/__init__.py", line 106, in <module>
    from . import _dtype_ctypes
  File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/_dtype_ctypes.py", line 25, in <module>
    import _ctypes
ImportError: DLL load failed while importing _ctypes: No se puede encontrar el módulo especificado.
>>> exit()
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'readline'

---

EDIT #2: Adding info as requested. I just noticed I don't have pip. That matches the fact that I never installed any package with pip...

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/c/Users/user1/OneDrive/Documents/soft-hard-ware/linux-ubuntu:/c/Users/user1/OneDrive/Documents/soft-hard-ware/linux-ubuntu/rsync:/c/Users/user1/Documents/appls_mydocs/science-math-visualization/gp524-win64-mingw_3/gnuplot/bin:/mingw64/bin
$ echo $PYTHONPATH
/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8:/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages
$ echo $PYTHONSTARTUP
/home/user1/.pythonrc
$ which python3.8
/mingw64/bin/python3.8
$ python3.8 -m pip freeze
C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/bin/python3.8.exe: No module named pip
$ python3.8 -c "import sys; print(sys.builtin_module_names)"
('_abc', '_ast', '_codecs', '_collections', '_functools', '_imp', '_io', '_locale', '_operator', '_signal', '_sre', '_stat', '_string', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', '_winapi', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'msvcrt', 'nt', 'sys', 'time', 'winreg', 'xxsubtype')

EDIT #3: Posting as requested, plus additional info.

$ cat .pythonrc
import os
print( "Reading " + os.path.realpath(__file__) )

# enable syntax completion
try:
    import readline
    print( "readline is in " + readline.__file__ )
except ImportError:
    print("Module readline not available.")
else:
    import rlcompleter
    readline.parse_and_bind("tab: complete")

# From https://docs.python.org/2/tutorial/interactive.html

# Add auto-completion and a stored history file of commands to your Python
# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
# bound to the Esc key by default (you can change it - see readline docs).
#
# Store the file in ~/.pystartup, and set an environment variable to point
# to it:  "export PYTHONSTARTUP=~/.pystartup" in bash.

import atexit
import os
#import readline
#import rlcompleter

historyPath = os.path.expanduser("~/.pyhistory")

def save_history(historyPath=historyPath):
    import readline
    readline.write_history_file(historyPath)

if os.path.exists(historyPath):
    #import readline
    readline.read_history_file(historyPath)

atexit.register(save_history)
del os, atexit, readline, rlcompleter, save_history, historyPath

I don't see why which python3.8 and PYTHONPATH were out of sync:

$ cygpath -w $(which python3.8)
C:\Users\user1\Documents\appls_mydocs\PortableApps\MSYS2Portable\App\msys32\mingw64\bin\python3.8.exe
$ echo $PYTHONPATH
/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8:/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages
$ cygpath -w /c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8
C:\Users\user1\Documents\appls_mydocs\PortableApps\MSYS2Portable\App\msys32\mingw64\lib\python3.8
$ cygpath -w /c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages
C:\Users\user1\Documents\appls_mydocs\PortableApps\MSYS2Portable\App\msys32\mingw64\lib\python3.8\site-packages
$ which python
/usr/bin/python

I seem to have two incomplete/broken python installations (3.7, 3.8). I don't know what led to an "incomplete" upgrade. A couple of observations (see code below):

1. python points to 3.7

2. readline is available for 3.7 and not for 3.8. I don't know why.

3. pandas (and many others) is available for 3.8 and not for 3.7. Many of them won't import in 3.8 anyway, due to missing dependencies (which I guess are available in 3.7). I don't know why.

4. python3.8 reports the location of .pythonrc in Windows format, while 3.7 reports it in Cygwin format. Is that normal?

5. Removing PYTHONPATH does not help.

$ unset PYTHONPATH
$ python3.8
Python 3.8.2 (default, Apr  9 2020, 13:17:39)  [GCC 9.3.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Reading C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/home/user1/.pythonrc
Module readline not available.
Traceback (most recent call last):
  File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/home/user1/.pythonrc", line 42, in <module>
    del os, atexit, readline, rlcompleter, save_history, historyPath
NameError: name 'readline' is not defined
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pandas/__init__.py", line 16, in <module>
    raise ImportError(
ImportError: Unable to import required dependencies:
numpy: DLL load failed while importing _ctypes: No se puede encontrar el módulo especificado.
>>> exit()
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'readline'
$ python
Python 3.7.4 (default, Jul 11 2019, 10:29:54)
[GCC 9.1.0] on msys
Type "help", "copyright", "credits" or "license" for more information.
Reading /home/user1/.pythonrc
readline is in /usr/lib/python3.7/lib-dynload/readline.cpython-37m.dll
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>> exit()

---

EDIT #4:

I checked for all python packages, and I found something strange. It seems the basic python installation comprises two separate packages, one mingw64 and another msys. I wouldn't know what does each one do.

$ pacman -Sl | grep "python" | grep "installed"
...
mingw64 mingw-w64-x86_64-python 3.8.2-2 [installed]
...
msys python 3.8.2-1 [installed: 3.7.4-1]
...

On one hand, there was a mismatch in installed versions mingw64 (3.8.2-2) vs msys (3.7.4-1). On the other hand, the subversions available are not exactly the same for both (3.8.2-2) vs (3.8.2-1). Anyway, I proceeded upgrading msys python, and this fixed things significantly.

$ pacman -S python
...
$ python
Python 3.8.2 (default, Apr 16 2020, 15:31:48)
[GCC 9.3.0] on msys
Type "help", "copyright", "credits" or "license" for more information.
Reading /home/user1/.pythonrc
Traceback (most recent call last):
  File "/home/user1/.pythonrc", line 9, in <module>
    import readline
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/readline.py", line 6, in <module>
    from pyreadline.rlmain import Readline
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/__init__.py", line 12, in <module>
    from . import logger, clipboard, lineeditor, modes, console
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/clipboard/__init__.py", line 13, in <module>
    from .win32_clipboard import GetClipboardText, SetClipboardText
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/clipboard/win32_clipboard.py", line 37, in <module>
    import ctypes.wintypes as wintypes
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/ctypes/wintypes.py", line 20, in <module>
    class VARIANT_BOOL(ctypes._SimpleCData):
ValueError: _type_ 'v' not supported
Failed calling sys.__interactivehook__
Traceback (most recent call last):
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site.py", line 412, in register_readline
    import readline
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/readline.py", line 6, in <module>
    from pyreadline.rlmain import Readline
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/__init__.py", line 12, in <module>
    from . import logger, clipboard, lineeditor, modes, console
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/clipboard/__init__.py", line 13, in <module>
    from .win32_clipboard import GetClipboardText, SetClipboardText
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/clipboard/win32_clipboard.py", line 37, in <module>
    import ctypes.wintypes as wintypes
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/ctypes/wintypes.py", line 20, in <module>
    class VARIANT_BOOL(ctypes._SimpleCData):
ValueError: _type_ 'v' not supported

So I have readline in 3.8 now. But there is another issue that pops up during loading of .pythonrc.

Plus, there is a problem with numpy. It does not stem from the python version (/usr/bin/python points now to 3.8).

$ python --version
Python 3.8.2
$ python
...
>>> import numpy
Traceback (most recent call last):
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/__init__.py", line 54, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.8 from "/usr/bin/python.exe",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.18.3" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

---

EDIT #5 As per @a_guest suggestion, and pointing me to "ValueError: _type_ 'v' not supported" error after installing PyReadline, I removed pyreadline, and this problem is gone.

Now the only remaining issue seems to be that numpy cannot load, possibly (but not certainly) due to two alternative python 3.8 systems present.

So the questions now is: Msys2: Two python installations?

Answer 1

The ImportError: cannot import name 'open_code' from 'io' (unknown location) comes from the fact that there are two different versions of Python conflicting with each other. python still points to the old version 3.7 but PYTHONPATH got updated to point to the new 3.8 version. As the documentation of PYTHONPATH states, it becomes prepended to the module search path and hence shadows any builtin modules:

The default search path is installation dependent, but generally begins with prefix/lib/pythonversion (see PYTHONHOME above). It is always appended to PYTHONPATH.

You can reproduce that behavior by creating two different virtual environments and then start one while having PYTHONPATH point to the other. In the following I used Miniconda to create two different environments, py37 and py38, containing a 3.7 and 3.8 installation respectively.

(py37) user@pc:~$ python --version
Python 3.7.6
(py37) user@pc:~$ PYTHONPATH=~/miniconda3/envs/py38/lib/python3.8/ python
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/py38/lib/python3.8/io.py", line 54, in <module>
ImportError: cannot import name 'open_code' from 'io' (unknown location)
Aborted (core dumped)