ModuleNotFoundError in tracebacks with Python3.6 on linux

Question

I installed Python 3.6 on Ubuntu 16.04 by using Jonathon Fernyhough's PPA:

sudo add-apt-repository ppa:jonathonf/python-3.6
sudo apt-get update
sudo apt-get install python3.6

I made a string, using the new literal string interpolation, but I supplied an invalid format specifier. I not only got the expected ValueError: Invalid format specifier, but also the unexpected ModuleNotFoundError: No module named 'apt_pkg'.

$ python3.6
Python 3.6.0 (default, Dec 29 2016, 21:40:36) 
[GCC 5.4.1 20161202] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> value = 4 * 20
>>> f'the value is {value:%A}'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier

I reported this to the Python bug tracker. There it was noted that:

It seems to be vendor's issue not CPython itself. This same issue also happens in Ubuntu 16.10's Python 3.6. Raise any exception can cause this:

Python 3.6.0b2 (default, Oct 11 2016, 05:27:10) 
[GCC 6.2.0 20161005] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> raise Exception
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Exception
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Exception
>>> 

Also see https://bugs.launchpad.net/ubuntu/+source/python3.6/+bug/1631367.

Finally, the issue was closed with the comment

Yes, this appears to be the vendor's failure reporting infrastructure that is failing. Why they'd want a report for every traceback at the interactive prompt is beyond me, but that appears to be what they are trying to do.

My questions now are:

1. How do I interpret this comment? Is the vendor in this case Jonathon Fernyhough's PPA? And did he change something to to the Python code he distributes so that it tries to file a report for every Exception that produces a traceback?
2. Who do I need to notify or where do I need to file a bug to get this resolved?

Answer 1

I solved this issue for Python 3.6 by first installing the python-apt package for Python 3:

sudo apt install python3-apt

After that, I changed to the dist-packages directory and copied the apt_pkg[...].so file to the new default file name apt_pkg.so, to be also recognized by Python 3.6:

cd /usr/lib/python3/dist-packages
sudo cp apt_pkg.cpython-34m-i386-linux-gnu.so apt_pkg.so

Now all ModuleNotFoundError: No module named 'apt_pkg' exceptions disappeared on expectedly thrown error messages.

Answer 2

After I posted this question to Stackoverflow, Barry A. Warsaw made the following comment to the issue tracker

Please understand that installing Python 3.6 from a random PPA does not provide full support for this version of the interpreter. Python 3.6 is not yet a supported version in any version of Ubuntu (which I'm assuming your using), although we are working on it for 17.04.

Very often, you can install a new Python 3 interpreter package and many things will work because the Ubuntu infrastructure shares pure-Python modules across all installed Python 3's. Technically speaking, they will all have /usr/lib/python3/dist-packages on their sys.path so any third party pure-Python module built for a support version of Python 3 will be importable by any (package-built) installed version of Python 3.

But that 1) is a long way from saying that those third-party modules will work; 2) does not include any packages containing C extension modules, which must be rebuilt for the specific interpreter version.

Supporting a new version of Python is a long process, for which we are just starting. Please engage with [email protected] for details.

Ubuntu does install a standard exception handler so that when Python applications and such crash, we can gather crash statistics, so that we can devote resources to fixing common problems and regressions. apport (which you see in the traceback) is that crash reporting infrastructure. apport calls apt_pkg, which is an (C++) extension module and thus won't have been built for the version of Python 3.6 you installed from that PPA, unless of course the PPA owner (who I don't know) has also done an archive-wide Python 3 rebuild. Since I'm in the process of setting that up, and I know it's quite a bit of work, I doubt that's been done for this rather random PPA.

The ubuntu-devel mailing list is a better place to discuss the ongoing work to bring Python 3.6 as a supported version on Ubuntu.

Answer 3

If most voted answer does not work for you please try below steps (Keep in mind you must change [version]):

1. run this command sudo apt install python3-apt
2. go to directory cd /usr/lib/python3/dist-packages
3. run ls to find correct version of apt_pkg.cpython-[version]-i386-linux-gnu.so in my case it was 35m
4. create a symlink sudo cp apt_pkg.cpython-[version]-i386-linux-gnu.so apt_pkg.so DON'T FORGET: type your [version]

Troubleshouting

If you get error cp: cannot stat 'apt_pkg.cpython-[your-version]-i386-linux-gnu.so': No such file or directory try commands below:

1. rm -rf apt_pkg.so
2. create a symlink sudo cp apt_pkg.cpython-[version]-i386-linux-gnu.so apt_pkg.so DON'T FORGET: type your [version]