Pyinstaller error ImportError: No module named 'requests.packages.chardet.sys

Question

I can't seem to find the root cause of this. I don't know if it's pyinstaller, a pip problem, the requests module, or something else as nothing can be eliminated conclusively.

I wrote a script in python that properly configures a new hardware sonicwall for our enterprise network when we have to deploy a new unit. It configures a proper .exp file in memory, logs into the sonicwall device with default credentials, imports the file via a multi-part data form, restarts the sonicwall, then logs in again and changes the shared secret properly. For security reasons, I can't post the code here, but I can explain the problem with a much simpler example. Previously, the code was using urllib and urllib2 to process http requests, but then I discovered the requests module when I had to re-write the script to include csrfTokens.

Long story short, the script works amazing when called by the python interpreter. However, when trying to compile it with pyinstaller, I get a series of errors now that I've switched to requests instead of the urllibs.

Some more background:

Windows 7 - Python2.7.9
pip 6.0.8 from C:\Python27\lib\site-packages\pip-6.0.8-py2.7.egg (python 2.7)
pip freeze output:
pyinstaller==2.1.1.dev0
pywin32==219
requests==2.5.3

As an example, I'll give you some code that bombs out, a simple request to google.

#!/usr/bin/python
import requests 
r = requests.get('https://google.com') 
print(r.text)

The above code works for a simple request when I call the file from python, but when I compile it, I get this:

(EDIT Having problems pasting output, here is the pastebin) pastebin

It creates a windows executable, but this is the following error when I try to run it:

NOTE I did install the ms c++ 2008 redistributable to help clear up the msvcr90.dll, but I still get the above requests.packages.chardet.sys error.

I've tried everything I can think of, installing chardet, installing chardet2, installing cchardet, forcing earlier versions of pyinstaller and requests incrementally. Scrapping pip and manually installing pyinstaler and requests. I'm at my wits end with things to try, I don't quite get the error on hand here. requests.packages.chardet exists on the system. I also have C:\Python27 in my windows PATH as I can call python from any directory.

If you need more information, please let me know. I tried to be as thorough with the errors as possible and what I have installed, but I can provide more if needed.

ALSO This issue ONLY appears to be happening when I try to compile when I import requests. creating test scripts, beautifulsoup, urllib/2, etc all compile a valid windows exe that runs properly.

Answer 1

I don't have a solution for this yet, but this is caused by latest changes in requests module (versions 2.5.2 & 2.5.3).

For now you can use version 2.5.1 until PyInstaller will have suitable hook for solving this issue.

I cannot really explain the issue, but it looks like there's some kind of collision between PyInstaller import hooks and some latest additions to requests (VendorAlias).

Answer 2

Good news, this has been fixed in the latest version of requests

pip install requests --upgrade

Easy.

Answer 3

As m1keil says, the problem is between PyInstaller import hooks and the new load_module features of requests, implemented in the requests.packages.__init__.py source file.

Debugging this file you can see that any import of the requests package is passing through the load_module function. This includes the python standard packages. This the reason of the error.

My workaround consists in editing the requests.packages.__init__.py (inside the virtualenv folder) of the version 2.5.3 and adding this piece of code at the very beginning of the load_module function:

    print "Requested name = ", name #Comment this line when it works
    direct_loaded_packages = ('sys', 'errno','logging','warnings'
            ,'socket','os','re','time','hashlib','base64'
            ,'time','collections','datetime','io', 'argparse'
            ,'codecs', 'Queue', 'zlib', 'ssl', 'operator'
            ,'types','platform','struct', 'StringIO','httplib'
            ,'simplejson','cookielib','urllib','urlparse'
            ,'urllib2','Cookie','http','binascii','certifi'
            ,'uuid','json','threading','dummy_thread','email'
            ,'email.utils','operator','mimetypes')    


    new_name = ''
    #Package with three directory deep 
    if '.'.join(name.split('.')[3:]) in direct_loaded_packages:
        new_name = '.'.join(name.split('.')[3:])
    #Package with four directory deep
    elif '.'.join(name.split('.')[4:]) in direct_loaded_packages:
        new_name = '.'.join(name.split('.')[4:])
    #Package with five directory deep
    elif '.'.join(name.split('.')[5:]) in direct_loaded_packages:
        new_name = '.'.join(name.split('.')[5:])

    if new_name != '':
        module = __import__(new_name)
        return module 

By test&error and grepping I have build the list of packages that must be directly loaded without passing by the load_module function.

Isn't elegant but it works. I've tested a more elegant solution but I haven't succeeded.