UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position X: character maps to <undefined>

Question

When I'm trying to install StringGenerator with pip, I am prompted with this error:

C:\Users\Administrator> pip install StringGenerator

Collecting StringGenerator 
Using cached StringGenerator-0.3.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\ADMINI~1\AppData\Local\Temp\2\pip-build-mdvrj2cf\StringGenerator\setup.py", line 7, in <module>
    long_description = file.read()
  File "c:\users\administrator\appdata\local\programs\python\python36-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1264: character maps to <undefined>

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\ADMINI~1\AppData\Local\Temp\2\pip-build-mdvrj2cf\StringGenerator\

Answer 1

The problem is caused during the setup process when reading README.txt. In Windows, the default encoding is cp1252, but that readme file is most likely encoded in UTF8.

The error message tells you that cp1252 codec is unable to decode the character with the byte 0x9D. When I browsed through the readme file, I found this character: ” (also known as: "RIGHT DOUBLE QUOTATION MARK"), which has the bytes 0xE2 0x80 0x9D, which includes the problematic byte.

What you can do is:

1. Download the package here
2. Decompress the package
3. Open setup.py
4. Change the following:

From:

with open('README.txt') as file:
    long_description = file.read()

Change into:

with open('README.txt', encoding="utf8") as file:
    long_description = file.read()

This will open the file with the proper encoding.

Or you can remove these two line altogether and also remove long_description=long_description, at line 18 inside setup().

5. In console, run python setup.py install
6. And you're done!

---

Since there's no actual setup in the setup.py script, you can just directly clone the source folder from GitHub, the package should still work properly.

Answer 2

Simply add encoding="utf8" inside "open('path', here)".

with open('path to csv file',  encoding="utf8") as csv_file:

Answer 3

I also had this problem with a pip install on a Windows version of python. The solution is to set the following environment variable:

PYTHONUTF8=1

You can unset it after the installation is complete if it could interfere with your normal development.

Answer 4

Go to https://pypi.python.org/pypi/StringGenerator/0.3.0 and download the latest version (or source in this case), extract the .gz file and then the .tar file.

Next go in StringGenerator-0.2.0 folder and open a terminal and run:

python setup.py build
python setup.py install 

Or from PowerShell run:

python ./setup.py build
python ./setup.py install