I keep getting this error while reading a text file. Is it possible to handle/ignore it and proceed?
UnicodeEncodeError: ‘charmap’ codec can’t decode byte 0x81 in position 7827: character maps to undefined.
Try to run your code normally. If you get any error reading a Unicode text file you need to use the encoding='utf-8' parameter while reading the file.
When we use such a string as a parameter to any function, there is a possibility of the occurrence of an error. Such error is known as Unicode error in Python. We get such an error because any character after the Unicode escape sequence (“ \u ”) produces an error which is a typical error on windows.
To decode a string encoded in UTF-8 format, we can use the decode() method specified on strings. This method accepts two arguments, encoding and error . encoding accepts the encoding of the string to be decoded, and error decides how to handle errors that arise during decoding.
In Python 3, pass an appropriate errors=
value (such as errors=ignore
or errors=replace
) on creating your file object (presuming it to be a subclass of io.TextIOWrapper
-- and if it isn't, consider wrapping it in one!); also, consider passing a more likely encoding than charmap
(when you aren't sure, utf-8
is always a good place to start).
For instance:
f = open('misc-notes.txt', encoding='utf-8', errors='ignore')
In Python 2, the read()
operation simply returns bytes; the trick, then, is decoding them to get them into a string (if you do, in fact, want characters as opposed to bytes). If you don't have a better guess for their real encoding:
your_string.decode('utf-8', 'replace')
...to replace unhandled characters, or
your_string.decode('utf-8', 'ignore')
to simply ignore them.
That said, finding and using their real encoding (rather than guessing utf-8
) would be preferred.
You should open the file with a codecs to make sure that the file gets interpreted as UTF8.
import codecs fd = codecs.open(filename,'r',encoding='utf-8') data = fd.read()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With