I am trying to use the Universal Encoding Detector (chardet) in Python to detect the most probable character encoding in a text file ('infile') and use that in further processing.
While chardet is designed primarily for detecting the character encoding of webpages, I have found an example of it being used on individual text files.
However, I cannot work out how to tell the script to set the most likely character encoding to the variable 'charenc' (which is used several times throughout the script).
My code, based on a combination of the aforementioned example and chardet's own documentation is as follows:
import chardet rawdata=open(infile,"r").read() chardet.detect(rawdata)
Character detection is necessary as the script goes on to run the following (as well as several similar uses):
inF=open(infile,"rb") s=unicode(inF.read(),charenc) inF.close()
Any help would be greatly appreciated.
Chardet: The Universal Character Encoding Detector Detects. ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants) Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)
There is a useful package in Python - chardet, which helps to detect the encoding used in your file. Actually there is no program that can say with 100% confidence which encoding was used - that's why chardet gives the encoding with the highest probability the file was encoded with.
chardet.detect()
returns a dictionary which provides the encoding as the value associated with the key 'encoding'
. So you can do this:
import chardet rawdata = open(infile, 'rb').read() result = chardet.detect(rawdata) charenc = result['encoding']
The chardet
documentation is not explicitly clear about whether text strings and/or byte strings are supposed to work with the module, but it stands to reason that if you have a text string you don't need to run character detection on it, so you should probably be passing byte strings. Hence the binary mode flag (b
) in the call to open()
. But chardet.detect()
might also work with a text string depending on which versions of Python and of the library you're using, i.e. if you do omit the b
you might find that it works anyway even though you're technically doing something wrong.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With