Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Character detection in a text file in Python using the Universal Encoding Detector (chardet)

Tags:

I am trying to use the Universal Encoding Detector (chardet) in Python to detect the most probable character encoding in a text file ('infile') and use that in further processing.

While chardet is designed primarily for detecting the character encoding of webpages, I have found an example of it being used on individual text files.

However, I cannot work out how to tell the script to set the most likely character encoding to the variable 'charenc' (which is used several times throughout the script).

My code, based on a combination of the aforementioned example and chardet's own documentation is as follows:

import chardet     rawdata=open(infile,"r").read() chardet.detect(rawdata) 

Character detection is necessary as the script goes on to run the following (as well as several similar uses):

inF=open(infile,"rb") s=unicode(inF.read(),charenc) inF.close() 

Any help would be greatly appreciated.

like image 960
木川 炎星 Avatar asked Jul 24 '10 04:07

木川 炎星


People also ask

What is Chardet Python?

Chardet: The Universal Character Encoding Detector Detects. ASCII, UTF-8, UTF-16 (2 variants), UTF-32 (4 variants) Big5, GB2312, EUC-TW, HZ-GB-2312, ISO-2022-CN (Traditional and Simplified Chinese)

How do I check the encoding of a text file in Python?

There is a useful package in Python - chardet, which helps to detect the encoding used in your file. Actually there is no program that can say with 100% confidence which encoding was used - that's why chardet gives the encoding with the highest probability the file was encoded with.


1 Answers

chardet.detect() returns a dictionary which provides the encoding as the value associated with the key 'encoding'. So you can do this:

import chardet     rawdata = open(infile, 'rb').read() result = chardet.detect(rawdata) charenc = result['encoding'] 

The chardet documentation is not explicitly clear about whether text strings and/or byte strings are supposed to work with the module, but it stands to reason that if you have a text string you don't need to run character detection on it, so you should probably be passing byte strings. Hence the binary mode flag (b) in the call to open(). But chardet.detect() might also work with a text string depending on which versions of Python and of the library you're using, i.e. if you do omit the b you might find that it works anyway even though you're technically doing something wrong.

like image 158
David Z Avatar answered Oct 10 '22 19:10

David Z