Detect charset and convert to utf-8 in Python? [duplicate]

Question

Is there any universal method to detect string charset? I user IPTC tags and have no known encoding. I need to detect it and then change them to utf-8.

Anybody can help?

Answer 1

You want to use chardet, an encoding detector

Answer 2

It's a bit late, but there is also another solution: try to use pyicu.

An example:

import icu def convert_encoding(data, new_coding='UTF-8'):     coding = icu.CharsetDetector(data).detect().getName()     if new_coding.upper() != coding.upper():         data = unicode(data, coding).encode(new_coding)     return data