Guessing the encoding of text represented as byte[] in Java

Question

Given an array of bytes representing text in some unknown encoding (usually UTF-8 or ISO-8859-1, but not necessarily so), what is the best way to obtain a guess for the most likely encoding used (in Java)?

Worth noting:

• No additional meta-data is available. The byte array is literally the only available input.
• The detection algorithm will obviously not be 100 % correct. If the algorithm is correct in more than say 80 % of the cases that is good enough.

Answer 1

The following method solves the problem using juniversalchardet, which is a Java port of Mozilla's encoding detection library.

public static String guessEncoding(byte[] bytes) {     String DEFAULT_ENCODING = "UTF-8";     org.mozilla.universalchardet.UniversalDetector detector =         new org.mozilla.universalchardet.UniversalDetector(null);     detector.handleData(bytes, 0, bytes.length);     detector.dataEnd();     String encoding = detector.getDetectedCharset();     detector.reset();     if (encoding == null) {         encoding = DEFAULT_ENCODING;     }     return encoding; } 

The code above has been tested and works as intented. Simply add juniversalchardet-1.0.3.jar to the classpath.

I've tested both juniversalchardet and jchardet. My general impression is that juniversalchardet provides the better detection accuracy and the nicer API of the two libraries.

Answer 2

There is also Apache Tika - a content analysis toolkit. It can guess the mime type, and it can guess the encoding. Usually the guess is correct with a very high probability.