Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Java 8 change in UTF-8 decoding

We recently migrated our application to JDK 8 from JDK 7. After the change, we ran into a problem with the following snippet of code.

String output = new String(byteArray, "UTF-8");

The byte array may contain invalid UTF-8 byte sequences. The same byte array upon UTF-8 decoding, results in two difference strings on Java 7 and Java 8.

According to the answer to this SO post, Java 8 "fixes" an error in Java 7 and replaces invalid UTF-8 byte sequences with a replacement string, which is in accordance with the UTF-8 specification.

But we would like to stick with Java 7's version of the decoded string.

We have tried to use CharsetDecoder with CodingErrorAction as REPLACE, REPORT and IGNORE on Java 8. Still, we were not able to generate the same string as Java 7.

Can we do this with a technique of reasonable complexity?

like image 688
Jiraiya Avatar asked Jun 01 '15 13:06

Jiraiya


People also ask

Is Java UTF-8 or 16?

The native character encoding of the Java programming language is UTF-16. A charset in the Java platform therefore defines a mapping between sequences of sixteen-bit UTF-16 code units (that is, sequences of chars) and sequences of bytes.


1 Answers

From the pointers provided by @Holger, It was clear that we had to write a custom CharsetDecoder.

I copied over OpenJDK's version of sun.nio.cs.UTF_8 class, renamed it to CustomUTF_8 and used it to construct a string like so

String output = new String(bytes, new CustomUTF_8());

I plan to run extensive tests cross verifying the outputs generated on Java 7 and Java 8. This is an interim solution while I am trying to fix the actual problem of passing output from hmac directly to String without Base64 encoding it first to.

 String output = new String(Base64.Encoder.encode(bytes), Charset.forname("UTF-8"));
like image 69
Jiraiya Avatar answered Oct 17 '22 21:10

Jiraiya