Is it possible to use ReadList
to read UTF-8 (or any other) encoded text files using ReadList[..., Word]
, or is it ASCII-only? If it's ASCII-only, is it possible to "fix" the encoding of the already read data with good performance (i.e. preserving the performance advantages of ReadList
over Import
)?
Import[..., CharacterEncoding -> "UTF8"]
works but it's quite a bit slower than ReadList
. $CharacterEncoding
has no effect on ReadList
Download a sample UTF-8 encoded file here.
For testing performance on a large input, see the test file in this question.
Here are the timings of the answers on a large-ish text file:
Import
In[2]:= Timing[
data = Import[file, "Text"];
]
Out[2]= {5.234, Null}
Heike
In[4]:= Timing[
data = ReadList[file, String];
FromCharacterCode[ToCharacterCode[data], "UTF8"];
]
Out[4]= {4.328, Null}
Mr. Wizard
In[5]:= Timing[
string = FromCharacterCode[BinaryReadList[file], "UTF-8"];
]
Out[5]= {2.281, Null}
In Java, the InputStreamReader accepts a charset to decode the byte streams into character streams. We can pass a StandardCharsets. UTF_8 into the InputStreamReader constructor to read data from a UTF-8 file. In Java 7+, many file read APIs start to accept charset as an argument, making reading a UTF-8 very easy.
This seems to work
FromCharacterCode[ToCharacterCode[ReadList["raw.php.txt", Word]], "UTF-8"]
The timings I get for the linked test file are
FromCharacterCode[ToCharacterCode[ReadList["test.txt", Word]], "UTF-8"]); // Timing
(* ==> {0.000195, Null} *)
Import["test.txt", "Text"]; // Timing
(* ==> {0.01784, Null} *)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With