Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

java read write unicode / UTF-8 filenames (not contents)

i have a few directories/files with Japanese characters. If i try to read a filename (not the contents) containing (as example) a ク i receive a String containing a �. If i try to create a file/directory containing an ク a file/directory appears containing a ?.

As example: I list the files with.

File file = new File(".");  
String[] filesAndDirs = file.list();

the filesAndDirs array now contains the directories this the special characters. The String now only contains ����. It seams there is nothing to decode because the a getbytes shows only "-17 -65 -67" for every char in the filename even for different chars.

I use MacOS 10.8.2 Java 7_10 and Netbeans.

Any ideas?

Thank You in advance :)

like image 289
uti.devel Avatar asked Jan 05 '13 12:01

uti.devel


2 Answers

Those bytes are 0xef 0xbf 0xbd, which is the UTF-8-encoded form of the \ufffd character you're seeing instead of the Japanese characters. It appears whatever OS function Java is using to list the files is in fact returning those incorrect characters.

Perhaps Files.newDirectoryStream will be more reliable. Try this instead:

try (DirectoryStream<Path> dir = Files.newDirectoryStream(Paths.get("."))) {
    for (Path child : dir) {
        String filename = child.getFileName().toString();

        System.out.println("name=" + filename);
        for (char c : filename.toCharArray()) {
            System.out.printf("%04x ", (int) c);
        }
        System.out.println();
    }
}
like image 97
VGR Avatar answered Oct 14 '22 06:10

VGR


It's a bug in the old java File api (maybe just on a mac). Anyway, it's all fixed in the new java.nio.

I have several files containing unicode characters in the filename and content that failed to load using java.io.File and related classes. After converting all my code to use java.nio.Path EVERYTHING started working. And I replaced org.apache.commons.io.FileUtils (which has the same problem) with java.nio.Files...

...and be sure to read and write the content of file using an appropriate charset, for example: Files.readAllLines(myPath, StandardCharsets.UTF_8)

like image 22
pomo Avatar answered Oct 14 '22 07:10

pomo