Can't read UTF-8 filenames when launched as an Upstart service

Question

My Java program reads the contents of a directory recursively. This is a sample tree (note the non-ASCII characters):

./sviluppo
./sviluppo/ciaò
./sviluppo/ciaò/subdir
./sviluppo/pippo
./sviluppo/pippo/prova2.txt <-file
./sviluppo/così

The program is started as an Upstart service, with a configuration file named like /init/myservice.conf

description "Private Service"
author "AD"
start on runlevel [2345]
stop on runlevel [! 2345]
exec java -jar /home/mainFind.jar >> /tmp/log.txt

When I launch the service:

root@mdr:/tmp#  service myservice start
myservice start/running, process 15344

it doesn't log filenames with non-ASCII characters in the name:

root@mdr:/tmp#  cat /tmp/log.txt
Found dir: /mnt/sviluppo/pippo

Instead, when I run the command (as root, to mimic what happens when it's started as a service) it works fine, with and without exec:

root@mdr:/tmp# java -jar /home/mainFind.jar  >> /tmp/log.txt
root@mdr:/tmp# exec java -jar /home/mainFind.jar  >> /tmp/log.txt

root@mdr:/tmp#  cat /tmp/log.txt
Found dir: /mnt/sviluppo/ciaò
Found dir: /mnt/sviluppo/ciaò/subdir
Found dir: /mnt/sviluppo/pippo
Found dir: /mnt/sviluppo/così

Why the same program run by the same user doesn't work in an Upstart service, but correctly processes all of the filenames when run from the command line? Here is the Java code

public static void aggiungiFileDir(File f){
  File[] lista= f.listFiles();
  for(int i=0;i<lista.length;i++){
    if(lista[i].isDirectory()){
      System.out.println("Found dir: "+lista[i]); 
    }
  }
}

Where the formal parameter f is the root dir. The function will be called recursively on each subdir.

EDIT 2: Post ls

root@mdr:/tmp# ls -al /mnt/sviluppo
totale 20
drwx------ 5 root root 4096 nov 15 15:10 .
drwxr-xr-x 7 root root 4096 nov  9 10:43 ..
drwxr-xr-x 2 root root 4096 nov 15 15:10 ciaò
drwxr-xr-x 2 root root 4096 nov 15 11:23 così
drwxr-xr-x 2 root root 4096 nov 15 17:57 pippo

Answer 1

Java uses a native call to list the contents of a directory. The underlying C runtime relies on the locale concept to build Java Strings from the byte blob stored by the filesystem as the filename.

When you execute a Java program from a shell (either as a privileged user or an unprivileged one) it carries an environment made of variables. The variable LANG is read to transcode the stream of bytes to a Java String, and by default on Ubuntu it's associated to the UTF-8 encoding.

Note that a process need not to be run from any shell, but looking at the code it seems that Upstart is smart enough to understand when the command in the configuration file is intended to be executed from a shell. So, assuming that the JVM is invoked through a shell, the problem is that the variable LANG is not set, so the C runtime assumes a default charset, which happens to not be UTF-8. The solution is in the Upstart stanza:

description "List UTF-8 encoded filenames"
author "Raffaele Sgarro"
env LANG=en_US.UTF-8
script
  cd /workspace
  java -jar list.jar test > log.txt
end script

I used en_US.UTF-8 as the locale, but any UTF-8 backed one will do just as well. The sources of the test list.jar

public static void main(String[] args) {
    for (File file : new File(args[0]).listFiles()) {
        System.out.println(file.getName());
    }
}

The directory /workspace/test contains filenames like ààà, èèè and so on. Now you can move to the database part ;)