File.exists() returns false for file (directory) that actually exists

Question

TLDR: File.exists() is buggy and i would like to understand why!

---

I am facing a weird issue (as so often happens) in my Android App. I will try to be as brief as i can.

First, i will show you the code and then provide some additional info. This is not the full code. Just the core of the issue.

Example code:

String myPath = "/storage/emulated/0/Documents";
File directory= new File(myPath);
if (!directory.exists() && !directory.mkdirs()) {
   throw new IllegalArgumentException("Could not create the specified directory: " + directory.getAbsolutePath() + ".");
}

Most of the time this works fine. A few times however the exception is thrown which means that the directory did not exist and could not be created. Out of every 100 runs, it works fine on 95-96 times and fails 4-5 times.

• I have declared the permissions for storage/read external storage/write external storage in my manifest and asked for the permissions on runtime. The problem does not lie there. (If anything i have too many permissions at this point :D ). After all, if it was a permission issue it would fail every time but in my case it fails at a rate of 4% or 5%.
• With the above code i am attempting to create a file that points to the 'Documents' folder. In my app i am actually using String myPath = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOCUMENTS).getPath();
  In the specific device where the error occurs this path happens to be "/storage/emulated/0/Documents" and this is why i hardcoded it in the example code i gave you.
• If i use a file explorer app on the device (i.e. 'Astro file manager' i can see that the folder does exist and has some contents and also confirm that the path really is "/storage/emulated/0/Documents".
• This has never happened to me locally. Only the users of the app experience the issue and i know the issue exists thanks to Firebase/Crashlytics. The users have the exact same tablet as the one i am using for development, namely a Lenovo TB-8504X. (I work for a company and we provide both the software and the hardware).

So, do you have any thoughts on why this issue occurs?

Has anyone ever experienced something similar?

Could the path to the 'Documents' folder sometimes be "/storage/emulated/0/Documents" and sometimes become something else on the same physical device?

I am an experienced Android developer but i am quite novice in Android architecture and the Android filesystem. Could it be that on start-up (when device is powered on or after a reboot) the filesystem has not yet 'mounted' the 'disk' at the point when my code checks if the directory exists? Here i am using the terms 'mount' and 'disk' as loosely as possible. Also my app is actually a launcher/parental control app so it is the first thing that gets fired when device starts. I am almost conviced that this does not make sense at all but at this point i am trying to see the greater picture and explore solutions that transcend typical Android development.

I would really appreciate your help as this issue is starting to get on my nerves.

Looking forward to any helpful responses.

Thanks in advance.

EDIT (27/08/2019) :

I came across this Java Bug Report although it is pretty outdated. According to this, when operating on NFS-mounted volumes, java.io.File.exists ends up performing a stat(2). If the stat fails (which it may do for several reasons), then File.exists (mistakenly) assumes that the file being stat'ed does not exist. Could this be the source of my troubles?

EDIT (28/08/2019) :

Today i am able to add a bounty to this question in an attempt to draw some more attention. I would encourage you to read the question carefully, look through the comments disregarding the one that claims that this has to do with costumer support from Realm. Realm code is indeed the one using the unreliable method but what i want to know is why the method is unreliable. Whether or not Realm can work around this and use some other code instead, is beyond the scope of the question. I simply want to know if one can safely use File.exists() and if not, why?

Once again, thank you all in advance. It would be really important to me to get an answer even if it is overly technical and involves a deeper understanding of NFS file systems, Java, Android, Linux, or whatever!

EDIT (30/08/2019) :

Because some users suggest replacing File.exists() with some other method, i'd like to state that what i am interested in at this point is understating why the method fails and not what one could use instead as a workaround.

Even if i wanted to replace File.exists() with something else, i am not able to do that because this piece of code resides in RealmConfiguration.java file (Read-only) which is part of the Realm Library that i use in my app.

To make things even more clear i will provide two pieces of code. The code i use in my activity and the method that get's called in RealmConfiguration.java as a consequence:

Code i use in my activity :

File myfile = new File("/storage/emulated/0/Documents");
if(myFile.exists()){        //<---- Notice that myFile exists at this point.        
   Realm.init(this);

   config = new RealmConfiguration.Builder()
   .name(".TheDatabaseName")
   .directory(myFile)       //<---- Notice this line of code.
   .schemaVersion(7)
   .migration(new MyMigration())
   .build();

   Realm.setDefaultConfiguration(config);
   realm = Realm.getDefaultInstance();        
}

At this point myFile exists and the code that resides in RealmConfiguration.java get's called.

The RealmConfiguration.java method that crashes :

    /**
         * Specifies the directory where the Realm file will be saved. The default value is {@code context.getFilesDir()}.
         * If the directory does not exist, it will be created.
         *
         * @param directory the directory to save the Realm file in. Directory must be writable.
         * @throws IllegalArgumentException if {@code directory} is null, not writable or a file.
         */
        public Builder directory(File directory) {
            //noinspection ConstantConditions
            if (directory == null) {
                throw new IllegalArgumentException("Non-null 'dir' required.");
            }
            if (directory.isFile()) {
                throw new IllegalArgumentException("'dir' is a file, not a directory: " + directory.getAbsolutePath() + ".");
            }
------>     if (!directory.exists() && !directory.mkdirs()) {   //<---- Here is the problem
                throw new IllegalArgumentException("Could not create the specified directory: " + directory.getAbsolutePath() + ".");
            }
            if (!directory.canWrite()) {
                throw new IllegalArgumentException("Realm directory is not writable: " + directory.getAbsolutePath() + ".");
            }
            this.directory = directory;
            return this;
        }

So, myFile exists in my activity, the Realm code get's called and suddenly myFile no longer exists.. Again i wish to point out that this is not consistent. I am noticing crashes at a rate of 4-5% meaning that most of the time myFile exists both in the activity and when the realm code makes it's check.

I hope this will be helpful.

Again thanks in advance!

Answer 1

First of all, if you are using Android, bug reports in the Java Bugs database are not relevant. Android does not use the Sun / Oracle codebase. Android started out as a clean-room re-implementation of the Java class libraries.

So if there are bugs in File.exists() on Android the bugs would be in the Android codebase, and any reports would be in the Android issue tracker.

But when you say this:

According to this, when operating on NFS-mounted volumes, java.io.File.exists ends up performing a stat(2). If the stat fails (which it may do for several reasons), then File.exists (mistakenly) assumes that the file being stat'ed does not exist.

1. Unless you are using NFS, that bug report is not directly relevant.
2. It is not a mistake / bug. It is a limitation.
3. At the file system level, it is a fact of life that Linux supports many different kinds of file system, and that many of them behave in unexpected ways ... compared to an "ordinary" file system. It is not possible for the JVM to hide all of the weird filesystem-specific edge cases at the Java API level.
4. On the API level, File.exists cannot report any errors. The signature doesn't allow it to throw an IOException, and throwing an unchecked exception would be a breaking change. All it can say is true or false.
5. If you want to distinguish the various reasons for a false, you should use the newer Files.exists(Path, LinkOptions...) method instead.

Could this be the source of my troubles?

Yes it could, and not just in the NFS case! See below. (With Files.exist, an NFS stat failure would most likely be an EIO, and that would raise an IOException rather than returning false.)

---

The File.java code in the Android codebase (version android-4.2.2_r1) is:

public boolean exists() {
    return doAccess(F_OK);
}

private boolean doAccess(int mode) {
    try {
        return Libcore.os.access(path, mode);
    } catch (ErrnoException errnoException) {
        return false;
    }
}

Note how it turns any ErrnoException into a false.

A bit more digging reveals that the os.access call is performing a native call which makes an access syscall, and throws ErrnoException if the syscall fails.

So now we need look at the documented behavior of the access syscall. Here's what man 2 access says:

1. F_OK tests for the existence of the file.
2. On error (at least one bit in mode asked for a permission that is denied, or mode is F_OK and the file does not exist, or some other error occurred), -1 is returned, and errno is set appropriately.

3. access() shall fail if:

   • EACCES The requested access would be denied to the file, or search per‐ mission is denied for one of the directories in the path prefix of pathname. (See also path_resolution(7).)

   • ELOOP Too many symbolic links were encountered in resolving pathname.

   • ENAMETOOLONG pathname is too long.

   • ENOENT A component of pathname does not exist or is a dangling symbolic link.

   • ENOTDIR A component used as a directory in pathname is not, in fact, a directory.

   • EROFS Write permission was requested for a file on a read-only filesystem.

4. access() may fail if:

   • EFAULT pathname points outside your accessible address space.

   • EINVAL mode was incorrectly specified.

   • EIO An I/O error occurred.

   • ENOMEM Insufficient kernel memory was available.

   • ETXTBSY Write access was requested to an executable which is being executed.

I have struck out the errors that I think are technically impossible or implausible, but the still leaves quite few to consider.

---

Another possibility is something (e.g. some other part of your application) is deleting or renaming the file or a (hypothetical) symlink, or changing file permissions ... behind your back.

But I don't think that File.exist() is broken¹, or that the host OS is broken. It is theoretically possible, but you would need some clear evidence to support the theory.

^{1 - It is not broken in the sense that it is not behaving differently to the known behavior of the method. You could argue until the cows come home about whether the behavior is "correct", but it has been like that since Java 1.0 and it can't be changed in OpenJDK or in Android without breaking thousands of existing applications written over the last 20+ years. It won't happen.}

---

What to do next?

Well my recommendation would be to use strace to track the syscalls that your app is making and see if you can get some clues as to why some access syscalls are giving you unexpected results; e.g. what the paths are and what the errno is. See https://source.android.com/devices/tech/debug/strace .

Answer 2

I have had a similar issue, but with a higher trouble rate, where the Anti Virus was locking FileSystem, and thus failing any requests (almost instantly)

the workaround was using java.nio.Files.exists() instead.