Files disappearing from NSLibraryDirectory

Question

I'm storing some files in the Library directory in an iOS app, using the following methods to construct it. In the end, I can call [MyClass dataDirectory] to do my file handling and all is well. I've recently discovered, however, that some files seem to be mysteriously disappearing out of this directory. According to the documentation, this should not be the case. Is this a safe place to store persistent files?

The console output of this directory is: ~/var/mobile/Containers/Data/Application/{id}/Library/Data

+ (NSString*)libraryDirectory
{
    return [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory, NSUserDomainMask, YES) lastObject];
}

+ (NSString*)dataDirectory
{
    NSString* dir = [[self libraryDirectory] stringByAppendingPathComponent:@"Data"];
    BOOL isDir=NO;
    NSError * error = nil;
    NSFileManager *fileManager = [NSFileManager new];

    if (![fileManager fileExistsAtPath:dir isDirectory:&isDir] && isDir)
    {

        [[NSFileManager defaultManager] createDirectoryAtPath:dir
                                  withIntermediateDirectories:YES
                                                   attributes:nil
                                                        error:&error];
    }

    [self addSkipBackupAttributeToItemAtURL:[NSURL fileURLWithPath:dir isDirectory:YES]];

    if (error != nil) {
        DDLogError(@"Fatal error creating ~/Library/Data directory: %@", error);
    }
    return dir;
}

And the skip method:

+ (BOOL)addSkipBackupAttributeToItemAtURL:(NSURL *)URL
{
    if ([[NSFileManager defaultManager] fileExistsAtPath:[URL path]])
    {
        assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);

        NSError *error = nil;
        BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
                                      forKey: NSURLIsExcludedFromBackupKey error: &error];
        if(!success){
            DDLogError(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
        }
        return success;
    }
    return YES;
}

Answer 1

In the code you posted, the first problem is here:

if (![fileManager fileExistsAtPath:dir isDirectory:&isDir] && isDir)

At the point where this is evaluated, isDir will default to NO, and will be set to NO if the file does not exist or is not a directory. This will prevent the directory from being created. Remove && isDir or change to || !isDir to get the logic you want.

Now on to your original question:

Is this (a subdirectory of NSLibraryDirectory) a safe place to store persistent files?

Yes. NSLibraryDirectory is backed up by default. To comply with the iOS Data Storage Guidelines an application should not store user-created data in that location, but it is a safe place to store application data. NSApplicationSupportDirectory is a directory that is generally within the NSLibraryDirectory, and is the preferred place to store this kind of data. Data within that location will be backed up, and will be migrated during application and OS updates.

The iOS Data Storage Guidelines, File System Programming Guide, and App Programming Guide for iOS all provide guidance on where to put files, and how they will be backed up from standard file system locations.

Unless those files have had their NSURLIsExcludedFromBackupKey/kCFURLIsExcludedFromBackupKey resource metadata value altered. Then it gets much more complicated.

Files 'Excluded From Backup'

Generally, if a file outside of a Documents directory can be backed up, the system assumes it can also purge it under low space or other conditions. This is why setting NSURLIsExcludedFromBackupKey to YES on a file allows the file to persist even in low storage conditions. If your application sets NSURLIsExcludedFromBackupKey to YES for a file, your application assumes responsibility for the life of that file.

The catch here is that the backup process and the purge process do not follow the same logic. Apple's documentation indicates that for the purposes of controlling the backup behavior, it is possible to set NSURLIsExcludedFromBackupKey on a directory. The children of that directory will effectively inherit that resource value (in practice, this may not be accurate). The purge process, however, does not seem to have the same behavior. It may not check the backup exclusions of the parent directories and apply it to children, and as a result if a file does not have NSURLIsExcludedFromBackupKey explictly set it may be purged.

This gets even more complicated. If you were to read the documentation for the constant NSURLIsExcludedFromBackupKey you would see:

Some operations commonly made to user documents cause this property to be reset to false; consequently, do not use this property on user documents.

This actually applies to much more than user documents. For example, if you were to perform an atomic write on a file such as:

[thing writeToURL:URL atomically:YES encoding:NSUTF8StringEncoding error:&error]

If the file at URL had NSURLIsExcludedFromBackupKey set to YES before the write, it would now appear to be set to NO. An atomic write like this will first create a temporary file, write to that, and replace the original with the new file. In doing so, file and URL resource flags are not preserved. The original file had the NSURLIsExcludedFromBackupKey resource value set, the newly created file at the same location now does not. This is just one example; many Foundation APIs perform atomic writes like this implictly.

There are scenarios where this gets even more complex. When an application is updated it is installed into a new location with a new application container path. Data inside the old application container is migrated. There are few guarantees regarding what may or may not be migrated as part of the update process. It may be everything, it may be only some things. In particular there are is no guidance concerning how files or directories marked with the NSURLIsExcludedFromBackupKey resource attribute will be treated. In practice it seems that these are often the least likely files to be migrated, and when they are migrated the NSURLIsExcludedFromBackupKey attribute is rarely preserved.

OS updates are also an issue. Historically Over-The-Air updates have been problematic and have caused the NSURLIsExcludedFromBackupKey resource attribute to be effectively cleared or ignored. A "major" OS update will clear the device and restore from a backup - which is equivalent to migrating to new hardware. Files marked with the NSURLIsExcludedFromBackupKey resource attribute will not be migrated, and the application will have to re-create them.

Update scenarios are described in TechNote 2285: Testing iOS App Updates

Because of this, when using NSURLIsExcludedFromBackupKey it is generally best to set the value on every access, and as always should be done through the File Coordination APIs (unless you are writing to a shared group container, which is an entirely different set of issues). If the NSURLIsExcludedFromBackupKey resource attribute value is lost files can be purged at any time. Ideally an application should not depend on the NSURLIsExcludedFromBackupKey or how the OS may (or may not!) handle it, but instead be designed such that the data could be recreated on demand. That may not always be possible.

It's clear from your question and the code that you posted that you are somewhat dependant on NSURLIsExcludedFromBackupKey ensuring that your file(s) have an application-controlled lifetime. As you can see from the above, that may not always be the case: there are many, many common scenarios where that resource attribute value can disappear, and with it your files.

It is also worth noting that NSFileProtection attributes work the same way, and can disappear in the same scenarios (and a few more).

TL;DR; What should I do?

Based on your question, code, and the description of the behavior you are seeing:

• Setting the NSURLIsExcludedFromBackupKey value on the directory containing the file(s) you are interested in preserving may not be enough to prevent them from being purged. It would be wise to set NSURLIsExcludedFromBackupKey on every access to the actual files, rather than just a parent directory. Also attempt to ensure this resource value is set after any write to the file, especially through a high level API that may be doing atomic writes, etc.

• All NSFileManager and file reading/writing operations should use file coordination. Even in an application that is single threaded there will be other processes interacting with "your" files. Processes like the daemons that run backups or purge files during low space conditions. Between your -fileExistsAtPath: and the -setResourceValue:forKey:error: another process could alter, delete, or move your file and its attributes. -setResourceValue:forKey:error: will actually return YES and no error in many cases where it did nothing, like the file not existing.

• Files and directories marked with NSURLIsExcludedFromBackupKey are the responsibility of the application to manage. The application should still purge those files or their contents at some appropriate time, or set limits on their growth. If you look at the per-application disk usage information on a device, you can probably guess the names of some applications that do not do this correctly.

• Test update scenarios as described in TechNote 2285: Testing iOS App Updates. often. Ideally the iOS Simulator would have a "Simulate Low Disk Space" capability similar to simulating memory warnings, but at this time it does not.

• If at all possible, alter application logic to recreate these files if they go missing.

Answer 2

In the documentation you linked it is stated that
Critical data should be stored in the /Documents directory. Critical data is any data that cannot be recreated by your app, such as user documents and other user-generated content.

It is also mentioned that
Cached data should be stored in the /Library/Caches directory. Examples of files you should put in the Caches directory include (but are not limited to) database cache files and downloadable content, such as that used by magazine, newspaper, and map apps. Your app should be able to gracefully handle situations where cached data is deleted by the system to free up disk space.

The directory you are using is not explicitly mentioned for storing user data, it is used by the system and is not save for your data. It's guarantied to be untouched by an update of your app, but that's it

To find the documents folder you could do something like

NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES); 
NSString *documentsFolderPath = [paths firstObject];