Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Allocate file on disk without zeroing

I need to allocate huge file without zeroing it's content. I'm producing this steps fopen => ftruncate => fclose => mmap => (...work...) => munmap with huge file sizes (hundreds of gigabytes). App hangs on termination for a few minutes while system is trying zeroing file bytes – IMHO because of ftruncate usage.

ftruncate(ofd, 0);

#ifdef HAVE_FALLOCATE

    int ret = fallocate(ofd, 0, 0, cache_size);
    if (ret == -1) {
        printf("Failed to expand file to size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
        exit(-1);
    }

#elif defined(HAVE_POSIX_FALLOCATE)

    int ret = posix_fallocate(ofd, 0, cache_size);
    if (ret == -1) {
        printf("Failed to expand file to size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
        exit(-1);
    }

#elif defined(__APPLE__)

    fstore_t store = {F_ALLOCATECONTIG, F_PEOFPOSMODE, 0, cache_size, 0};
    int ret = fcntl(ofd, F_PREALLOCATE, &store);
    if (ret == -1) {
        store.fst_flags = F_ALLOCATEALL;
        ret = fcntl(ofd, F_PREALLOCATE, &store);
    }
    if (ret == -1) { // read fcntl docs - must test against -1
        printf("Failed to expand file to size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
        exit(-1);
    }
    struct stat sb;
    ret = fstat(ofd, &sb);
    if (ret != 0) {
        printf("Failed to write to file to establish the size.\n");
        exit(-1);
    }
    //ftruncate(ofd, cache_size); <-- [1]

#endif

It seems it does not work with commented line [1]. But uncommenting this line produces file zeroing which I am trying to avoid. I really don't care dirty file content before writing. I just wanna avoid hang on app termination.

SOLUTION:

According to @torfo's answer, replaced all my Apple-related code with this few lines:

unsigned long long result_size = cache_size;
int ret = fcntl(ofd, F_SETSIZE, &result_size);
if(ret == -1) {
    printf("Failed set size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
    exit(-1);
}

But only works for superuser!

like image 723
k06a Avatar asked Jun 02 '17 07:06

k06a


1 Answers

This is MacOS X, apparently.

You can try to replace the ftruncate call with

fcntl(ofd, F_SETSIZE, &size);

(note needs root privilege and might create a security hole because it might provide access to old file contents that was previously there, so must be handled with extreme caution. The "dirty file content" you don't care about might actually be the user's bank account passwords he deleted a week ago...)

MacOS X doesn't really support sparse files - It does create and maintain them, but its file system driver is very eager to fill the holes as soon as possible.

like image 118
tofro Avatar answered Oct 18 '22 13:10

tofro