malloc vs mmap performance

Question

I ran a performance test on writing 128 million ints to memory allocated with malloc and to a mapped memory file (backed by file on disk) using mmap...I had expected the results to be somewhat similar as it's my understanding that when writing to a mapped memory file, the data is initially written to memory and pdflush writes to disk in the background (at a frequency that can that can be configured). With malloc, writing the 128M ints took .55 secs; mmap took 1.9 secs.

So my question is: why the difference. My initials thoughts were that pdflush is crowding the bus or that when pdflush is accessing the memory, it is blocking writes...However, running the mmap version for a second time produced a result of .52 secs (due to caching) which leads me to believe that the each page behind mmap is not allocated until it is written to (despite reserving it with call to mmap)...it is also my understanding that the memory produced by malloc is not actually allocated until the first write...could the initial difference be because after the initial write to memory by malloc, the entire chunk is allocated and with mmap, every time a new page is written to, the os must allocate it first?

UPDATE:

os: CentOS Linux release 7.0.1406 (Core) kernel: 3.10.0-123.el7.x86_64 gcc: 4.8.2

CODE:

int* pint = malloc(128000000 * sizeof(int));
int* pint_copy = pint;

clock_t start = clock();

int i;
for(i = 0; i < 128000000; ++i)
{
    *pint++ = i;
}   

clock_t end = clock();

double cpu_time_used = ((double) (end - start)) / CLOCKS_PER_SEC;
printf("%f\n", cpu_time_used);

free(pint_copy);

vs

int fd = open("db", O_RDWR | O_CREAT, 0666);
const size_t region_size = ((512000000 / sysconf(_SC_PAGE_SIZE)) + 1) * sysconf(_SC_PAGE_SIZE); 

int return_code = ftruncate(fd, region_size);

if (return_code < 0)
    printf("mapped memory file could not be truncated: %u\n", return_code);

int* pint = mmap(NULL, region_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
int* pint_copy = pint;
close(fd);  

clock_t start = clock();

int i;
for(i = 0; i < 128000000; ++i)
{
    *pint++ = i;
}   

clock_t end = clock();

double cpu_time_used = ((double) (end - start)) / CLOCKS_PER_SEC;
printf("%f\n", cpu_time_used);

fgetc(stdin);

munmap(pint_copy, region_size);

ADDING:

int z = 512;
while(z < 128000000)
{
    pint[z] = 0;

    z += 1024;
}

BEFORE:

  clock_t start = clock();     

Produces .37 secs for both trials, leading me to believe that "touching" each page causes the os to allocate physical memory (both for mmap and malloc)...it could also partially be because "touching" the pages moves some of the memory to cache...does anyone know if during heavy writes to memory (for an extended period of time), pdflush would block or slow memory writes?

Answer 1

Yes your are right. Pages obtained with mmap are not filled until you try accessing them. You can't have a guarantee for this, but usually OSes use write-back (there is no penalty for this only gain) and demand-paging (you must pay the first access).

Answer 2

I don't know the answer, but this looks to me like you are comparing apples to oranges.

To wit, on the one hand you are writing to (malloc'd) memory, while on the other hand you're writing to memory and to (mmap'd) disc. I would expect the second, which incurs device I/O activity, to be an order of magnitude slower than the first, which does not incur any I/O.