Performance: memset

Question

I have simple C code that does this (pseudo code):

#define N 100000000
int *DataSrc = (int *) malloc(N);
int *DataDest = (int *) malloc(N);
memset(DataSrc, 0, N);
for (int i = 0 ; i < 4 ; i++) {
    StartTimer();
    memcpy(DataDest, DataSrc, N);
    StopTimer();
}
printf("%d\n", DataDest[RandomInteger]);

My PC: Intel Core i7-3930, with 4x4GB DDR3 1600 memory running RedHat 6.1 64-bit.

The first memcpy() occurs at 1.9 GB/sec, while the next three occur at 6.2 GB/s. The buffer size (N) is too big for this to be caused by cache effects. So, my first Question:

• Why is the first memcpy() so much slower? Maybe malloc() doesn't fully allocate the memory until you use it?

If I eliminate the memset(), then the first memcpy() runs at about 1.5 GB/sec, but the next three run at 11.8 GB/sec. Almost 2x speedup. My second question:

• Why is memcpy() 2x faster if I don't call memset()?

Answer 1

As others already pointed out, Linux uses an optimistic memory allocation strategy.

The difference between the first and the following memcpys is the initialization of DataDest.

As you have already seen, when you eliminate memset(DataSrc, 0, N), the first memcpy is even slower, because the pages for the source must be allocated as well. When you initialize both, DataSrc and DataDest, e.g.

memset(DataSrc, 0, N);
memset(DataDest, 0, N);

all memcpys will run with roughly the same speed.

For the second question: when you initialize the allocated memory with memset all pages will be laid out consecutively. On the other side, when the memory is allocated as you copy, the source and destination pages will be allocated interleaved, which might make the difference.

Answer 2

This is most likely due to lazy allocation in your VM subsystem. Typically when you allocate a large amount of memory only the first N pages are actually allocated and wired to physical memory. When you access beyond these first N pages then page faults are generated and further pages are allocated and wired in on an "on demand" basis.

As to the second part of the question, I believe some VM implementations actually track zeroed pages and handle them specially. Try initialising DataSrc to actual (e.g. random) values and repeat the test.