malloc() in Linux - "there is no guarantee that the memory really is available"?

Question

I'm making a game where the world is divided into chunks of data describing the world. I keep the chunks in a dynamically allocated array so I have to use malloc() when initializing the world's data structures.

Reading the malloc() man page, there is a Note as follows:

By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available. In case it turns out that the system is out of memory, one or more processes will be killed by the OOM killer. For more information, see the description of /proc/sys/vm/overcommit_memory and /proc/sys/vm/oom_adj in proc(5), and the Linux kernel source file Documentation/vm/overcommit-accounting.

If Linux is set to use optimistic memory allocation then does this mean it doesn't always return the full amount of memory I requested in the call to malloc()?

I read that optimistic memory allocation an be disabled by modifying the kernel, but I don't want to do that.

So is there a way to check whether the program has allocated the requested amount?

Answer 1

This is not something you need to deal with from an application perspective. Users who don't want random processes killed by the "OOM killer" will disable overcommit themselves via

echo "2" > /proc/sys/vm/overcommit_memory

This is their choice, not yours.

But from another standpoint, it doesn't matter. Typical "recommended" amounts of swap are so ridiculous that no reasonable amount of malloc is going to fail to have physical storage to back it. However, you could easily allocate so much (even with forced MAP_POPULATE or manually touching it all) to keep the system thrashing swap for hours/days/weeks. There is no canonical way to ask the system to notify you and give an error if the amount of memory you want is going to bog down the system swapping.

The whole situation is a mess, but as an application developer, your role in the fix is just to use malloc correctly and check for a null return value. The rest of the responsibility is on distributions and the kernel maintainers.

Answer 2

Instead of malloc you can allocate the necessary memory directly by mmap, with MAP_POPULATE that advises the kernel to map the pages immediately.

#include <sys/mman.h>

// allocate length bytes and prefault the memory so 
// that it surely is mapped
void *block = mmap(NULL, length, PROT_READ|PROT_WRITE, 
       MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE,
       -1, 0);

// free the block allocated previously
// note, you need to know the size
munmap(block, length);

But the better alternative is that usually the world is saved to a file, so you would mmap the contents directly from a file:

int fd = open('world.bin', 'r+');
void *block = mmap(NULL, <filesize>, PROT_READ|PROT_WRITE,
    MAP_SHARED, fd, 0);

The file world.bin is mapped into the memory starting from address block; all changes to the memory would also written transparently to the file - no need to worry if there is enough RAM as linux will take care of mapping the pages in and out automatically.

---

Do note that some of these flags are not defined unless you have a certain feature test macro defined:

Certain flags constants are defined only if either _BSD_SOURCE or _SVID_SOURCE is defined. (Requiring _GNU_SOURCE also suffices, and requiring that macro specifically would have been more logical, since these flags are all Linux-specific.) The relevant flags are: MAP_32BIT, MAP_ANONYMOUS (and the synonym MAP_ANON), MAP_DENYWRITE, MAP_EXECUTABLE, MAP_FILE, MAP_GROWSDOWN, MAP_HUGETLB, MAP_LOCKED, MAP_NONBLOCK, MAP_NORESERVE, MAP_POPULATE, and MAP_STACK.