Why doesn't this memory eater really eat memory?

Question

I want to create a program that will simulate an out-of-memory (OOM) situation on a Unix server. I created this super-simple memory eater:

#include <stdio.h>
#include <stdlib.h>

unsigned long long memory_to_eat = 1024 * 50000;
size_t eaten_memory = 0;
void *memory = NULL;

int eat_kilobyte()
{
    memory = realloc(memory, (eaten_memory * 1024) + 1024);
    if (memory == NULL)
    {
        // realloc failed here - we probably can't allocate more memory for whatever reason
        return 1;
    }
    else
    {
        eaten_memory++;
        return 0;
    }
}

int main(int argc, char **argv)
{
    printf("I will try to eat %i kb of ram\n", memory_to_eat);
    int megabyte = 0;
    while (memory_to_eat > 0)
    {
        memory_to_eat--;
        if (eat_kilobyte())
        {
            printf("Failed to allocate more memory! Stucked at %i kb :(\n", eaten_memory);
            return 200;
        }
        if (megabyte++ >= 1024)
        {
            printf("Eaten 1 MB of ram\n");
            megabyte = 0;
        }
    }
    printf("Successfully eaten requested memory!\n");
    free(memory);
    return 0;
}

It eats as much memory as defined in memory_to_eat which now is exactly 50 GB of RAM. It allocates memory by 1 MB and prints exactly the point where it fails to allocate more, so that I know which maximum value it managed to eat.

The problem is that it works. Even on a system with 1 GB of physical memory.

When I check top I see that the process eats 50 GB of virtual memory and only less than 1 MB of resident memory. Is there a way to create a memory eater that really does consume it?

System specifications: Linux kernel 3.16 (Debian) most likely with overcommit enabled (not sure how to check it out) with no swap and virtualized.

Answer 1

All the virtual pages start out copy-on-write mapped to the same zeroed physical page. To use up physical pages, you can dirty them by writing something to each virtual page.

If running as root, you can use mlock(2) or mlockall(2) to have the kernel wire up the pages when they're allocated, without having to dirty them. (normal non-root users have a ulimit -l of only 64kiB.)

As many others suggested, it seems that the Linux kernel doesn't really allocate the memory unless you write to it

An improved version of the code, which does what the OP was wanting:

This also fixes the printf format string mismatches with the types of memory_to_eat and eaten_memory, using %zi to print size_t integers. The memory size to eat, in kiB, can optionally be specified as a command line arg.

The messy design using global variables, and growing by 1k instead of 4k pages, is unchanged.

#include <stdio.h>
#include <stdlib.h>

size_t memory_to_eat = 1024 * 50000;
size_t eaten_memory = 0;
char *memory = NULL;

void write_kilobyte(char *pointer, size_t offset)
{
    int size = 0;
    while (size < 1024)
    {   // writing one byte per page is enough, this is overkill
        pointer[offset + (size_t) size++] = 1;
    }
}

int eat_kilobyte()
{
    if (memory == NULL)
    {
        memory = malloc(1024);
    } else
    {
        memory = realloc(memory, (eaten_memory * 1024) + 1024);
    }
    if (memory == NULL)
    {
        return 1;
    }
    else
    {
        write_kilobyte(memory, eaten_memory * 1024);
        eaten_memory++;
        return 0;
    }
}

int main(int argc, char **argv)
{
    if (argc >= 2)
        memory_to_eat = atoll(argv[1]);

    printf("I will try to eat %zi kb of ram\n", memory_to_eat);
    int megabyte = 0;
    int megabytes = 0;
    while (memory_to_eat-- > 0)
    {
        if (eat_kilobyte())
        {
            printf("Failed to allocate more memory at %zi kb :(\n", eaten_memory);
            return 200;
        }
        if (megabyte++ >= 1024)
        {
            megabytes++;
            printf("Eaten %i  MB of ram\n", megabytes);
            megabyte = 0;
        }
    }
    printf("Successfully eaten requested memory!\n");
    free(memory);
    return 0;
}

Answer 2

When your malloc() implementation requests memory from the system kernel (via an sbrk() or mmap() system call), the kernel only makes a note that you have requested the memory and where it is to be placed within your address space. It does not actually map those pages yet.

When the process subsequently accesses memory within the new region, the hardware recognizes a segmentation fault and alerts the kernel to the condition. The kernel then looks up the page in its own data structures, and finds that you should have a zero page there, so it maps in a zero page (possibly first evicting a page from page-cache) and returns from the interrupt. Your process does not realize that any of this happened, the kernels operation is perfectly transparent (except for the short delay while the kernel does its work).

This optimization allows the system call to return very quickly, and, most importantly, it avoids any resources to be committed to your process when the mapping is made. This allows processes to reserve rather large buffers that they never need under normal circumstances, without fear of gobbling up too much memory.

---

So, if you want to program a memory eater, you absolutely have to actually do something with the memory you allocate. For this, you only need to add a single line to your code:

int eat_kilobyte()
{
    if (memory == NULL)
        memory = malloc(1024);
    else
        memory = realloc(memory, (eaten_memory * 1024) + 1024);
    if (memory == NULL)
    {
        return 1;
    }
    else
    {
        //Force the kernel to map the containing memory page.
        ((char*)memory)[1024*eaten_memory] = 42;

        eaten_memory++;
        return 0;
    }
}

Note that it is perfectly sufficient to write to a single byte within each page (which contains 4096 bytes on X86). That's because all memory allocation from the kernel to a process is done at memory page granularity, which is, in turn, because of the hardware that does not allow paging at smaller granularities.