malloc returns null when run with Valgrind

Question

I have a huge allocation (tens of GBs) happening with a malloc call which works in normal scenario. The system does have a huge RAM and is a 64 bit machine running 2.6 x86_64 kernel.

The mem rlimit has been done as INFINITY with setrlimit.

I thought of running it with Valgrind to do the memory analysis and check for the leaks.

But malloc fails with NULL pointer returned when run with valgrind.

I tried reducing the size of allocation but that doesn't help.

Any inputs ?

Regards, -J

Answer 1

Note that malloc(3) is lying to you -- it doesn't actually allocate all the memory at once, it just asks the OS for it, and the OS lies to malloc(3). This is perfectly normal behavior that most of the time works fine. The description for /proc/sys/vm/overcommit_memory in proc(5) contains the details:

   /proc/sys/vm/overcommit_memory
          This file contains the kernel virtual memory
          accounting mode.  Values are:

                 0: heuristic overcommit (this is the default)
                 1: always overcommit, never check
                 2: always check, never overcommit

          In mode 0, calls of mmap(2) with MAP_NORESERVE are not
          checked, and the default check is very weak, leading
          to the risk of getting a process "OOM-killed".  Under
          Linux 2.4 any nonzero value implies mode 1.  In mode 2
          (available since Linux 2.6), the total virtual address
          space on the system is limited to (SS + RAM*(r/100)),
          where SS is the size of the swap space, and RAM is the
          size of the physical memory, and r is the contents of
          the file /proc/sys/vm/overcommit_ratio.

Valgrind cannot be so flippant; it actually keeps track of allocated, initialized, and uninitialized memory for a process. It therefore requires more memory than the process does on its own, and it does not have the same tolerance for over-committing memory.

I do not know how much more memory you will need to run the program under valgrind, but try adding a few more gigabytes of swap space. You can make a new swap file by using dd to write zeros to a file -- do not use a sparse file -- and then run mkswap(8) on the file to initialize it and run swapon(8) with the filename to tell the system to use it as a swap file.