64 bit large mallocs

Question

What are the reasons a malloc() would fail, especially in 64 bit?

My specific problem is trying to malloc a huge 10GB chunk of RAM on a 64 bit system. The machine has 12GB of RAM, and 32 GB of swap. Yes, the malloc is extreme, but why would it be a problem? This is in Windows XP64 with both Intel and MSFT compilers. The malloc sometimes succeeds, sometimes doesn't, about 50%. 8GB mallocs always work, 20GB mallocs always fail. If a malloc fails, repeated requests won't work, unless I quit the process and start a fresh process again (which will then have the 50% shot at success). No other big apps are running. It happens even immediately after a fresh reboot.

I could imagine a malloc failing in 32 bit if you have used up the 32 (or 31) bits of address space available, such that there's no address range large enough to assign to your request.

I could also imagine malloc failing if you have used up your physical RAM and your hard drive swap space. This isn't the case for me.

But why else could a malloc fail? I can't think of other reasons.

I'm more interested in the general malloc question than my specific example, which I'll likely replace with memory mapped files anyway. The failed malloc() is just more of a puzzle than anything else... that desire to understand your tools and not be surprised by the fundamentals.

Answer 1

malloc tries to allocate a contiguous memory range, and this will initially be in real memory simply due to how swap memory works (at least as far as I remember). It could easily be that your OS sometimes can't find a contiguous block of 10gb of memory and still leave all the processes that require real memory in RAM at the same time (at which point your malloc will fail).

Do you actually require 10gb of contiguous memory, or would you be able to wrap a storage class/struct around several smaller blocks and use your memory in chunks instead? This relaxes the huge contiguous requirement and should also allow your program to use the swap file for less used chunks.