What is the largest amount of memory I can allocate on my MacBook Pro? [duplicate]

Question

I'm trying to figure out how much memory I can allocate before the allocation will fail.

This simple C++ code allocates a buffer (of size 1024 bytes), assigns to the last five characters of the buffer, reports, and then deletes the buffer. It then doubles the size of the buffer and repeats until it fails.

Unless I'm missing something, the code is able to allocate up to 65 terabytes of memory before it fails on my MacBook Pro. Is this even possible? How can it allocate so much more memory than I have on the machine? I must be missing something simple.

int main(int argc, char *argv[])
{
        long long size=1024;
        long cnt=0;
        while (true)
        {
                char *buffer = new char[size];
                // Assume the alloc succeeded. We are looking for the failure after all.

                // Try to write to the allocated memory, may fail
                buffer[size-5] = 'T';
                buffer[size-4] = 'e';
                buffer[size-3] = 's';
                buffer[size-2] = 't';
                buffer[size-1] = '\0';

                // report
                if (cnt<10)
                        cout << "size[" << cnt << "]: " << (size/1024.) << "Kb ";
                else if (cnt<20)
                        cout << "size[" << cnt << "]: " << (size/1024./1024.) << "Mb ";
                else
                        cout << "size[" << cnt << "]: " << (size/1024./1024./1024.) << "Gi ";
                cout << "addr: 0x" << (long)buffer << " ";
                cout << "str: " << &buffer[size-5] << "\n";

                // cleanup
                delete [] buffer;

                // double size and continue
                size *= 2;
                cnt++;
        }
        return 0;
}

Answer 1

When you ask for memory, an operating system reserves the right not to actually give you that memory until you actually use it.

That's what's happening here: you're only ever using 5 bytes. My ZX81 from the 1980s could handle that.

Answer 2

MacOS X, like almost every modern operating system, uses "delayed allocation" for memory. When you call new, the OS doesn't actually allocate any memory. It simply makes a note that your program wants a certain amount of memory, and that memory area you want starts at a certain address. Memory is only actually allocated when your program tries to use it.

Further, memory is allocated in units called "pages". I believe MacOS X uses 4kb pages, so when your program writes to the end of the buffer, the OS gives you 4096 bytes there, while retaining the rest of the buffer as simply a "your program wants this memory" note.

As for why you're hitting the limit at 64 terabytes, it's because current x86-64 processors use 48-bit addressing. This gives 256 TB of address space, which is split evenly between the operating system and your program. Doubling the 64 TB allocation would exactly fit in your program's 128 TB half of the address space, except that the program is already taking up a little bit of it.