Allocating a single object larger than 2GB using new in C++ (on Windows)

Question

I am on Windows, x64 mode. Compiled using MSVC on Visual Studio. The new operator seems not to be working as intended when I do this:

char* buf = new char[1LLU << 32];

But if I pass in a variable instead of directly typing the size, it works without a problem:

uint64_t sz = 1LLU << 32;
char* buf = new char[sz];

Looking at the assembly code, the compiler ignored the size I provided and simply xor ecx, ecx which I believe is just passing 0 to the new operator.

This is really puzzling! Is there some rule specified in the standard that I am not aware of?

p.s. char* buf = new char[1LLU << 31]; works without a problem. So I think its about the size of the integer I am using. But in documentation, new takes in a size_t as argument, which on x64 should be uint64_t.

VirtualAlloc()/HeapAlloc() are good alternatives that are safer to use than new in this context.

Full Code (note* try reproduce this locally in Visual Studio):

#include <iostream>
#include <cstdint>

int main()
{
    char* ptr = new char[1LLU << 32];
    memset(ptr, 0, sizeof(char) * (1LLU << 32)); //Access violation writing location...
}

#include <iostream>
#include <cstdint>

int main()
{
    uint64_t sz = 1LLU << 32;
    char* ptr = new char[sz];
    memset(ptr, 0, sizeof(char) * (1LLU << 32)); //no problem
}

Answer 1

Add const to your declaration:

const uint64_t sz = 1LLU << 32;

and you'll get the same error:

Compiler Error C2148

This is a 'safety' MSVC threshold for an array allocation, but since you give non-const size variable the compiler does not resolve it at compile-time.

Answer 2

You definitely found a compiler bug (targetting x64) and you should submit it to microsoft.

It seems that whenever the compiler knows the actual size to allocate an array to be greater_equal than 2^33 it will wrongfully 'optimize' the size to 0. Since allocating a 0-size array is perfectly valid ,you'll get invalid access when accessing it beyond some index (the allocated array will occupy some memory).

bug reported: https://developercommunity.visualstudio.com/content/problem/779749/msvc-2019-erroneously-replaces-known-arraysize-gre.html

BTW: std::array<> doesn't suffer from it ,it seems.