Why can I use a large buffer as a vector but not with new on Windows?

Question

I'm using 64 bit Windows 7 Pro and Visual Studio 2010 Pro.

I'm trying to allocate and use a buffer that is bigger than 4 GB (for high data rate data capture).

Allocating and writing the buffer as a vector of bytes works fine. Allocating the buffer as an array of bytes works fine, but writing to that array crashes quickly. (The last message printed is "buffer allocated".)

Commenting out the vector section does not fix the problem.

The following is my test program:

#include <iostream>
#include <vector>
#include <BaseTsd.h>

using namespace std;

int main() {
  const ULONG64 BUF_SIZE = 4 * 1024ULL * 1024ULL * 1024ULL;

  {
    vector<unsigned __int8> v(BUF_SIZE);
    cout << "vector allocated" << endl;
    for (ULONG64 i = 0; i < BUF_SIZE; ++i) {
      v[i] = 0xff;
    }
    cout << "vector written" << endl;
  }

  {
    unsigned __int8* buffer = new unsigned __int8[BUF_SIZE];
    cout << "buffer allocated" << endl;
    for (ULONG64 i = 0; i < BUF_SIZE; ++i) {
      buffer[i] = 0xff;
    }
    cout << "buffer written" << endl;
    delete[] buffer;
  }

  return 0;
}

UPDATE: I believe this is a compiler bug. See here: http://connect.microsoft.com/VisualStudio/feedback/details/553756/invalid-check-for-maximum-array-size-in-x64-compiler-c2148

Answer 1

I just tried compiling the given code with VS2010 Pro (64-bit version), and the compiler produced a C2148 error for the new call:

error C2148: total size of array must not exceed 0x7fffffff bytes

I compiled it from the command line after running vcvarsx86_amd64.bat. It seems that the limit given here is maybe somehow coming into play. Changing the new to [BUF_SIZE-1] allowed it to compile and run (although that is still larger than the 0x7fffffff number discussed in those links).