How to dynamically allocate big memory , like 10 G ,using new operator in c++ on 64-linux?

Question

I need to dynamically allocate a larger float array for a special application using C++ new operator, like 10G. The code running on 64-ubuntu-14.04 Linux OS with 64G memory. When I set the request of memory as about 7G ,1879048192x4/(1024x1024x1024)=7G (float has 4 bytes), like this :

float * data; 
data = new float[1879048192];

The program works well, but when I try to increase the request to 10G , I got a what(): std::bad_alloc. I also try to use malloc() to take place of new operator:

data =(float*)malloc(1879048192*sizeof(float));

But obtain the same result. My ulimit -a is this:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514689
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 514689
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Someone may say there might be no 10G continual memory for the allocation, but I close all other progresses and the total memory is 64G. I want to know whether I can obtain this larger array or not, and how. Does the Linux limit the max number for this dynamically allocation? Where and how?

Answer 1

I don't see a problem when I try it. Both new and malloc worked. My system runs Ubuntu 15.04 and has 16G of ram.

However, if I try to use the memory, I found I needed to be very careful about the types of vars used to index into the data array.

For instance, the program below can do undesirable things with only long intand float, because a long int has a max value of 2^31 and the array length for 10Gi will be longer than 2^31. Also, floats only add 1.0 at a time to 16777216.0. With doubles, it is ok to use long int indexes here because the array is shorter.

use10G.c++

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv){
  long long int ten = 10;
  long long int megi = 1024*1024;
  long long int gigi = 1024*megi;
  long long int asize = (ten*gigi)/((long int) sizeof(double));
  double * data = new double[asize];
  long long int i=2;
  printf("A double is %zd bytes\n", (size_t) sizeof(double));
  printf("Array size is %lli \n", asize);
  data[0]=0.0;
  data[1]=1.0;
  while (i<asize) {
    data[i]=data[i-1]+1.0;
    ++i;
  }
  printf("%lf\n", (double) data[asize-1]);
  printf("success\n");
  exit(EXIT_SUCCESS);
}

A double is 8 bytes
Array size is 1342177280 
1342177279.000000
success