Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to dynamically allocate big memory , like 10 G ,using new operator in c++ on 64-linux?

Tags:

c++

linux

memory

I need to dynamically allocate a larger float array for a special application using C++ new operator, like 10G. The code running on 64-ubuntu-14.04 Linux OS with 64G memory. When I set the request of memory as about 7G ,1879048192x4/(1024x1024x1024)=7G (float has 4 bytes), like this :

float * data; 
data = new float[1879048192];

The program works well, but when I try to increase the request to 10G , I got a what(): std::bad_alloc. I also try to use malloc() to take place of new operator:

data =(float*)malloc(1879048192*sizeof(float));

But obtain the same result. My ulimit -a is this:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514689
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 514689
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Someone may say there might be no 10G continual memory for the allocation, but I close all other progresses and the total memory is 64G. I want to know whether I can obtain this larger array or not, and how. Does the Linux limit the max number for this dynamically allocation? Where and how?

like image 686
changingivan Avatar asked May 11 '15 15:05

changingivan


People also ask

How can we dynamically allocate memory in C?

In C, dynamic memory is allocated from the heap using some standard library functions. The two key dynamic memory functions are malloc() and free(). The malloc() function takes a single parameter, which is the size of the requested memory area in bytes. It returns a pointer to the allocated memory.

Which operator is used for dynamic memory allocation in C?

C uses the malloc() and calloc() function to allocate memory dynamically at run time and uses a free() function to free dynamically allocated memory.

How do we dynamically allocate memory?

To allocate memory dynamically, library functions are malloc() , calloc() , realloc() and free() are used. These functions are defined in the <stdlib. h> header file.


1 Answers

I don't see a problem when I try it. Both new and malloc worked. My system runs Ubuntu 15.04 and has 16G of ram.

However, if I try to use the memory, I found I needed to be very careful about the types of vars used to index into the data array.

For instance, the program below can do undesirable things with only long intand float, because a long int has a max value of 2^31 and the array length for 10Gi will be longer than 2^31. Also, floats only add 1.0 at a time to 16777216.0. With doubles, it is ok to use long int indexes here because the array is shorter.

use10G.c++

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv){
  long long int ten = 10;
  long long int megi = 1024*1024;
  long long int gigi = 1024*megi;
  long long int asize = (ten*gigi)/((long int) sizeof(double));
  double * data = new double[asize];
  long long int i=2;
  printf("A double is %zd bytes\n", (size_t) sizeof(double));
  printf("Array size is %lli \n", asize);
  data[0]=0.0;
  data[1]=1.0;
  while (i<asize) {
    data[i]=data[i-1]+1.0;
    ++i;
  }
  printf("%lf\n", (double) data[asize-1]);
  printf("success\n");
  exit(EXIT_SUCCESS);
}

A double is 8 bytes
Array size is 1342177280 
1342177279.000000
success
like image 114
Paul Avatar answered Oct 16 '22 16:10

Paul