Run out of ram C++

Question

I´m working with a 4D matrix (using STL vectors). Usually, the dimensions are different.For example, I´m reading a matrix whose dimensions are 192x256x128x96, and the following code complete with 0´s to the bigger dimension (256 in this case).

while(matriz.size() < width)   //width es el tamaño de N
{
    vector<vector<vector<short>>> aux;
    matriz.push_back(aux);
}

for(auto i = 0; i < matriz.size(); i++)
{
    while(matriz[i].size() < width)
    {
        vector<vector<short>> aux;
        matriz[i].push_back(aux);
    }
}


for(auto i = 0; i < matriz.size(); i++)
    for(auto j = 0; j < matriz[i].size(); j++)
        while(matriz[i][j].size() < width)
        {
            vector<short> aux;
            matriz[i][j].push_back(aux);
        }   
            


for(auto i = 0; i < matriz.size(); i++) 
    for(auto j = 0; j < matriz[i].size(); j++)
        for(auto k = 0; k < matriz[i][j].size(); k++)  
            while(matriz[i][j][k].size() < width)  
            {
                matriz[i][j][k].push_back(0);
            }
               

That code works with little-medium size 4D matrix, I´ve been tried with 200x200x200x200 and it really works, but I need to use it with a 256x256x256x256 matrix and when I run it my computer doesn´t respond.

I´m not sure if it is a RAM issue. My computer has 12GB RAM, and if I'm not mistaken, the size of the matrix is 8GB.

Any idea how to fix this ?

edit

If I let the program works, a time later it is killed

The memory usage with a 200x200x200x200 matrix is 56.7%

Answer 1

Let's see if I have this right.

You are producing:

• 1 vector that holds:
• 256 vectors that each hold
• 256 vectors that each hold (65,536 in total)
• 256 vectors that each hold (16,777,216 in total)
• 256 shorts (4,294,967,296 in total, or 8,589,934,592 Bytes as you indicated)

I don't know the entire size of each vector itself, but probably well under 1k, so you're using less than 10 gig of memory.

However, that's a LOT going on. Is it really hanging, or is it just taking a very, very long time.

Some debug output periodically would help answer that.

Answer 2

Some tips (from the comments):

1. Run an optimized build (-O3), this should speed up processing.

2. Instead of push_back() of an empty vector in a loop, use resize(). This will prevent costly reallocation.

   So for example, replace

    while(matriz.size() < width)   //width es el tamaño de N
    {
        vector<vector<vector<short>>> aux;
        matriz.push_back(aux);
    }
   

   With

    matriz.resize(width);
   

   If you do still need to use push_back() in a loop, at least reserve() the capacity beforehand. This can again prevent costly reallocations. Reallocating a vector can briefly double the amount of memory that it would normally use.

3. Use tools like top to watch memory and swap usage on the machine in real time. If you notice swap space used increasing, that means the machine is running out of memory.