C++ std::bad_alloc error

Question

I'm working on a C++ program (C++ 98). It reads a text file with lots of lines (10000 lines). These are tab separated values and then I parse it into Vector of Vector objects. However It seems to work for some files (Smaller) but One of my files gives me the following error (this file has 10000 lines and it's 90MB). I'm guessing this is a memory related issue? Can you please help me?

Error

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Abort

Code

void AppManager::go(string customerFile) {

    vector<vector<string> > vals = fileReader(customerFile);

    for (unsigned int i = 0; i < vals.size();i++){

        cout << "New One\n\n";

        for (unsigned int j = 0; j < vals[i].size(); j++){

            cout << vals[i][j] << endl;
        }

        cout << "End New One\n\n";
    }
}

vector<vector<string> > AppManager::fileReader(string fileName) {

    string line;
    vector<vector<string> > values;

    ifstream inputFile(fileName.c_str());

    if (inputFile.is_open()) {

        while (getline(inputFile,line)) {

            std::istringstream iss(line);
            std::string val;
            vector<string> tmp;

            while(std::getline(iss, val, '\t')) {

                tmp.push_back(val);
            }

            values.push_back(tmp);
        }

        inputFile.close();
    }
    else {

        throw string("Error reading the file '" + fileName + "'");
    }

    return values;
}

Answer 1

There's nothing wrong with your code, you're simply running on a platform likely with small memory limits, likely an old compiler and likely an old C++ library. It all conspires against you. You'll have to micro-optimize :(

Here's what you can do, starting with lowest hanging fruit first:

1. Do a dry run through the file, just counting the lines. Then values.resize(numberOfLines) , seek to the beginning and only then read the values. Of course you won't be using values.push_back anymore, merely values[lineNumber] = tmp. Resizing the values vector as you add to it may more than double the amount of memory your process needs on a temporary basis.

2. At the end of the line, do tmp.resize(tmp.size() - it'll shrink the vector to just fit the data.

3. You can reduce overheads in the existing code by storing all the values in one vector.

   1. If each line has a different number of elements, but you access them sequentially later on, you can store an empty string as an internal delimiter, it may have lower overhead than the vector.

   2. If each line has same number of values, then splitting them by lines adds unnecessary overhead - you know the index of the first value in each line, it's simply lineNumber * valuesPerLine, where first line has number 0.

4. Memory-map the file. Store the beginning and end of each word in a structure element of a vector, perhaps with a line number as well if you need it split up in lines.