C++ reading random lines of txt?

Question

I am running C++ code where I need to import data from txt file. The text file contains 10,000 lines. Each line contains n columns of binary data.

The code has to loop 100,000 times, each time it has to randomly select a line out of the txt file and assign the binary values in the columns to some variables.

What is the most efficient way to write this code? should I load the file first into the memory or should I randomly open a random line number?

How can I implement this in C++?

Answer 1

To randomly access a line in a text file, all lines need to have the same byte-length. If you don't have that, you need to loop until you get at the correct line. Since this will be pretty slow for so much access, better just load it into a std::vector of std::strings, each entry being one line (this is easily done with std::getline). Or since you want to assign values from the different columns, you can use a std::vector with your own struct like

struct MyValues{
  double d;
  int i;
  // whatever you have / need
};

std::vector<MyValues> vec;

Which might be better instead of parsing the line all the time.

With the std::vector, you get your random access and only have to loop once through the whole file.

Answer 2

10K lines is a pretty small file. If you have, say, 100 chars per line, it will use the HUGE amount of 1MB of your RAM.

Load it to a vector and access it the way you want.