Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how to efficiently read a binary file into a vector C++

I need to read a large binary file (~1GB) into a std::vector<double>. I'm currently using infile.read to copy the whole thing into a char * buffer (shown below) and I currently plan to convert the whole thing into doubles with reinterpret_cast. surely there must be a way to just put the doubles straight into the vector?

I'm also not sure about the format of the binary file, the data was produced in python so it's probably all floats

ifstream infile(filename, std--ifstream--binary);

infile.seekg(0, infile.end);     //N is the total number of doubles
N = infile.tellg();              
infile.seekg(0, infile.beg);

char * buffer = new char[N];

infile.read(buffer, N);
like image 365
Aaron Sear Avatar asked Feb 24 '15 22:02

Aaron Sear


People also ask

How do I read a binary file in C?

Use the fread Function to Read Binary File in C FILE* streams are retrieved by the fopen function, which takes the file path as the string constant and the mode to open them. The mode of the file specifies whether to open a file for reading, writing or appending.

Which method is used to read data from a binary file?

The BinaryReader class is used to read binary data from a file. A BinaryReader object is created by passing a FileStream object to its constructor.

Are binary files more efficient?

Binary files are more efficient because they use less disk space and because you do not need to convert data to and from a text representation when you store and retrieve data. A binary file can represent 256 values in 1 byte of disk space.


1 Answers

Assuming the entire file is double, otherwise this wont work properly.

std::vector<double> buf(N / sizeof(double));// reserve space for N/8 doubles
infile.read(reinterpret_cast<char*>(buf.data()), buf.size()*sizeof(double)); // or &buf[0] for C++98
like image 65
Tony J Avatar answered Sep 30 '22 19:09

Tony J