ifstream:: What is the maximum file size that a ifstream can read

Question

I tried to read a 3GB data file using ifstream and it gives me wrong file size, whereas when I read a 600MB file, it gives me the correct result. In addition to wrong file size, I am also unable to read the entire file using ifstream.

Here is the code that I used

        std::wstring name;
        name.assign(fileName.begin(), fileName.end());
        __stat64 buf;
        if (_wstat64(name.c_str(), &buf) != 0)
            std::cout << -1; // error, could use errno to find out more

        std::cout << " Windows file size : " << buf.st_size << std::endl;;


        std::ifstream fs(fileName.c_str(), std::ifstream::in | std::ifstream::binary);
        fs.seekg(0, std::ios_base::end);

        std::cout << " ifstream  file size: " << fs.tellg() << std::endl;

The output for 3GB file was

 Windows file size : 3147046042
 ifstream  file size: -1147921254

Whereas the output for 600 MB file was

 Windows file size : 678761111
 ifstream  file size: 678761111

Just in case, I also tested for 5GB file and 300 MB file,

The output for 5GB file was

Windows file size : 5430386900
 ifstream  file size: 1135419604

The output for 300MB file was

Windows file size : 318763632
 ifstream  file size: 318763632

It looks to me like it is reaching some limit.

I am testing the code using Visual Studio 2010 on a Windows Machine which has plenty of memory and disk space.

I am trying to read some large files. Which is a good stream reader to use if ifstream can't read large files?

Answer 1

I think you want to say:

std::cout << " ifstream  file size: " << fs.tellg().seekpos() << std::endl;

At least that works correctly for a 6GB file I have laying around. But I'm compiling with Visual Studio 2012. And even your original code works fine too on that environment.

So I suspect that this is an bug in the std library on VS 2010 that got fixed in VS 2012. Whether it's a bug in the operator overloading for the pos_type or if that class isn't 64-bit aware is unknown. I'd have to install VS 2010 to validate, but that is likely the problem.

Answer 2

I modified your code slightly, to something that would compile:

#include <fstream>
#include <iostream>
#include <string>
#include <windows.h>

int main() { 

    std::wstring name(L"whatever.txt");

    __stat64 buf;
    if (_wstat64(name.c_str(), &buf) != 0)
        std::cout << -1; // error, could use errno to find out more

    std::cout << " Windows file size : " << buf.st_size << std::endl;;


    std::ifstream fs(name.c_str(), std::ifstream::in | std::ifstream::binary);
    fs.seekg(0, std::ios_base::end);

    std::cout << " ifstream  file size: " << fs.tellg() << std::endl;

    return 0;
}

I tried this on a ~3 Gigabyte file. With VS 2012 (either 32- or 64-bit) it produced:

 Windows file size : 3581853696
 ifstream  file size: 3581853696

With 32-bit VS 2008 (sorry, don't have a copy of VS 2010 installed right now) I got:

 Windows file size : 3581853696
 ifstream  file size: -713113600

So, it would appear that older versions of VS/VC++ used a 32-bit signed number for file sizes, so their practical limit for iostreams was probably 2 gigabytes. With VS 2012, that has apparently been corrected.