Node.js read big file with fs.readFileSync

Question

I try to load big file (~6Gb) into memory with fs.readFileSync on the server with 96GB RAM.

The problem is it fails with the following error message

RangeError: Attempt to allocate Buffer larger than maximum size: 0x3fffffff bytes

Unfortunately I didn't find how it is possible to increase Buffer, it seems like it's a constant.

How I can overcome this problem and load a big file with Node.js?

Thank you!

Answer 1

I have also with same problem when try to load 6.4G video file to create file hash. I read whole file by fs.readFile() and it cause an error RangeError [ERR_FS_FILE_TOO_LARGE]. Then i use stream to do it:

let hash = crypto.createHash('md5'),
    stream = fs.createReadStream(file_path);

stream.on('data', _buff => { hash.update(_buff, 'utf8'); });
stream.on('end', () => { 
    const hashCheckSum = hash.digest('hex');
    // Save the hashCheckSum into database.
});

Hope it helped.

Answer 2

From a joyent FAQ:

What is the memory limit on a node process?

Currently, by default v8 has a memory limit of 512mb on 32-bit systems, and 1gb on 64-bit systems. The limit can be raised by setting --max_old_space_size to a maximum of ~1024 (~1 GiB) (32-bit) and ~1741 (~1.7GiB) (64-bit), but it is recommended that you split your single process into several workers if you are hitting memory limits.

If you show more detail about what's in the file and what you're doing with it, we can probably offer some ideas on how to work with it in chunks. If it's pure data, then you probably want to be using a database and let the database handle getting things from disk as needed and manage the memory.

Here's a fairly recent discussion of the issue: https://code.google.com/p/v8/issues/detail?id=847

And, here's a blog post that claims you can edit the V8 source code and rebuilt node to remove the memory limit. Try this at your own discretion.