How to write files larger than 2 GB?

Question

The following code:

use std::fs::File;
use std::io::Write;

fn main() {
    let encoded: Vec<u8> = vec![0; 2500000000];
    let mut buffer = File::create("file.bin").unwrap();
    let written_bytes = buffer.write(&encoded).unwrap();
    assert_eq!(written_bytes, encoded.len());
}

errors with:

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `2147479552`,
 right: `2500000000`', src/main.rs:8:5

So there seems to be a limit of 2^31 - 4096 bytes.

How can I get around this? I'd like to write a larger file. :)

Answer 1

Rusts write relies on the underlying OS to write the bytes.

For Linux systems the write syscall will be used.

According to POSIX.1, if count is greater than SSIZE_MAX, the result is implementation-defined; see NOTES for the upper limit on Linux.

Notes:

On Linux, write() (and similar system calls) will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred. (This is true on both 32-bit and 64-bit systems.)

So there the magic number is coming from.

To circumvent your problem, use write_all instead of write, which will make sure, that all bytes are written.

As a note: If you run the program under Windows, the will run just fine.