Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Write large file

Tags:

c++

file-io

win64

I try to write to a large file, but it seems like it does not work for files larger than 2GB. I have tried with boost::iostreams::file_sink. Is this just a limit with the boost stream? Is there some other way I can write a large file on Win64 and win32?

like image 987
user38941 Avatar asked Nov 19 '08 14:11

user38941


People also ask

How do I open a 15gb text file?

To be able to open such large CSV files, you need to download and use a third-party application. If all you want is to view such files, then Large Text File Viewer is the best choice for you. For actually editing them, you can try a feature-rich text editor like Emacs, or go for a premium tool like CSV Explorer.

Which is the best method to write large amount of data to a file?

Using FileChannel Next, we will cover an example of using Java FileChannels to transfer a very large amount of data from one file to other. Here, we are using a buffer of (4 * 1024) size. From the output it is clear that, this is so far the fastest and most memory efficient way of processing large files.


1 Answers

This depends on:

  • The file system which you're using. Old file systems like FAT allow only files up to 2GB. Use NTFS.
  • Your library must be compiled to allow large files (64bit seek offsets). If they use 32bit ints to seek in file (check the arguments and results of the calls "seek" and "tell"), you can only ever access 2GB (2^31bits, the 32th is the sign +/-)

This might also help: http://www.boost.org/doc/libs/1_37_0/libs/iostreams/doc/faq.html#offsets

like image 147
Aaron Digulla Avatar answered Sep 26 '22 00:09

Aaron Digulla