Accessing Large Files In C

Question

I need to access a file larger than 2gb using C. During one run of the program a variable number of bytes will be read from the file and the location of the next position saved. During the next run of the program the file position is read and a number of bytes are read starting from that location.

The complication is that occasionally the file may be 'compressed' by copying it to a new file, less any bytes that have already been read (I think copying is the only way to do that). The number of bytes removed in this way will also be saved.

I need to know the current position of the file from the original start for synchronizing with another file. This should be easy because it is just (current_offset + deleted_bytes).

The reason it is not easy is the fseek only uses long int indexes which limits the file to 2gb, and fsetpos uses an fpos_t structure for the position index, which is not a number and can't be converted back and forth to one. I don't know of any way to use a long long int index for file positioning, which would be the ideal solution.

What do I do?

Answer 1

On windows you can use _lseeki64() to do 64 bit seeks.

For compatibility with linux, you can also add -D_FILE_OFFSET_BITS=64 at compile time, and then do this in one of your headers:

#ifdef __MINGW32__ // or whatever you use to find out you're compiling on windows
#define lseek _lseeki64
#endif

then use lseek() everywhere as usual. This works because windows ignores the _FILE_OFFSET_BITS flag, and linux won't see the redefinition of lseek.

There's also _fseeki64() if you prefer a FILE* version, and there are equivalent 64 bit tell() and ftell() functions too (_telli64() and _ftelli64()).

Answer 2

Add the compilation flag -D_FILE_OFFSET_BITS=64, which makes fopen, fseek, off_t etc. become 64-bit and usable for manipulation of files greater than 2 GB in size. See Large File Support in Linux for more information.