How to remove X bytes from the end of a large file without reading the whole file?

Question

In Linux, I have a rather large file with some extraneous information tacked on to the end of it. Let's say for example I know there are 314 bytes of extraneous data at the end of a 1.6GB file.

Of course it is very easy and efficient to add more data to the end of a file, but what can I do to remove it without having to copy the first portion of that file into another (or overwrite said file)?

Edit

I'm seeing some good advice on doing this in C. I was hoping to script it from the commandline, but failing that I would be more inclined to doing it in python than C.

I see that python has a truncate method on its file object but it seems to be demolishing my file no matter how i use it--I should be able to figure this one out, but of course answers are more than welcome still.

Answer 1

use the function truncate

http://linux.die.net/man/2/truncate

int truncate(const char *path, off_t length);
int ftruncate(int fd, off_t length); 

truncate takes the file name
ftruncate takes an open file descriptor

both of these set the file length to length so it either truncates or elongates (in the latter case, the rest of the file will be filled with NULL/ZERO)

[edit]
truncate (linux shell command) will work also

**SYNTAX**

truncate -s integer <filename>  
**OPTIONS**

-s number specify the new file length. If the new length is smaller than the current filelength data is lost. If the new length is greater the file is padded with 0. You can specify a magnitude character to ease large numbers:
b or B size is bytes.
k size is 1000 bytes.
K size is 1024 bytes.
m size is 10^6 bytes.
M size is 1024^2 bytes.
g size is 10^9 bytes.
G size is 1024^3 bytes.


**EXAMPLES**

To shrink a file to 10 bytes:

truncate -s 10 /tmp/foo

To enlarge or shrink a file to 345 Megabytes:

truncate -s 345M /tmp/foo

[/edit]

Answer 2

Although there were plenty of references to the truncate function in this thread, no one really answered the OP's question about reducing a file by a fixed amount from a scripting environment. Kevin's answer used truncate to resize the target file to a fixed amount, but of course the correctness of this solution requires the user to first know the size of the target file minus the extraneous data at the end. So, we have:

   -s, --size=SIZE
          set or adjust the file size by SIZE bytes

Truncate actually supports removing data from the end of a target file directly by prefixing SIZE with a - character.

For example, to resize a file by 314 bytes you can do:

truncate --size=-314 target_file.bin

Answer 3

Using "truncate" is best way, I just post some examples:

1. I have a file "PNav-h.db", it has 50176 bytes.

   -rw-r--r--  1 user user 50176 Mar  8 23:43 PNav-h.db
    $truncate -s 1000 PNav-h.db
   

   it set the file size to 1000 byptes

    -rw-r--r--  1 user user  1000 Mar  9 00:02 PNav-h.db
   

2. For your case, use $truncate --size=xxx xxfilename, using -<size number> to reduce the file size

   $truncate --size=-300 PNav-h.db
   -rw-r--r--  1 user user   700 Mar  9 00:07 PNav-h.db
   

   final file size = 1000-300=700

3. using +<size number> to increase the file size

   $truncate --size=+500 PNav-h.db
   -rw-r--r--  1 user user  1200 Mar  9 00:09 PNav-h.db
   

   final file size = 700 + 500 = 1200

4. if there is no - or +, it is to set the file size.

   $truncate --size=60000 PNav-h.db
   -rw-r--r--  1 user user 60000 Mar  9 00:12 PNav-h.db 
   

   final file size set to 60000

Answer 4

what about the truncate command?

http://linux.about.com/library/cmd/blcmdl2_truncate.htm