I'm trying to remove the first 37 lines from a very, very large file. I started trying sed and awk, but they seem to require copying the data to a new file. I'm looking for a "remove lines in place" method, that unlike sed -i
is not making copies of any kind, but rather is just removing lines from the existing file.
Here's what I've done...
awk 'NR > 37' file.xml > 'f2.xml'
sed -i '1,37d' file.xml
Both of these seem to do a full copy. Is there any other simple CLI that can do this quickly without a full document traversal?
Using the tac and the sed Commands However, if we can reverse the order of lines in the input file, the problem will turn into “remove first n lines from a file.” A straightforward sed one-liner sed '1,n d' can remove the top n lines. After that, if we reverse the lines again, our problem gets solved.
To Remove the lines from the source file itself, use the -i option with sed command. If you dont wish to delete the lines from the original source file you can redirect the output of the sed command to another file.
There's no simple way to do inplace editing using UNIX utilities, but here's one inplace file modification solution that you might be able to modify to work for you (courtesy of Robert Bonomi at https://groups.google.com/forum/#!topic/comp.unix.shell/5PRRZIP0v64):
bytes=$(head -37 "$file" |wc -c)
dd if="$file" bs="$bytes" skip=1 conv=notrunc of="$file"
The final file should be $bytes
bytes smaller than the original (since the goal was to remove $bytes
bytes from the beginning), so to finish we must remove the final $bytes
bytes. We're using conv=notrunc
above to make sure that the file doesn't get completely emptied rather than just truncated (see below for example). On a GNU system such as Linux doing the truncation afterwards can be accomplished by:
truncate -s "-$bytes" "$file"
For example to delete the first 5 lines from this 12-line file
$ wc -l file
12 file
$ cat file
When chapman billies leave the street,
And drouthy neibors, neibors, meet;
As market days are wearing late,
And folk begin to tak the gate,
While we sit bousing at the nappy,
An' getting fou and unco happy,
We think na on the lang Scots miles,
The mosses, waters, slaps and stiles,
That lie between us and our hame,
Where sits our sulky, sullen dame,
Gathering her brows like gathering storm,
Nursing her wrath to keep it warm.
First use dd
to remove the target 5 lines (really "$bytes" bytes) from the start of the file and copy the rest from the end to the front but leave the trailing "$bytes" bytes as-is:
$ bytes=$(head -5 file |wc -c)
$ dd if=file bs="$bytes" skip=1 conv=notrunc of=file
1+1 records in
1+1 records out
253 bytes copied, 0.0038458 s, 65.8 kB/s
$ wc -l file
12 file
$ cat file
An' getting fou and unco happy,
We think na on the lang Scots miles,
The mosses, waters, slaps and stiles,
That lie between us and our hame,
Where sits our sulky, sullen dame,
Gathering her brows like gathering storm,
Nursing her wrath to keep it warm.
s, waters, slaps and stiles,
That lie between us and our hame,
Where sits our sulky, sullen dame,
Gathering her brows like gathering storm,
Nursing her wrath to keep it warm.
and then use truncate
to remove those leftover bytes from the end:
$ truncate -s "-$bytes" "file"
$ wc -l file
7 file
$ cat file
An' getting fou and unco happy,
We think na on the lang Scots miles,
The mosses, waters, slaps and stiles,
That lie between us and our hame,
Where sits our sulky, sullen dame,
Gathering her brows like gathering storm,
Nursing her wrath to keep it warm.
If we had tried the above without dd ... conv=notrunc
:
$ wc -l file
12 file
$ bytes=$(head -5 file |wc -c)
$ dd if=file bs="$bytes" skip=1 of=file
dd: file: cannot skip to specified offset
0+0 records in
0+0 records out
0 bytes copied, 0.0042254 s, 0.0 kB/s
$ wc -l file
0 file
See the google groups thread I referenced for other suggestions and info.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With