Edit the first and last line of a huge file

Question

I would like to edit the first and last line in a very huge file (~500GB). How can do so? For example, in the first line I have:

-flag </begin> 

and I would like to omit the "-flag". I tried using sed (as shown) to edit the first line but I didn't work:

sed -i '1s/-flag <begin>/<begin>/g' file.txt 

Answer 1

I can't think of a way you can do this in-place (I'd be interested to hear one!)

Hardly a one-liner but you could give this a try:

# substitute the first line and exit
sed '1s/-flag \(.*\)/\1/;q' file > new        
# add the rest of the file (probably quicker than sed)
tail -n +2 file >> new    
# cut off the last line of the file
truncate -s $(( $(stat -c "%s" new) - $(tail -n 1 new | wc -c) )) new
# substitute the last line                             
tail -n 1 file | sed 's/-flag \(.*\)/\1/' >> new

This assumes you have a couple of tools like truncate and that you can do arithmetic in your shell (my shell is bash).

The truncate -s removes the last line by taking the difference between the total file size stat -c "%s" and the length of the last line in bytes.

I'm not sure what you were trying to remove from the last line but I assumed that it was the same as the first (remove -flag from the start of the line).

Suggested modifications are welcome.

Answer 2

if you only want to get rid of -flag (note the trailing blank) in the first line:

sed -i '1s/-flag //' file

If you want to completely replace the content of the first line, you can issue

sed -i '1s/.*/new first line/' file

To do the same to the last line (I'm providing this as an example because you did not say what you want to do with the last line), you'd do

sed -i '$s/.*/new last line/' file