Append to the top of a large file: bash

Question

I have a nearly 3 GB file that I would like to add two lines to the top of. Every time I try to manually add these lines, vim and vi freeze up on the save (I let them try to save for about 10 minutes each). I was hoping that there would be a way to just append to the top, in the same way you would append to the bottom of the file. The only things I have seen so far however include a temporary file, which I feel would be slow due to the file size. I was hoping something like:

grep -top lineIwant >> fileIwant

Does anyone know a good way to append to the top of the file?

Answer 1

Try

cat file_with_new_lines file > newfile

Answer 2

I did some benchmarking to compare using sed with in-place edit (as suggested here) to cat (as suggested here).

~3GB bigfile filled with dots:

$ head -n3 bigfile
................................................................................
................................................................................
................................................................................

$ du -b bigfile
3025635308      bigfile

File newlines with two lines to insert on top of bigfile:

$ cat newlines
some data
some other data

$ du -b newlines
26      newlines

---

Benchmark results using dumbbench v0.08:

cat:

$ dumbbench -- sh -c "cat newlines bigfile > bigfile.new"
cmd: Ran 21 iterations (0 outliers).
cmd: Rounded run time per iteration: 2.2107e+01 +/- 5.9e-02 (0.3%)

sed with redirection:

$ dumbbench -- sh -c "sed '1i some data\nsome other data' bigfile > bigfile.new"
cmd: Ran 23 iterations (3 outliers).
cmd: Rounded run time per iteration: 2.4714e+01 +/- 5.3e-02 (0.2%)

sed with in-place edit:

$ dumbbench -- sh -c "sed -i '1i some data\nsome other data' bigfile"
cmd: Ran 27 iterations (7 outliers).
cmd: Rounded run time per iteration: 4.464e+01 +/- 1.9e-01 (0.4%)

So sed seems to be way slower (80.6%) when doing in-place edit on large files, probably due to moving the intermediary temp file to the location of the original file afterwards. Using I/O redirection sed is only 11.8% slower than cat.

Based on these results I would use cat as suggested in this answer.

Answer 3

Try doing this :

using sed :

sed -i '1i NewLine' file

Or using ed :

ed -s file <<EOF
1i
NewLine
.
w
q
EOF