Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Constant time text file modification

Tags:

python

bash

I have a set of fairly large files (about 50 megabytes each, and at least a hundred of them), but I need to insert a small header (about 2 dozen lines) onto each one for processing purposes. I was hoping to write a script in either bash or python to do it, but I can't find a constant-time function that will let me insert at the front of a text file. If it is not constant time, I think it'll take too long to complete. Does anyone have experience with this issue?

like image 694
user2401982 Avatar asked Sep 06 '26 00:09

user2401982


1 Answers

Similar to Uwe's answer, but if your processing tool can only accept parameters as filenames, you can fake one up with mkfifo(1).

For example, in bash...

echo 'My header' > header.txt
echo 'My content' > content.txt
mkfifo fakefile.txt
cat header.txt content.txt > fakefile.txt &
cat fakefile.txt

...would stream the contents of the two files, rather than creating a new file.

like image 123
Aya Avatar answered Sep 07 '26 13:09

Aya



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!