Using the same file for stdin and stdout with redirection

Question

I'm writing a application that acts like a filter: it reads input from a file (stdin), processes, and write output to another file (stdout). The input file is completely read before the application starts to write the output file.

Since I'm using stdin and stdout, I can run is like this:

$ ./myprog <file1.txt >file2.txt

It works fine, but if I try to use the same file as input and output (that is: read from a file, and write to the same file), like this:

$ ./myprog <file.txt >file.txt

it cleans file.txt before the program has the chance to read it.

Is there any way I can do something like this in a command line in Unix?

Answer 1

There's a sponge utility in moreutils package:

./myprog < file.txt | sponge file.txt

To quote the manual:

Sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constructing pipelines that read from and write to the same file.

Answer 2

The shell is what clobbers your output file, as it's preparing the output filehandles before executing your program. There's no way to make your program read the input before the shell clobbers the file in a single shell command line.

You need to use two commands, either moving or copying the file before reading it:

mv file.txt filecopy.txt
./myprog < filecopy.txt > file.txt

Or else outputting to a copy and then replacing the original:

./myprog < file.txt > filecopy.txt
mv filecopy.txt file.txt

If you can't do that, then you need to pass the filename to your program, which opens the file in read/write mode, and handles all the I/O internally.

./myprog file.txt                 # reads and writes according to its own rules