Pipe vs. Temporary File

Question

Is there a big performance difference between:

• Process A writing to a temp file, and process B reading that file
• Process A writing to a pipe, and process B reading from that pipe

I'm curious to know what the answer is for both Windows and *nix.

EDIT: I should have asked: Does the buffer cache eliminate the difference between a temp file and a pipe?

Answer 1

One big difference is that with the pipe, processes A and B can be running concurrently, so that B gets to work on the output from A before A has finished producing it. Further, the size of the pipe is limited, so A won't be able to produce vastly more data than B has consumed; it will be made to wait for B to catch up.

If the volume of data is big, then writing to the temporary file involves disk activity, even if only for creating and then destroying the file. The data might well stay in the in-memory buffer pools - so no disk I/O there - even for surprisingly large files. Writing to the pipe 'never' involves writing to disk.

Answer 2

The big difference is that the first method actually uses on-disk storage, whereas a pipe will use memory (unless you get really pedantic and start thinking about swap space).

Performance-wise, memory is faster than disk (almost always). This should be generally true for all operating systems.

The only time when using a temp file really makes sense is if process B has to examine the data in multiple passes (like certain kinds of video encoding). For this use, the whole data stream would need to be buffered and if there were enough data yes it would probably negate the in-memory advantage. So for multi-pass (seek-bound) operations, go with a temp file.