Piping as interprocess communication

Question

I am interested in writing separate program modules that run as independent threads that I could hook together with pipes. The motivation would be that I could write and test each module completely independently, perhaps even write them in different languages, or run the different modules on different machines. There are a wide variety of possibilities here. I have used piping for a while, but I am unfamiliar with the nuances of its behaviour.

• It seems like the receiving end will block waiting for input, which I would expect, but will the sending end block sometimes waiting for someone to read from the stream?
• If I write an eof to the stream can I keep continue writing to that stream until I close it?
• Are there differences in the behaviour named and unnamed pipes?
• Does it matter which end of the pipe I open first with named pipes?
• Is the behaviour of pipes consistent between different Linux systems?
• Does the behaviour of the pipes depend on the shell I'm using or the way I've configured it?
• Are there any other questions I should be asking or issues I should be aware of if I want to use pipes in this way?

Answer 1

Wow, that's a lot of questions. Let's see if I can cover everything...

It seems like the receiving end will block waiting for input, which I would expect

You expect correctly an actual 'read' call will block until something is there. However, I believe there are some C functions that will allow you to 'peek' at what (and how much) is waiting in the pipe. Unfortunately, I don't remember if this blocks as well.

will the sending end block sometimes waiting for someone to read from the stream

No, sending should never block. Think of the ramifications if this were a pipe across the network to another computer. Would you want to wait (through possibly high latency) for the other computer to respond that it received it? Now this is a different case if the reader handle of the destination has been closed. In this case, you should have some error checking to handle that.

If I write an eof to the stream can I keep continue writing to that stream until I close it

I would think this depends on what language you're using and its implementation of pipes. In C, I'd say no. In a linux shell, I'd say yes. Someone else with more experience would have to answer that.

Are there differences in the behaviour named and unnamed pipes? As far as I know, yes. However, I don't have much experience with named vs unnamed. I believe the difference is:

• Single direction vs Bidirectional communication
• Reading AND writing to the "in" and "out" streams of a thread

Does it matter which end of the pipe I open first with named pipes?

Generally no, but you could run into problems on initialization trying to create and link the threads with each other. You'd need to have one main thread that creates all the sub-threads and syncs their respective pipes with each other.

Is the behaviour of pipes consistent between different linux systems?

Again, this depends on what language, but generally yes. Ever heard of POSIX? That's the standard (at least for linux, Windows does it's own thing).

Does the behaviour of the pipes depend on the shell I'm using or the way I've configured it?

This is getting into a little more of a gray area. The answer should be no since the shell should essentially be making system calls. However, everything up until that point is up for grabs.

Are there any other questions I should be asking

The questions you've asked shows that you have a decent understanding of the system. Keep researching and focus on what level you're going to be working on (shell, C, so on). You'll learn a lot more by just trying it though.