Interprogram communication in python on Linux

Question

There's many examples on how to do these things to be found:

1) Communicate between diffrent processes in the same program.

2) Communicate between client/server over a network

However, this question has no good example anywhere I've looked:

• What is a canonical way to send a string from python program A to program B, which blocks and processes this string and then waits for another in a loop?

I feel like I've come close to an answer many times, but never managed to create a working example.

Additional implied requirements:

• Actually two diffrent programs: the example needs to actually have two diffrent programs (i.e. two files progA.py, progB.py that may be separately run from the commandline in two screens on the same machine), not use any kind of forking or multiprocess to create the client and server.
• Please suggest a way of doing this that allows variable length delimited strings up to a reasonable length to be sent, instead of having to get the exact byte count of the data size correct. (The latter is much more error-prone in implementation).
• Ideally do this without utilizing localhost internet connections

For example; when the reader uses:

pipein = open(pipe_name, 'r')
while program.KeepRunning:
    action = pipein.readline()[:-1]
    program.processLine(line)
    time.sleep(1)

and the writer uses:

command = "enable"
pipeout = os.open(pipe_name, os.O_WRONLY)
os.write(pipeout, command)
os.write(pipeout, "\n")

as is suggested at http://www.python-course.eu/pipes.php the reader gets stuck in an infinite loop reading out the empty string.

Interestingly, adding if(action == 'enable'): longFunction() to the program.processLine function results in part of the code in the longFunction being executed before getting stuck reading out empty lines forever.

On the other hand, all examples utilizing the more modern less low-level subprocess module only involve multi-threaded applications, not multiple applications. Other implementations involve sockets and networking.

While I've tried utilizing sockets, this results in the generic 'something went wrong'-type error with many possible causes Error 111: “connection refused” showing up 'some of the time'. As part of the python code that is executed upon receiving certain commands actually modifies the network config (e.g. it calls commands such as ip, tc and iptables with various arguments) utilizing a network connection to localhost is something that probably should be avoided, causing hard to debug and generally nasty issues. Besides the part where it's unneccessary as the second program runs on the same machine, so any interprogram communication should not need to use network interfaces.

Answer 1

That's intended behaviour. Take a look at this answer for similar problem and overview of the FIFO behaviour. The part relevant for your question is:

When there are no more writers (...) readers are notified about that through read() returning the EOF.

The file.readline() docs say that '' (empty string) means, that EOF is reached:

if f.readline() returns an empty string, the end of the file has been reached, while a blank line is represented by '\n', a string containing only a single newline.

That's it. In the infinite loop on each attempt to read you get an empty string, which says that there is no more writers connected.

There is nothing preventing you from using named pipes to solve your task. The simplest way is just sleep for some time, when there is no writers. Here is the working example:

# server.py

import os
import time

pipe_name = 'pipe_test'

if not os.path.exists(pipe_name):
    os.mkfifo(pipe_name)

with open(pipe_name, 'r') as pipe:
    print("Listening for actions...")
    while True:
        action = pipe.readline()[:-1]
        if action == '':
            print("No clients. Sleeping...")
            time.sleep(1)
        else:
            print("Action received:", repr(action))

# client.py

import os

pipe_name = 'pipe_test'

if not os.path.exists(pipe_name):
    os.mkfifo(pipe_name)

print("Waiting for server to start...")
with open(pipe_name, 'w') as pipe:
    action = input("Enter action to send: ")
    pipe.write(action + '\n')

Notes:

• There is no point in using os.open(), which is low-level function. You can use open() to interact with named pipes.
• Opening named pipe for reading blocks until first writer is connected. So you won't see Listening for actions... in the output, until first client is connected to pipe. The same goes with writer started without readers.
• You asked for "which blocks and processes this string and then waits for another in a loop". Unless you're processing the string in the separate thread it won't attempt to read next string until current one is processed.