Does bidirectional popen() work on Mac OS X in C?

Question

My Question Where can I find a clear working example of C code that sets up and uses a bidirectional pipe with popen() on MacOS?

I am trying to use popen() to open a bidirectional pipe between my C program and an external program on Mac OS X.

My C program has to repeatedly:

• read an input line from its own stdin
• reformat this as input for an external program
• call the external program, passing the formatted input into the external program's stdin
• read the results from the stdout of the external program
• process these results and produce some diagnostic output on its own stdout

I can do this easily if I use an actual file to store the input/output and then use system() or similar, but this is too slow, as I have billions of inputs. So I'd like to do it entirely in memory.

The relevant man page, and Internet discussion in this group and others show that I can open a pipe to the external program, then write to it with fprintf just like writing to a file.

pout = popen(external_program, "w")
fprintf(pout,input_to_external_program);

This works fine as a 1-directional pipe, feeding the formatted inputs INTO the external program.

Apparently it is possible on Mac OS to open a bidirectional pipe by specifying "r+" instead of "r" or "w" as the relevant argument to popen, and thereby read and write to the program using the same pipe.

pout = popen(external_program, "r+")
fprintf(pout,"%s", input_to_external_program);
fscanf(fpout,"%s", &output_from_external_program);

I do not need to alternate between reading, writing, reading, writing etc as all that needs to happen is that my program will finish writing the input to the external program before it starts reading the output from the external program.

The external program should just read one self-contained input sequence (i.e. it ends with a special line to indicate "end of input"), and then run to completion, writing output as it goes, then terminating.

But when I try to use this feature, it just doesn't work, because something blocks - the external program starts, but does not complete even one input.

Even when I just forget about doing any reading, but just replace "w" with "r+" then the previously-working functionality stops working.

I have searched for most of this morning for a working popen() example that I can modify, but all that I have found only use a unidirectional pipe.

So we end up back with

My Question *Where can I find a clear working example of C code that sets up and uses a bidirectional pipe with popen() on MacOS?**

If it makes it easier, I might be able to replace the C part with Perl.

UPDATE

Thanks for the answers, but I am still unable to make it work properly though I can make some small-scale versions work. For example, I wrote a little program that would just add one to any numbers fed to it, quitting if the input is -99/ I called this "copy".

#include <stdio.h>
#include <unistd.h>

int 
main(int argc, char **argv) {

  int k;

  setbuf(stdout,NULL);

  while (scanf("%d",&k) == 1) {
    if (k == -99) break;
    printf("%d\n",k+1);
  }

}

Then I wrote a parent program that will create the pipe, write some stuff to it, and then wait for the output (not doing NOHANG because I want the parent to keep sucking up the output of the child until the child is done).

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

int main(void) {

    FILE *p;
    int  k;

    p = popen("copy", "r+");
    if (!p) {
        puts("Can't connect to copy");
        return 1;
    }

    fprintf(p,"%d\n",100);
    fprintf(p,"%d\n",200);
    fprintf(p,"%d\n",300);
    fprintf(p,"%d\n",-99);

    fflush(p);

    while (fscanf(p,"%d",&k) == 1)  {
      printf("%d\n", k);
    }

    pclose(p);

    return 0;
 }

This works perfectly, but as soon as I replace the dummy program "copy" with my actual subprocess, it just hangs.

Everything should be the same - the parent process creates an entire input sequence for the child, which reads it, does some work, outputs some arbitrary number of lines of output and then terminates.

But something is different - either the child process is not receiving all of its input, or is buffering its output or something. Mysterious.

Answer 1

Here's a program that interacts with bc to perform integer calculations. In the simplest case, all you need to do is use fgets() and fputs() to read and write to/from the pipe.

However, if you enter something that produces no output, like setting a variable (e.g., x=1) or a runtime error (e.g., 1/0), then fgets() will continue waiting for input indefinitely. To avoid this situation, you have to make the pipe non-blocking and check it repeatedly for output from the piped process.

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

int main(void) {
    FILE *p;
    char buf[1025];
    int i, fd, delay;

    p = popen("bc", "r+");
    if (!p) {
        puts("Can't connect to bc");
        return 1;
    }
    /* Make the pipe non-blocking so we don't freeze if bc produces no output */
    fd = fileno(p);
    fcntl(fd, F_SETFL, fcntl(fd, F_GETFL, 0) | O_NONBLOCK);

    puts("Give me a sum to do (e.g., 2+2):");
    if (fgets(buf, 1024, stdin)) {
        fputs(buf, p);
        fflush(p);
        for (delay=1,i=0; i<20; delay*=2,i++) {
            usleep(delay);
            if (fgets(buf, 1024, p)) {
                printf("The answer is %s", buf);
                break;
            }
        }
    }
    pclose(p);

    return 0;
}

Also, don't forget that pipes are generally line buffered. This isn't a problem here because fgets() includes a newline character at the end of the input. Correction: pipes are block buffered, so call fflush() after sending data into the pipe.

Answer 2

Make sure that you fflush(pout) in the parent process and that the child process does fflush(stdout) after every output.