fast producer / slow consumer in C

Question

I wrote a program in C that waits for an event, and then run an external system command by system() function.

while( true ){
    wait_for_event();
    system("cmd");
}

I have a seriuos problem on this, The cmd is a heavy command and takes several seconds to be completed, my app misses some events in this time frame.

So I decided to move the system function, which is very heavy, to another program, So I changed my program as follows:

while( true ){
    wait_for_event();
    write_to_fifo("cmd");
}

and wrote another program:

while(true){
    system(read_from_pipe());
}

but it doesn't help, because if producer (1st program) writes faster than consumer (2nd program), then consumer misses some data!

Is there any way to handle this problem?

Answer 1

You should return the code to its original form — that is, a single program calling a second program — except that you replace the system(3) call with a popen(3) call. Now the calling program can interleave calls for event checking with reading lines from the external program.

The Unix pipe mechanism ensures that a slow consumer will cause the fast producer to wait while writing when the pipe gets full.

You might also want to look into the fileno(3) function, combined with select(2) or poll(2) in order to make reading from the external program asynchronous, so that it can never block the calling program.

Answer 2

If all you need is the number of events, you can have a global counter. To avoid race conditions, you may need to use a semaphore instead. Of course you will need to have two threads.

Since your events contain important, a list (or array with sufficient number of slots) can be used to store the incoming data. You can use a mutex to protect this list.