Why is there a built-in delay when reading with popen()?

Question

I'm executing a long-running (and often blocked) command via popen() : "ls -R /"

Problem: popen() reads into a buffer that you supply, and it seemingly attempts to populate the ENTIRE buffer before returning. This causes it to block quite often (if your buffer is large).

The solution would seem to be to make the underling fd non-blocking. When I do this, popen() still blocks, usually about 1 second each time. Why is this happening?

Here is my code. Make sure to compile with -std=c++11:

#include <cstdio>
#include <iostream>
#include <sys/time.h>
#include <unistd.h>
#include <fcntl.h>

static constexpr size_t SIZE = 65536;

struct Time
{
   friend std::ostream &operator<<(std::ostream &os, Time const &t)
   {
      (void)t;

      timeval tv;
      gettimeofday(&tv, nullptr);

      os << tv.tv_sec << "." << std::fixed << tv.tv_usec << " ";
      return os;
   }
};

int main()
{
   FILE *file;
   file = popen("ls -R /", "r");
   if(!file)
   {
      std::cerr << "Could not open app: " << errno;
      return -1;
   }

   // Make it non-blocking
   int fd = fileno(file);
   fcntl(fd, F_SETFL, O_NONBLOCK);

   char buffer[SIZE];
   Time t;
   while(true)
   {
      int rc = fread(buffer, 1, SIZE, file);
      if(rc <= 0)
      {
         if(EAGAIN == errno)
         {
            usleep(10);
            continue;
         }
         std::cerr << t << "Error reading: " << errno << std::endl;
         break;
      }
      std::cerr << t << "Read " << rc << std::endl;
   }
   pclose(file);
   return 0;
}

Output (notice that they are about 1 second apart, even though the fd is nonblocking and I only have a 10mS pause in the loop):

1429625100.983786 Read 4096
1429625101.745369 Read 4096
1429625102.426967 Read 4096
1429625103.185273 Read 4096
1429625103.834241 Read 4096
1429625104.512131 Read 4096
1429625105.188010 Read 4096
1429625105.942257 Read 4096
1429625106.642877 Read 4096

Answer 1

First, you should use read rather than fread. The stdio functions have their own layer of buffering beyond the OS's, so they can block even on non-blocking file descriptors. Use read to avoid this.

Second, you need to stop ls from buffering its output. The default behavior for programs that link to glibc is to use line buffering when stdout is connecting to a TTY, and full buffering when it is connected to a pipe or redirected to a file. Full buffering means the output is only flushed when the 4KB buffer fills up, rather than flushing every time a newline is output.

You can use stdbuf to override this behavior. Note that it will only work for programs that use C streams and link to glibc. That is most programs, but not all.

popen("stdbuf -oL ls -R /", "r");