No input from ProcessBuilder's InputStream until external process is closed

Question

Whilst building an wrapper for an console application I came across this weird issue where the Input Stream connected to the output (stdout) of the external process is completely blank until the external process exits.

My code as below:

import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;

public class Example{

    public static void main(String args[]) throws IOException{
        File executable = ...

        ProcessBuilder pb = new ProcessBuilder(executable.getCanonicalPath());

        pb.redirectErrorStream(true);

        Process p = pb.start();

        BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream(), Charset.forName("UTF-8")));

        String line;

        while((line = br.readLine()) != null){
            System.out.println(line);
        }
    }
}

I've tried several variants of reading from the input stream and all resulted in the same behavior.

I've tried:

CharBuffer charBuf = CharBuffer.allocate(1000);

InputStreamReader isr = new InputStreamReader(p.getInputStream(), Charset.forName("UTF-8"));

while(isr.read(charBuf) != -1){
    System.out.print(charBuf.flip().toString());
}

and

byte[] buf = new byte[1000];
int r;

while((r = p.getInputStream().read(buf)) != -1){
    System.out.print(new String(buf, 0, r));
}

all to no avail.

Somewhere along the line the output from the external process is being buffered (indefinitely) and I can't really figure out where. Loading the process from the command line seems to work fine where I see output coming out instantaneously. The strangest part is where the fact that the termination of the external process results in a flood of all the "buffered" output at once (a lot of stuff).

Unfortunately I don't have access to the source of the external process but given that it writes to stdout fine when in a console shouldn't really make a difference there (as far as I know).

Any ideas are welcome.

Edit:

One answer recommended me to rewrite the reader for the output and error streams to run on a separate thread. My actual implementation is doing that! And yet the problem still exists. The code posted above is a SSCCE of my actual code condensed for readability purposes, the actual code involves a separate thread for reading from the InputStream.

Edit 2:

User FoggyDay seems to have provided the answer which defines how the behavior of output buffering change when outputting between console and non-consoles. Whilst processes which detect that they are writing to a console use line buffering (buffered flushed every new line), writing to non-consoles (everything that it detects to not be a console) may be fully buffered (to a size of something like 8K). If I make the external process spam (8K of lorem ipsum in a for loop) output does indeed appear. I guess my question now is how to make my java program trigger line buffering on the external process.

Answer 1

To your question "how to make my java program trigger line buffering on the external process":

On Linux you can use the "stdbuf" program (coreutils package): stdbuf -oL your_program program_args

You only need to change stdout since stderr is unbuffered by default. The man page of setlinebuf gives additional background information if you're interested: http://linux.die.net/man/3/setlinebuf