What is the equivalent of unbuffer program on Windows?

Question

Hi according to this post, unbuffer connects to a command via a pseudo-terminal (pty), which makes the system treat it as an interactive process, therefore not using any stdout buffering.

I would like to use this function on Windows. May I know what is the equivalent of unbuffer program on Windows? Thanks.

Answer 1

The behaviour you're describing is typical of applications using run-time libraries for I/O. By default, most runtime libraries check to see whether the handle is a character mode device such as a console, and if so, they don't do any buffering. (Ideally the run-time library would treat a pipe in the same way as a console, but it seems that most don't.)

I'm not aware of any sensible way to trick such an application into thinking it is writing to a console when it is actually writing to a pipe.

Addendum: seven years later, Windows finally supports pseudoconsoles. If you are running on Windows 10 v1809 or later, this new API should solve your problem.

On older versions of Windows, if you know how much data to expect, you could in principle use the console API functions to create a console for the application to write to, and then read the output from the console. But you can't do that from Java, you would need to write a C application to do it for you.

Similarly, in principle it should presumably be possible to write a device driver equivalent to a Unix pseudo-terminal, one that acts like a pipe but reports itself to be a character-mode device. But writing device drivers requires specific expertise, and they have to be digitally signed, so unless there is an existing product out there this approach isn't likely to be feasible.

Answer 2

I spent some time on this and succeeded. I found this blog during research, and decided to return and provide my solution to save the next guy some time. I'm responding as a guest with a false email so I won't be interacting, but no further information should be required.

On Jul 18 '12 at 19:41 Harry Johnston wrote:

"In principle, if you know how much data to expect, you could use the console API functions to create a console for the application to write to, and then read the output from the console. But you can't do that from Java, you would need to write a C application to do it for you."

Thing is, there is already a utility that does this. It's written for a slightly different use, but it can be coxed into providing the desired result. Its intended purpose is to enable a windows console app to interact with a Linux style tty terminal. It does this by running a hidden console and accesses the console buffer directly. If you tried to use it – you'd fail. I got lucky and discovered that there are undocumented switches for this utility which will allow it to provide simple unbuffered output. Without the switches it fails with the error – the output is not a tty – when trying to pipe output.

The utility is called winpty. You can get it here:

https://github.com/rprichard/winpty/releases

The undocumented switches are mentioned here:

https://github.com/rprichard/winpty/issues/103

I’m using the MSYS2 version. You’ll need the msys-2.0.dll to use it.

Simply run:

winpty.exe -Xallow-non-tty -Xplain your_program.exe | receive_unbuffered_output.exe

-Xallow-non-tty , will allow piped output

-Xplain , will remove the added Linux terminal escape codes (or whatever they’re called)

Required files are:

winpty.exe
winpty-agent.exe
winpty.dll
msys-2.0.dll

winpty-debugserver.exe – Not needed

Answer 3

Disclaimer: My answer only deals with executables compiled using MSVC.

The buffering policy is coded inside Microsoft C Runtime (CRT) Library. You can learn the details here. This article suggests using console handles and manipulate console buffers to receive unbuffered output.

However, there's an undocumented feature inside Microsoft C Runtime to inherit file handles with some internal flags directly from its parent process using lpReserved2 and cbReserved2 fields of STARTUPINFO structure. You can find the details in the crt source code provided by Microsoft Visual Studio. Or search for something like posfhnd on GitHub.

We can exploit this undocumented feature to provide a pipe handle and specify FOPEN | FDEV flags to the child process, to fool the child process treat that pipe handle the same way as a FILE_TYPE_CHAR handle.

I have a working Python3 script to demonstrate this method.