stdout redirection with ctypes

Question

I'm trying to redirect the output of printf functions to a file on Windows. I'm using ctypes with python3 to invoke the functions. My code is:

import os, sys
from ctypes import *

if __name__ == '__main__':

 print("begin")
 saved_stdout=os.dup(1)
 test_file=open("TEST.TXT", "w")
 os.dup2(test_file.fileno(), 1)
 test_file.close()
 print("python print")
 cdll.msvcrt.printf(b"Printf function 1\n")
 cdll.msvcrt.printf(b"Printf function 2\n")
 cdll.msvcrt.printf(b"Printf function 3\n")
 os.dup2(saved_stdout, 1)
 print("end")

But when I run the code from Eclipse I get the following on the screen:

begin
end
Printf function 1
Printf function 2
Printf function 3

...and the following in the TEST.txt

python print

When I run this from cmd, this is what is on the screen:

begin
end

..and this is in the TEST.txt:

python print

When I comment out the second dup2() statement e.g.

import os, sys
from ctypes import *
if __name__ == '__main__':

    print("begin")
    saved_stdout=os.dup(1)
    test_file=open("TEST.TXT", "w")
    os.dup2(test_file.fileno(), 1)
    test_file.close()
    print("python print")
    cdll.msvcrt.printf(b"Printf function 1\n")
    cdll.msvcrt.printf(b"Printf function 2\n")
    cdll.msvcrt.printf(b"Printf function 3\n")
    #os.dup2(saved_stdout, 1)
    print("end")

From Eclipse, on the screen:

begin

...and in the TEST.txt file:

python print
end
Printf function 1
Printf function 2
Printf function 3

From cmd, on the screen:

begin

...and in the TEST.txt file:

python print
end

I'm totally confused now. I read all the redirection threads here on StackOverflow and I can't understand what's going on. Anyway, what I've gathered is that C functions access the stdout that is bind directly to the file descriptor, while python uses a special object for that - stdout File Object. So the elementary sys.stdout=*something* doesn't work with ctypes. I've even tried os.fdopen(1) on the dup2-ed output and then calling flush() after every printf statement but this isn't working again. I'm totally out of ideas now and would appreciate if someone have a solution for this.

Answer 1

Use the same C runtime that CPython 3.x uses (e.g. msvcr100.dll for 3.3). Also include a call to fflush(NULL) before and after redirecting stdout. For good measure, redirect the Windows StandardOutput handle, in case a program uses the Windows API directly.

This can get complicated if the DLL uses a different C runtime, which has its own set of POSIX file descriptors. That said, it should be OK if it gets loaded after you've redirected Windows StandardOutput.

Edit:

I've modified the example to run in Python 3.5+. VC++ 14's new "Universal CRT" makes it much more difficult to use C standard I/O via ctypes.

import os
import sys
import ctypes, ctypes.util

kernel32 = ctypes.WinDLL('kernel32')

STD_OUTPUT_HANDLE = -11

if sys.version_info < (3, 5):
    libc = ctypes.CDLL(ctypes.util.find_library('c'))
else:
    if hasattr(sys, 'gettotalrefcount'): # debug build
        libc = ctypes.CDLL('ucrtbased')
    else:
        libc = ctypes.CDLL('api-ms-win-crt-stdio-l1-1-0')

    # VC 14.0 doesn't implement printf dynamically, just
    # __stdio_common_vfprintf. This take a va_array arglist,
    # which I won't implement, so I escape format specificiers.

    class _FILE(ctypes.Structure):
        """opaque C FILE type"""

    libc.__acrt_iob_func.restype = ctypes.POINTER(_FILE)    

    def _vprintf(format, arglist_ignored):
        options = ctypes.c_longlong(0) # no legacy behavior
        stdout = libc.__acrt_iob_func(1)
        format = format.replace(b'%%', b'\0')
        format = format.replace(b'%', b'%%')
        format = format.replace(b'\0', b'%%')
        arglist = locale = None        
        return libc.__stdio_common_vfprintf(
            options, stdout, format, locale, arglist)

    def _printf(format, *args):
        return _vprintf(format, args)

    libc.vprintf = _vprintf
    libc.printf = _printf

def do_print(label):
    print("%s: python print" % label)
    s = ("%s: libc _write\n" % label).encode('ascii')
    libc._write(1, s, len(s))
    s = ("%s: libc printf\n" % label).encode('ascii')
    libc.printf(s)
    libc.fflush(None) # flush all C streams

if __name__ == '__main__':
    # save POSIX stdout and Windows StandardOutput
    fd_stdout = os.dup(1)
    hStandardOutput = kernel32.GetStdHandle(STD_OUTPUT_HANDLE)

    do_print("begin")

    # redirect POSIX and Windows
    with open("TEST.TXT", "w") as test:
        os.dup2(test.fileno(), 1)
        kernel32.SetStdHandle(STD_OUTPUT_HANDLE, libc._get_osfhandle(1))

    do_print("redirected")

    # restore POSIX and Windows
    os.dup2(fd_stdout, 1)
    kernel32.SetStdHandle(STD_OUTPUT_HANDLE, hStandardOutput)

    do_print("end")