Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Print LF with Python 3 to Windows stdout

How to get \n printed to stdout on Windows? This code works in Python 2, but not with Python 3:

# set sys.stdout to binary mode on Windows
import sys, os, msvcrt
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

# the length of testfile created with
#     python test_py3k_lf_print.py > testfile
# below should be exactly 4 symbols (23 0A 23 0A)
print("#\n#")
like image 902
anatoly techtonik Avatar asked Jan 23 '16 07:01

anatoly techtonik


1 Answers

Python 3 already configures standard I/O in binary mode, but it has its own I/O implementation that does newline translation. Instead of using print, which requires a text-mode file, you could manually call sys.stdout.buffer.write to use the binary-mode BufferedWriter. If you need to use print, then you'll need a new text I/O wrapper that doesn't use universal newlines. For example:

stdout = open(sys.__stdout__.fileno(), 
              mode=sys.__stdout__.mode, 
              buffering=1, 
              encoding=sys.__stdout__.encoding, 
              errors=sys.__stdout__.errors, 
              newline='\n', 
              closefd=False)

Since closefd is false, closing this file won't close the original sys.stdout file descriptor. You can use this file explicitly via print("#\n#", file=stdout), or replace sys.stdout = stdout. The original is available as sys.__stdout__.

Background

Python 3's io module was designed to provide a cross-platform and cross-implementation (CPython, PyPy, IronPython, Jython) specification for all filelike objects in terms of the abstract base classes RawIOBase, BufferedIOBase, and TextIOBase. It includes a reference pure Python implementation in the _pyio module. The common denominator for the raw io.FileIO implementation is the set of low-level POSIX system calls such as read and write, which eliminates the problem of CRT stdio inconsistencies. On Windows, the POSIX layer is just the low I/O layer of the CRT, but at least that's limited to the quirks of a single platform.

One of the Windows quirks is having non-standard text and binary modes in its POSIX I/O layer. Python addresses this by always using binary mode and calling setmode on the stdio file descriptors 1.

Python can avoid using the Windows CRT for I/O by implementing a WinFileIO registered subclass of RawIOBase. There's a proposed patch for this in issue 12939. Another example is the win_unicode_console module, which implements WindowsConsoleRawReader and WindowsConsoleRawWriter classes.


1. This has caused problems for programs that embed Python and expect stdio to use the default text mode. For example, in binary mode printing wide-character strings no longer casts to char as it does in ANSI text mode, and it certainly doesn't print using WriteConsoleW as it would in UTF-16 text mode. For example:

Python 2.7.10 (default, May 23 2015, 09:44:00) 
[MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, os, msvcrt, ctypes 
>>> ctypes.cdll.msvcr90.wprintf(b'w\x00i\x00d\x00e\x00\n\x00') 
wide
5
>>> msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) 
16384
>>> ctypes.cdll.msvcr90.wprintf(b'w\x00i\x00d\x00e\x00\n\x00')
w i d e
 5

like image 53
Eryk Sun Avatar answered Oct 08 '22 19:10

Eryk Sun