Why does PowerShell stderr redirection mess up unicode characters?

Question

Why does redirecting stderr to stdout, when launching an external program from PowerShell, turn unicode characters into their code point representation? For example:

PS > py -3.13 -c "import logging; logging.error('\u27a4')" 2>&1
ERROR:root:\u27a4

whereas without the redirection:

PS > py -3.13 -c "import logging; logging.error('\u27a4')"
ERROR:root:➤

This is a minimal example - in my case the reason for wanting to redirect is that I then wish to pipe the entire output of this program (a python script which logs a bunch of things) through Select-String -NotMatch -CaseSensitive -Pattern "exclude_this|and_this" in order to omit certain lines - and if I don't redirect stderr to stdout then the python logging (which is on stderr) is not handled by Select-String & thus exclusions are not applied.

System info: Windows 11 (25H2) and PowerShell version:

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.5.4
PSEdition                      Core
GitCommitId                    7.5.4
OS                             Microsoft Windows 10.0.26200
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Answer 1

This is to do with how PowerShell can either redirect the stdio handles of the new process or just let it write directly to the console. Whenever you do redirection, capture the output to a variable, or pipe it to something else, PowerShell will start the new process with the stdout, and stderr in this case, to a pipe it controls. This is important because the underlying Windows APIs that are used when writing to the console vs a pipe is different. A console accepts a string value so there are no encoding problems, whereas a pipe needs that string to be encoded to bytes and it will use the console codepage to do this conversion. It is this conversion that is causing the problem as the default console codepage of most en-US systems is 431 437 and that codepage cannot represent the Unicode character U+27A4. As it cannot represent it, the Python's codec that encodes the string when writing to the console instead will "escape" it showing the \u27a4 value that you see.

What you can do is ensure that the console output encoding is set to UTF-8 and that Python is set to encode any console output as UTF-8. You control this with [Console]::OutputEncoding but it is good to keep all of them aligned to avoid any weirdness.

[Console]::OutputEncoding = [Console]::InputEncoding = $OutputEncoding = [Text.UTF8Encoding]::new()

# Use "-X utf8" to get Python to write to pipes using UTF-8 by default
py -3.13 -X utf8 -c "import logging; logging.error('\u27a4')"

# Otherwise you can set PYTHONIOENCODING to do the same thing
$env:PYTHONIOENCODING = 'utf8'
py -3.13 -c "import logging; logging.error('\u27a4')"

PowerShell 7.6 is also introducing a new variable $PSApplicationOutputEncoding that can replace [Console]::OutputEncoding but work in a scope so you don't have to reset it back. Unfortunately it isn't documented yet but there is a PR to add it https://github.com/MicrosoftDocs/PowerShell-Docs/pull/10876.