Force StandardOutputEncoding to UTF8

Question

I'm looking to parse UTF8 characters from the standard output stream of another application in my C# project. Using the default approach, characters outside of the ANSI spectrum are corrupted when read from the process' standard output stream.

Now according to Microsoft, what I need to do is set the StandardOutputEncoding:

If the value of the StandardOutputEncoding property is Nothing, the process uses the default standard output encoding for the standard output. The StandardOutputEncoding property must be set before the process is started. Setting this property does not guarantee that the process will use the specified encoding. The application should be tested to determine which encodings the process supports.

However, try as I might to set StandardOutputEncoding to UTF8/CP65001 the output as read, when dumped to a binary file, shows the same castration of foreign language characters. They are always read as '?' (aka 0x3F) instead of what they're supposed to be.

I know the assumption at this point would be that the application whose output I'm parsing is simply not sending UTF8 output, but this is definitely not the case as when I attempt to dump the output of the application to a file from the commandline after forcing the codepage of the commandprompt to 65001, everything looks fine.

chcp 65001 && slave.exe > file.txt

By this, I know for a fact that the application slave.txt is capable of spitting out UTF8-encoded standard output, but try as I might, I'm unable to get StandardOutputEncoding to do the same in my C# application.

Each and every time I end up dealing with encoding in .NET, I wish I were back in the C++ world were everything required more work but was so much more transparent. I'm contemplating writing a C application to read the output of slave.txt into a UTF8-encoded text file ready for C# parsing, but I'm holding off on that approach for now.

Answer 1

The only effect that StandardOutputEncoding has no impact whatsoever on the stdout of the executed application. The only thing it does is set the encoding of the StreamReader that sits on top of the binary stdout stream captured from the application being run.

This is OK for applications that will natively output UTF8 or Unicode stdout, but most Microsoft utilities do not do so, and instead will only encode the results per the console's codepage. The codepage of the console is manually set with the WIN32 API SetConsoleOutputCP and SetConsoleCP, and needs to be manually forced to UTF8 if that's what you'd like to read. This needs to be done on the console the exe is being executed within, and to the best of my knowledge, cannot be done from the host's .NET environment.

As such, I have written a proxy application dubbed UtfRedirect, the source code of which I have published on GitHub under the terms of the MIT license, which is intended to be spawned in the .NET host, then told which exe to execute. It'll set the codepage for the console of the final slave exe, then run it and pipe the stdout back to the host.

Sample UtfRedirector invocation:

//At the time of creating the process:
_process = new Process
                {
                    StartInfo =
                        {
                            FileName = application,
                            Arguments = arguments,
                            RedirectStandardInput = true,
                            RedirectStandardOutput = true,
                            StandardOutputEncoding = Encoding.UTF8,
                            StandardErrorEncoding =  Encoding.UTF8,
                            UseShellExecute = false,
                        },
                };

_process.StartInfo.Arguments = "";
_process.StartInfo.FileName = "UtfRedirect.exe"

//At the time of running the process
_process.Start();

//Write the name of the final slave exe to the stdin of UtfRedirector in UTF8
var bytes = Encoding.UTF8.GetBytes(application);
_process.StandardInput.BaseStream.Write(bytes, 0, bytes.Length);
_process.StandardInput.WriteLine();

//Write the arguments to be sent to the final slave exe to the stdin of UtfRedirector in UTF8
bytes = Encoding.UTF8.GetBytes(arguments);
_process.StandardInput.BaseStream.Write(bytes, 0, bytes.Length);
_process.StandardInput.WriteLine();

//Read the output that has been proxied with a forced codepage of UTF8
string utf8Output = _process.StandardOutput.ReadToEnd();

Answer 2

modern .NET option:

Console.OutputEncoding = System.Text.Encoding.UTF8;

Source