os.popen().read() - charmap decoding error

Question

I have already read UnicodeDecodeError: 'charmap' codec can't decode byte X in position Y: character maps to <undefined>. While the error message is similar, the code is completely different, because I use os.popen in this question, not open. I cannot use the answers from the other questions to solve this problem.

output = os.popen("dir").read()

This line, which is supposed to assign the output of command "dir" to variable "output", is causing this error:

'charmap' codec can't decode byte 0x88 in position 260: character maps to <undefined>

I think this might be happenning because some files in the folder contain letters such as ł, ą, ę and ć in their names. I have no idea how to fix this though.

Answer 1

os.popen is just a wrapper around subprocess.Popen along with a io.TextIOWrapper object:

The returned file object reads or writes text strings rather than bytes.

If Python's default encoding doesn't work for you, you should use subprocess.Popen directly.

The underlying issue is that cmd writes ansi garbage by default, even when the output is to a pipe. This behavior may depend on your Windows version.

You can fix this by passing /U flag to cmd:

p = subprocess.Popen('cmd /u /c dir', stdout=subprocess.PIPE)
result = p.communicate()
text = result[0].decode('u16')

Answer 2

In this case, using subprocess.Popen is too general, too verbose and too hard to remember. Use subprocess.check_output instead.

It returns a bytes object, which can be converted to str with decode function.

import subprocess
x = subprocess.check_output(['ls','/'])
print(x.decode('utf-8'))

Try it online!