exec() not working with unicode characters

Question

Im trying to execute a .py program from within my python code, but non-ASCII characters behave oddly when printed and dealt with.

module1.py:

test = "áéíóúabcdefgçë"

print(test)

Main code:

exec(open("module1.py").read(), globals())

I want this to print áéíóúabcdefgçë but it instead prints áéíóúabcdefgçë. This happens with all non-ASCII characters i have tried.

I am using Python 3.7 and Windows 10.

Running module1.py individually does not produce this error, but i want to run the program using exec() or something else that has roughly the same function.

Answer 1

I found a way to fix the issue. Python's open is assuming some encoding other than UTF-8. Changing the main code to the following fixes the issue on my computer (python 3.7 and windows 10):

exec(open("module1.py", encoding="utf-8").read(),globals())

Thanks @jjramsey for additional information:

According to the Python documentation for open(), "The default encoding is platform dependent (whatever locale.getpreferredencoding() returns)."

For me, if I run the following check:

import locale
print(locale.getpreferredencoding())

I get cp1252, which is notably not UTF-8 and so open() will cause the issues we have seen in this question, unless we specify the encoding.