I want my Python script to be able to read Unicode command line arguments in Windows. But it appears that sys.argv is a string encoded in some local encoding, rather than Unicode. How can I read the command line in full Unicode?
Example code: argv.py
import sys first_arg = sys.argv[1] print first_arg print type(first_arg) print first_arg.encode("hex") print open(first_arg)
On my PC set up for Japanese code page, I get:
C:\temp>argv.py "PC・ソフト申請書08.09.24.doc" PC・ソフト申請書08.09.24.doc <type 'str'> 50438145835c83748367905c90bf8f9130382e30392e32342e646f63 <open file 'PC・ソフト申請書08.09.24.doc', mode 'r' at 0x00917D90>
That's Shift-JIS encoded I believe, and it "works" for that filename. But it breaks for filenames with characters that aren't in the Shift-JIS character set—the final "open" call fails:
C:\temp>argv.py Jörgen.txt Jorgen.txt <type 'str'> 4a6f7267656e2e747874 Traceback (most recent call last): File "C:\temp\argv.py", line 7, in <module> print open(first_arg) IOError: [Errno 2] No such file or directory: 'Jorgen.txt'
Note—I'm talking about Python 2.x, not Python 3.0. I've found that Python 3.0 gives sys.argv
as proper Unicode. But it's a bit early yet to transition to Python 3.0 (due to lack of 3rd party library support).
Update:
A few answers have said I should decode according to whatever the sys.argv
is encoded in. The problem with that is that it's not full Unicode, so some characters are not representable.
Here's the use case that gives me grief: I have enabled drag-and-drop of files onto .py files in Windows Explorer. I have file names with all sorts of characters, including some not in the system default code page. My Python script doesn't get the right Unicode filenames passed to it via sys.argv in all cases, when the characters aren't representable in the current code page encoding.
There is certainly some Windows API to read the command line with full Unicode (and Python 3.0 does it). I assume the Python 2.x interpreter is not using it.
1. Python 2 uses str type to store bytes and unicode type to store unicode code points. All strings by default are str type — which is bytes~ And Default encoding is ASCII.
In Python, the built-in functions chr() and ord() are used to convert between Unicode code points and characters. A character can also be represented by writing a hexadecimal Unicode code point with \x , \u , or \U in a string literal.
Windows' console has A LOT of support for Unicode — but it is not perfect (just “good enough”; see below). chcp 65001 is very dangerous. Unless a program was specially designed to work around defects in the Windows' API (or uses a C runtime library which has these workarounds), it would not work reliably.
Python's string type uses the Unicode Standard for representing characters, which lets Python programs work with all these different possible characters. Unicode (https://www.unicode.org/) is a specification that aims to list every character used by human languages and give each character its own unique code.
Here is a solution that is just what I'm looking for, making a call to the Windows GetCommandLineArgvW
function:
Get sys.argv with Unicode characters under Windows (from ActiveState)
But I've made several changes, to simplify its usage and better handle certain uses. Here is what I use:
win32_unicode_argv.py
""" win32_unicode_argv.py Importing this will replace sys.argv with a full Unicode form. Windows only. From this site, with adaptations: http://code.activestate.com/recipes/572200/ Usage: simply import this module into a script. sys.argv is changed to be a list of Unicode strings. """ import sys def win32_unicode_argv(): """Uses shell32.GetCommandLineArgvW to get sys.argv as a list of Unicode strings. Versions 2.x of Python don't support Unicode in sys.argv on Windows, with the underlying Windows API instead replacing multi-byte characters with '?'. """ from ctypes import POINTER, byref, cdll, c_int, windll from ctypes.wintypes import LPCWSTR, LPWSTR GetCommandLineW = cdll.kernel32.GetCommandLineW GetCommandLineW.argtypes = [] GetCommandLineW.restype = LPCWSTR CommandLineToArgvW = windll.shell32.CommandLineToArgvW CommandLineToArgvW.argtypes = [LPCWSTR, POINTER(c_int)] CommandLineToArgvW.restype = POINTER(LPWSTR) cmd = GetCommandLineW() argc = c_int(0) argv = CommandLineToArgvW(cmd, byref(argc)) if argc.value > 0: # Remove Python executable and commands if present start = argc.value - len(sys.argv) return [argv[i] for i in xrange(start, argc.value)] sys.argv = win32_unicode_argv()
Now, the way I use it is simply to do:
import sys import win32_unicode_argv
and from then on, sys.argv
is a list of Unicode strings. The Python optparse
module seems happy to parse it, which is great.
Dealing with encodings is very confusing.
I believe if your inputing data via the commandline it will encode the data as whatever your system encoding is and is not unicode. (Even copy/paste should do this)
So it should be correct to decode into unicode using the system encoding:
import sys first_arg = sys.argv[1] print first_arg print type(first_arg) first_arg_unicode = first_arg.decode(sys.getfilesystemencoding()) print first_arg_unicode print type(first_arg_unicode) f = codecs.open(first_arg_unicode, 'r', 'utf-8') unicode_text = f.read() print type(unicode_text) print unicode_text.encode(sys.getfilesystemencoding())
running the following Will output: Prompt> python myargv.py "PC・ソフト申請書08.09.24.txt"
PC・ソフト申請書08.09.24.txt <type 'str'> <type 'unicode'> PC・ソフト申請書08.09.24.txt <type 'unicode'> ?日本語
Where the "PC・ソフト申請書08.09.24.txt" contained the text, "日本語". (I encoded the file as utf8 using windows notepad, I'm a little stumped as to why there's a '?' in the begining when printing. Something to do with how notepad saves utf8?)
The strings 'decode' method or the unicode() builtin can be used to convert an encoding into unicode.
unicode_str = utf8_str.decode('utf8') unicode_str = unicode(utf8_str, 'utf8')
Also, if your dealing with encoded files you may want to use the codecs.open() function in place of the built-in open(). It allows you to define the encoding of the file, and will then use the given encoding to transparently decode the content to unicode.
So when you call content = codecs.open("myfile.txt", "r", "utf8").read()
content
will be in unicode.
codecs.open: http://docs.python.org/library/codecs.html?#codecs.open
If I'm miss-understanding something please let me know.
If you haven't already I recommend reading Joel's article on unicode and encoding: http://www.joelonsoftware.com/articles/Unicode.html
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With