How do I tell Python that sys.argv is in Unicode?

Question

Here is a little program:

import sys

f = sys.argv[1]
print type(f)
print u"f=%s" % (f)

Here is my running of the program:

$ python x.py 'Recent/רשימת משתתפים.LNK'
<type 'str'>
Traceback (most recent call last):
  File "x.py", line 5, in <module>
    print u"f=%s" % (f)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 7: ordinal not in range(128)
$ 

The problem is that sys.argv[1] is thinking that it's getting an ascii string, which it can't convert to Unicode. But I'm using a Mac with a full Unicode-aware Terminal, so x.py is actually getting a Unicode string. How do I tell Python that sys.argv[] is Unicode and not Ascii? Failing that, how do I convert ASCII (that has unicode inside it) into Unicode? The obvious conversions don't work.

Answer 1

The UnicodeDecodeError error you see is due to you're mixing the Unicode string u"f=%s" and the sys.argv[1] bytestring:

• both bytestrings:

    $ python2 -c'import sys; print "f=%s" % (sys.argv[1],)' 'Recent/רשימת משתתפים'
  

  This passes bytes transparently from/to your terminal. It works for any encoding.

• both Unicode:

    $ python2 -c'import sys; print u"f=%s" % (sys.argv[1].decode("utf-8"),)' 'Rec..
  

  Here you should replace 'utf-8' by the encoding your terminal uses. You might use sys.getfilesystemencoding() here if the terminal is not Unicode-aware.

Both commands produce the same output:

f=Recent/רשימת משתתפים

In general you should convert bytestrings that you consider to be text to Unicode as soon as possible.