what's the difference between unicode(self) and self.__unicode__() in a Python Class?

Question

while handling unicode problem, I found that unicode(self) and self.__unicode__() have different behaviour:

#-*- coding:utf-8 -*-
import sys
import dis
class test():
    def __unicode__(self):
        s = u'中文'
        return s.encode('utf-8')

    def __str__(self):
        return self.__unicode__()
print dis.dis(test)
a = test()
print a

the above code works okay, but if I change self.__unicode__() to unicode(self), it will show error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)

the code with problem is:

#-*- coding:utf-8 -*-
import sys
import dis
class test():
    def __unicode__(self):
        s = u'中文'
        return s.encode('utf-8')

    def __str__(self):
        return unicode(self)
print dis.dis(test)
a = test()
print a

very curious about how python handle this, I tried dis module but didn't see too many difference:

Disassembly of __str__:
 12           0 LOAD_FAST                0 (self)
              3 LOAD_ATTR                0 (__unicode__)
              6 CALL_FUNCTION            0
              9 RETURN_VALUE   

VS

Disassembly of __str__:
 10           0 LOAD_GLOBAL              0 (unicode)
              3 LOAD_FAST                0 (self)
              6 CALL_FUNCTION            1
              9 RETURN_VALUE       

Answer 1

You return bytes from your __unicode__ method.

To make it clear:

In [18]: class Test(object):
    def __unicode__(self):
        return u'äö↓'.encode('utf-8')
    def __str__(self):
        return unicode(self)
   ....:     

In [19]: class Test2(object):
    def __unicode__(self):
        return u'äö↓'
    def __str__(self):
        return unicode(self)
   ....:     

In [20]: t = Test()

In [21]: t.__str__()
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
/home/dav1d/<ipython-input-21-e2650f29e6ea> in <module>()
----> 1 t.__str__()

/home/dav1d/<ipython-input-18-8bc639cbc442> in __str__(self)
      3         return u'äö↓'.encode('utf-8')
      4     def __str__(self):
----> 5         return unicode(self)
      6 

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

In [22]: unicode(t)
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
/home/dav1d/<ipython-input-22-716c041af66e> in <module>()
----> 1 unicode(t)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

In [23]: t2 = Test2()

In [24]: t2.__str__()
Out[24]: u'\xe4\xf6\u2193'

In [25]: str(_) # _ = last result
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
/home/dav1d/<ipython-input-25-3a1a0b74e31d> in <module>()
----> 1 str(_) # _ = last result

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)'

In [26]: unicode(t2)
Out[26]: u'\xe4\xf6\u2193'

In [27]: class Test3(object):
def __unicode__(self):
    return u'äö↓'
def __str__(self):
    return unicode(self).encode('utf-8')
....:     

In [28]: t3 = Test3()

In [29]: t3.__unicode__()
Out[29]: u'\xe4\xf6\u2193'

In [30]: t3.__str__()
Out[30]: '\xc3\xa4\xc3\xb6\xe2\x86\x93'

In [31]: print t3
äö↓

In [32]: print unicode(t3)
äö↓

print a or in my case print t will call t.__str__ which es expected to return bytes you let it return unicode so it tries to encode it with ascii which will not work.

Easy fix: let __unicode__ return unicode and __str__ bytes.