Argument types differ in normal and reflected operator overload (__sub__ / __rsub__)

Question

How do I get access to the properties of an numpy array after passing it through an righthand operator like __rsub__?

I wrote a very simple class in python that defines the two functions:

class test(object):
    def __sub__(self,  other):
        return other

    def __rsub__(self,  other):
        return other 

Basically they should do the same. The left-hand operator __sub__ works as expected, but it seems that the numpy array is stripped off its properties on the right-hand operator

from skimage import data
from skimage.color import rgb2gray
lena = data.lena()

grayLena = rgb2gray(lena)
t = test()

## overloaded - operator
left_hand = t - grayLena
print left_hand
# Output: 
#array([[ 0.60802863,  0.60802863,  0.60779059, ...,  0.64137412,
#    0.57998235,  0.46985725],
#  [ 0.60802863,  0.60802863,  0.60779059, ...,  0.64137412,
#    0.57998235,  0.46985725],
#  [ 0.60802863,  0.60802863,  0.60779059, ...,  0.64137412,
#    0.57998235,  0.46985725],
#  ..., 
#  [ 0.13746353,  0.13746353,  0.16881412, ...,  0.37271804,
#    0.35559529,  0.34377725],
#  [ 0.14617059,  0.14617059,  0.18730588, ...,  0.36788784,
#    0.37292549,  0.38467529],
#  [ 0.14617059,  0.14617059,  0.18730588, ...,  0.36788784,
#    0.37292549,  0.38467529]])

right_hand = grayLena - t
print right_hand
# Output:
# array([[0.6080286274509803, 0.6080286274509803, 0.6077905882352941, ...,
#   0.6413741176470589, 0.5799823529411765, 0.4698572549019608],
#  [0.6080286274509803, 0.6080286274509803, 0.6077905882352941, ...,
#   0.6413741176470589, 0.5799823529411765, 0.4698572549019608],
#  [0.6080286274509803, 0.6080286274509803, 0.6077905882352941, ...,
#   0.6413741176470589, 0.5799823529411765, 0.4698572549019608],
#  ..., 
#  [0.1374635294117647, 0.1374635294117647, 0.1688141176470588, ...,
#   0.3727180392156863, 0.35559529411764706, 0.34377725490196076],
#  [0.1461705882352941, 0.1461705882352941, 0.18730588235294118, ...,
#   0.3678878431372549, 0.37292549019607846, 0.3846752941176471],
#  [0.1461705882352941, 0.1461705882352941, 0.18730588235294118, ...,
#   0.3678878431372549, 0.37292549019607846, 0.3846752941176471]], dtype=object)

So the difference between both operations is that __rsub__ receives an array of dtype=object. If I would just set the dtype of this array, everything would work fine.

However, it works only with the return value, outside of __rsub__. Inside my __rsub__ I get only rubbish, that I cannot convert back, namely if I do

npArray = np.array(other,  dtype=type(other))

I get a 1D array of the type (floats in my case). But for some reason, the shape information is lost. Has anyone done this or an idea how I can access the original properties of the array (shape and type)?

Answer 1

I am not sure of what the exact control flow inside the ndarray's machinery is, but what's happening in your case is more or less clear:

What ndarray is delegating to your object's __rsub__ method is not the overall substraction operation, but the substraction of your object from each of the items in the array. And apparently when it has to delegate an operation to the object's methods, the return type is set to object regardless of what gets returned. You can check it with this slight modification of your code:

class test(object):
    def __sub__(self,  other):
        return other

    def __rsub__(self,  other):
        return other if other != 1 else 666

In [11]: t = test()

In [12]: t - np.arange(4)
Out[12]: array([0, 1, 2, 3])

In [13]: np.arange(4) - t
Out[13]: array([0, 666, 2, 3], dtype=object)

I don't think there is an easy way of overriding this behaviour. You could try making test a subclass of ndarray with a high __array_priority__ and abuse a little the __array_wrap__ method:

class test(np.ndarray):
    __array_priority__ = 100

    def __new__(cls):
        obj = np.int32([1]).view(cls)
        return obj

    def __array_wrap__(self, arr, context) :
        if context is not None :
            ufunc = context[0]
            args = context[1]
            if ufunc == np.subtract :
                if self is args[0] :
                    return args[1]
                elif self is args[1] :
                    return args[0]
        return arr

And now:

>>> t = test()
>>> np.arange(4) - t
array([0, 1, 2, 3])
>>> t - np.arange(4)
array([0, 1, 2, 3])

But:

>>> np.arange(4) + t
test([1, 2, 3, 4])
>>> t + np.arange(4)
test([1, 2, 3, 4])

It is a little bit wasteful, because we are doing the operation of adding the 1 inside t to every value in the array, and then silently discarding it, but I can't think of any way of overriding that.