Numpy __getitem__ delayed evaluation and a[-1:] not the same as a[slice(-1, None, none)]

Question

So this is two questions about what I'm assuming is the same basic underlying confusion on my part. I hope that's ok.

Here some code:

import numpy as np

class new_array(np.ndarray):

    def __new__(cls, array, foo):
        obj = array.view(cls)
        obj.foo = foo
        return obj

    def __array_finalize__(self, obj):
        print "__array_finalize"
        if obj is None: return
        self.foo = getattr(obj, 'foo', None)

    def __getitem__(self, key):
        print "__getitem__"
        print "key is %s"%repr(key)
        print "self.foo is %d, self.view(np.ndarray) is %s"%(
            self.foo,
            repr(self.view(np.ndarray))
            )
        self.foo += 1
        return super(new_array, self).__getitem__(key)

print "Block 1"
print "Object construction calls"
base_array = np.arange(20).reshape(4,5)
print "base_array is %s"%repr(base_array)
p = new_array(base_array, 0)
print "\n\n"

print "Block 2"
print "Call sequence for p[-1:] is:"
p[-1:]
print "p[-1].foo is %d\n\n"%p.foo

print "Block 3"
print "Call sequence for s = p[-1:] is:"
s = p[-1:]
print "p[-1].foo is now %d"%p.foo
print "s.foo is now %d"%s.foo
print "s.foo + p.foo = %d\n\n"%(s.foo + p.foo)

print "Block 4"
print "Doing q = s + s"
q = s + s
print "q.foo = %d\n\n"%q.foo

print "Block 5"
print "Printing s"
print repr(s)
print "p.foo is now %d"%p.foo
print "s.foo is now %d\n\n"%s.foo

print "Block 6"
print "Printing q"
print repr(q)
print "p.foo is now %d"%p.foo
print "s.foo is now %d"%s.foo
print "q.foo is now %d\n\n"%q.foo

print "Block 7"
print "Call sequence for p[-1]"
a = p[-1]
print "p[-1].foo is %d\n\n"%a.foo

print "Block 8"
print "Call sequence for p[slice(-1, None, None)] is:"
a = p[slice(-1, None, None)]
print "p[slice(None, -1, None)].foo is %d"%a.foo
print "p.foo is %d"%p.foo
print "s.foo + p.foo = %d\n\n"%(s.foo + p.foo)

The output of this code is

Block 1
Object construction calls
base_array is array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
__array_finalize



Block 2
Call sequence for p[-1:] is:
__array_finalize
p[-1].foo is 0


Block 3
Call sequence for s = p[-1:] is:
__array_finalize
p[-1].foo is now 0
s.foo is now 0
s.foo + p.foo = 0


Block 4
Doing q = s + s
__array_finalize
q.foo = 0


Block 5
Printing s
__getitem__
key is -1
self.foo is 0, self.view(np.ndarray) is array([[15, 16, 17, 18, 19]])
__array_finalize
__getitem__
key is -5
self.foo is 1, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
__getitem__
key is -4
self.foo is 2, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
__getitem__
key is -3
self.foo is 3, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
__getitem__
key is -2
self.foo is 4, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
__getitem__
key is -1
self.foo is 5, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
new_array([[15, 16, 17, 18, 19]])
p.foo is now 0
s.foo is now 1


Block 6
Printing q
__getitem__
key is -1
self.foo is 0, self.view(np.ndarray) is array([[30, 32, 34, 36, 38]])
__array_finalize
__getitem__
key is -5
self.foo is 1, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
__getitem__
key is -4
self.foo is 2, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
__getitem__
key is -3
self.foo is 3, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
__getitem__
key is -2
self.foo is 4, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
__getitem__
key is -1
self.foo is 5, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
new_array([[30, 32, 34, 36, 38]])
p.foo is now 0
s.foo is now 1
q.foo is now 1


Block 7
Call sequence for p[-1]
__getitem__
key is -1
self.foo is 0, self.view(np.ndarray) is array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
__array_finalize
p[-1].foo is 1


Block 8
Call sequence for p[slice(-1, None, None)] is:
__getitem__
key is slice(-1, None, None)
self.foo is 1, self.view(np.ndarray) is array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
__array_finalize
p[slice(None, -1, None)].foo is 2
p.foo is 2
s.foo + p.foo = 3

Please note two things:

1. The call to p[-1:] does not result in a call to new_array.__getitem__. This is true if p[-1:] is replaced by things like p[0:], p[0:-1], etc... but statements like p[-1] and p[slice(-1, None, None)] do result in a call to new_array.__getitem__. It's also true for statements like p[-1:] + p[-1:] or s = p[-1] but isn't true for statements like print s. You can see this by looking at "blocks" given above.

2. The variable foo is correctly updated during calls to new_array.__getitem__ (see blocks 5 and 6) but is not correct once evaluation of new_array.__getitem__ is complete (see, again, blocks 5 and 6). I should also add that replacing the line return super(new_array, self).__getitem__(key) with return new_array(np.array(self.view(np.ndarray)[key]), self.foo) does not work either. The following blocks are the only differences in output.

   Block 5
   Printing s
   __getitem__
   key is -1
   self.foo is 0, self.view(np.ndarray) is array([[15, 16, 17, 18, 19]])
   __array_finalize__
   __getitem__
   key is -5
   self.foo is 1, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   __getitem__
   key is -4
   self.foo is 2, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   __getitem__
   key is -3
   self.foo is 3, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   __getitem__
   key is -2
   self.foo is 4, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   __getitem__
   key is -1
   self.foo is 5, self.view(np.ndarray) is array([15, 16, 17, 18, 19])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   new_array([[15, 16, 17, 18, 19]])
   p.foo is now 0
   s.foo is now 1
   
   
   Block 6
   Printing q
   __getitem__
   key is -1
   self.foo is 0, self.view(np.ndarray) is array([[30, 32, 34, 36, 38]])
   __array_finalize__
   __getitem__
   key is -5
   self.foo is 1, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   __getitem__
   key is -4
   self.foo is 2, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   __getitem__
   key is -3
   self.foo is 3, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   __getitem__
   key is -2
   self.foo is 4, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   __getitem__
   key is -1
   self.foo is 5, self.view(np.ndarray) is array([30, 32, 34, 36, 38])
   __array_finalize__
   __array_finalize__
   __array_finalize__
   new_array([[30, 32, 34, 36, 38]])
   p.foo is now 0
   s.foo is now 1
   q.foo is now 1
   

   Which now contains excessive calls to new_array.__array_finalize__, but has no change in the "problem" with the variable foo.

3. It was my expectation that a call like p[-1:] to a new_array object with p.foo = 0 would result in this statement p.foo == 1 returning True. Clearly that isn't the case, even if foo was being correctly updated during calls to __getitem__, since a statement like p[-1:] results in a large number of calls to __getitem__ (once the delayed evaluation is taken into account). Moreover the calls p[-1:] and p[slice(-1, None, None)] would result in different values of foo (if the counting thing was working correctly). In the former case foo would have had 5 added to it, while in the later case foo would have had 1 added to it.

The Question

While the delayed evaluation of slices of numpy arrays isn't going to cause problems during evaluation of my code, it has been a huge pain for debugging some code of mine using pdb. Basically statements appear to evaluate differently at run time and in the pdb. I figure that this isn't good. That is how I stumbled across this behaviour.

My code uses the input to __getitem__ to evaluate what type of object should be returned. In some cases it returns a new instance of the same type, in other cases it returns a new instance of some other type and in still other cases it returns a numpy array, scalar or float (depending on whatever the underlying numpy array thinks is right). I use the key passed to __getitem__ to determine what the correct object to return is. But I can't do this if the user has passed a slice, e.g. something like p[-1:], since the method just gets individual indices, e.g. as if the user wrote p[4]. So how do I do this if the key in __getitem__ of my numpy subclass doesn't reflect if the user is requesting a slice, given by p[-1:], or just an entry, given by p[4]?

As a side point the numpy indexing documentation implies that slice objects, e.g. slice(start, stop, step) will be treated the same as statements like, start:stop:step. This makes me think that I'm missing something very basic. The sentence that implies this occurs very early:

Basic slicing occurs when obj is a slice object (constructed by start:stop:step notation inside of brackets), an integer, or a tuple of slice objects and integers.

I can't help but feel that this same basic mistake is also the reason why I think the self.foo += 1 line should be counting the number of times a user requests a slice, or an element of an instance of new_array (rather than the number of elements "in" a slice). Are these two issues actually related and if so how?

Answer 1

You've been bit by a nasty bug indeed. It's kind of a relief to know I am not the only one! Fortunately it is easy to solve. Just add something like the following to your class. This is actually a copy-paste from some code I wrote a few months back, the docstring sort of tells what is going on, but you may want to read the python docs as well.

def __getslice__(self, start, stop) :
    """This solves a subtle bug, where __getitem__ is not called, and all
    the dimensional checking not done, when a slice of only the first
    dimension is taken, e.g. a[1:3]. From the Python docs:
       Deprecated since version 2.0: Support slice objects as parameters
       to the __getitem__() method. (However, built-in types in CPython
       currently still implement __getslice__(). Therefore, you have to
       override it in derived classes when implementing slicing.)
    """
    return self.__getitem__(slice(start, stop))