Numpy-flipped image + cv2.filter2D = assertion failed?

Question

I'm trying to use OpenCV's filter2D() for convolution. In my algorithm I need to flip kernel before passing it to the function. My first attempt was to use Numpy's fliplr() and flipud() methods:

def test_np():
    im = np.random.uniform(size=(32, 32))
    k = np.ones((3, 3), dtype=np.float32) / 9.
    k = np.fliplr(np.flipud(k))                # Numpy-flipped kernel
    return cv2.filter2D(im, -1, k)

Surprisingly, it gave me an assertion error during filtering:

OpenCV Error: Assertion failed (src.dims <= 2 && esz <= (size_t)32) in transpose, file /build/buildd/opencv-2.4.2+dfsg/modules/core/src/matrix.cpp, line 1877 terminate called after throwing an instance of 'cv::Exception' what(): /build/buildd/opencv-2.4.2+dfsg/modules/core/src/matrix.cpp:1877: error: (-215) src.dims <= 2 && esz <= (size_t)32 in function transpose

However, if I change flipping method to OpenCV's flip():

def test_cv2():
    im = np.random.uniform(size=(32, 32))
    k = np.ones((3, 3), dtype=np.float32) / 9.
    k = cv2.flip(k, -1)                        # OpenCV-flipped kernel
    return cv2.filter2D(im, -1, k)

filter2D() works without any problem.

I checked results of np.fliplr(np.flipud(...)) and cv2.flip(...) and they are the same:

k_np = np.fliplr(np.flipud(k))
k_cv2 = cv2.flip(k, -1)
(k_np == k_cv2).all()    # gives True

So we have 2 arrays that look the same but behave differently.

I'm curious, what is the difference between an array flipped with Numpy and the one flipped with OpenCV? Also, should I expect similar problems with other functions?

Answer 1

I think below explanation is the reason for this misbehavior.

Short Explanation:

When you flip with numpy functions, just strides of the ndarray is changed, not the whole array, ie it just create a view with different strides. But when you flip with OpenCV function, the whole array is reshaped. So when you apply filter2D() function, it calls transpose() internally. And initially Python wrappers of OpenCV couldn't perform translation of arrays with negative strides to Mat structure the way it was intended.

So possible solution was to copy the array manually using copy() method.

Later, this solution is solved, so you can use later versions of opencv. (I use OpenCV 3.x, compiled from OpenCV master branch, and it works fine)

Detailed Explanation:

First understand the structure of Numpy ndarray

Numpy array modifies its strides to achieve many operations like flip, transpose etc. It has a great advantage, No need of copying the array, which improves the performance. So these functions doesn't create copies, but just a view. These created views may not be continuous array, but copying always creates continuous array.

But OpenCV always creates copies of the array. So in these cases, OpenCV functions may be slower than the Numpy functions because numpy just edit the strides value, not array. You can check it as follows with following transpose function:

In [39]: x = np.ones((3,3),dtype=np.float32)

In [40]: x.strides
Out[40]: (12, 4)

In [43]: x.flags
Out[43]: 
  C_CONTIGUOUS : True    # Original array is continuous
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

Now try with Numpy transpose

In [41]: y = np.transpose(x)

In [42]: y.strides
Out[42]: (4, 12)         # transpose just change the strides

In [44]: y.flags
Out[44]: 
  C_CONTIGUOUS : False   # tranposed array is not continuous in numpy
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

Now try with OpenCV transpose

In [45]: z = cv2.transpose(x)

In [46]: np.all(z==y)   # result of numpy and OpenCV are same
Out[46]: True

In [47]: z.strides      # Check the strides
Out[47]: (12, 4)

In [48]: z.flags
Out[48]: 
  C_CONTIGUOUS : True   # OpenCV result is continuous.
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

Finally try with numpy tranpose with manual copying

In [53]: q = np.transpose(x).copy()

In [55]: np.all(z==q)
Out[55]: True

In [56]: q.strides     # Strides is same as OpenCV function
Out[56]: (12, 4)

In [57]: q.flags       
Out[57]: 
  C_CONTIGUOUS : True  # result is continuous also
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

And the performance comparison

In [49]: %timeit y = np.transpose(x)
1000000 loops, best of 3: 701 ns per loop

In [50]: %timeit y = np.transpose(x).copy()
1000000 loops, best of 3: 1.48 us per loop

In [51]: %timeit y = cv2.transpose(x)
1000000 loops, best of 3: 1.04 us per loop

Similarly flipping with numpy functions will create negative strides. But OpenCV functions won't create so.

In [58]: a = np.fliplr(x)

In [59]: a.strides
Out[59]: (12, -4)

In [60]: b = cv2.flip(x,-1)

In [61]: b.strides
Out[61]: (12, 4)

In earlier version of OpenCV, python wrappers couldn't translate a negative strided array to corresponding Mat structure. So possible solution was to make the array continuous with copy() method.

But in later versions of OpenCV, they added this support. It would check the strides and if it is negative, a copy of the array will be made by the python wrapper. So this is not an issue in later version of OpenCV.

I am using OpenCV 3, compiled from master branch of OpenCV. Let's check it:

In [62]: cv2.__version__
Out[62]: '3.0.0-dev'

In [63]: im = np.random.uniform(size=(32,32))

In [64]: k = np.ones((3,3), dtype=np.float32)/9.

In [65]: k = np.fliplr(np.flipud(k))

In [66]: z = cv2.filter2D(im, -1, k)

In [70]: print z[:5,:5]
[[ 0.65543429  0.53362787  0.45040413  0.52151458  0.61432061]
 [ 0.53666124  0.49690944  0.40779054  0.50330829  0.60923295]
 [ 0.39288601  0.42130001  0.41378173  0.5080897   0.58349994]
 [ 0.32685086  0.4340541   0.46039198  0.48272091  0.45093509]
 [ 0.25456175  0.40217766  0.4459138   0.49665956  0.4198618 ]]