Why do I get an overflow error multiplying Numpy product outputs?

Question

I've encountered an overflow warning as a result of multiplying the output of Numpy products that I'm looking to understand. A simplified version of their actual use within my larger project is detailed below:

import numpy as np

class MyClass(object):
    def __init__(self,
                 array_1,
                 array_2):

        # Assigning arrays to be used in later methods
        self.array_1 = array_1
        self.array_2 = array_2

        # Assigning some scaling factors to be used in later methods.
        self.value_1 = np.prod(self.array_1.shape)
        self.value_2 = np.prod(self.array_2.shape)
        print("Numpy Product Assignment: {0}, {1}".format(self.value_1, self.value_2))

        # Alternative assignment of scaling factors
        self.alt_value_1 = self.array_1.shape[0] * self.array_1.shape[1]
        self.alt_value_2 = self.array_2.shape[0] * self.array_2.shape[1]
        print("Regular Product Assignment: {0}, {1}".format(self.alt_value_1, self.alt_value_2))

        pass

    def mymethod(self):
        print("Direct Multiplication: {0}".format(80160 * 262144))
        print("Numpy Product Multiplication: {0}".format(self.value_1 * self.value_2))
        print("Regular Product Multiplcation {0}".format(self.alt_value_1 * self.alt_value_2))

if __name__ == '__main__':
    test_array_1 = np.zeros([512, 512], dtype=complex)
    test_array_2 = np.zeros([1002, 80], dtype=complex)

    test_class = MyClass(test_array_1, test_array_2)
    test_class.mymethod()

Including the use with the class structure for completeness, though heavily edited down to the bare minimum. If I run this code (on Python 3.6.0) I get the following output:

C:/somepath/scratch.py:247: RuntimeWarning: overflow encountered in long_scalars
  print("Numpy Product Multiplication: {0}".format(self.value_1 * self.value_2))
Numpy Product Assignment: 262144, 80160
Regular Product Assignment: 262144, 80160
Direct Multiplication: 21013463040
Numpy Product Multiplication: -461373440
Regular Product Multiplcation 21013463040

Process finished with exit code 0

Clearly I can get around the problem using the regular multiplication, but I would like to understand why there is a problem and if it can be fixed as is. I think there is some dtype=X subtlety that I have missed, so my question is what is causing these overflow errors?

Answer 1

This looks like an overflow caused by 32 integers. You can convert your values to 64 bit like this:

    self.value_1 = np.prod(self.array_1.shape, dtype=np.int64)
    self.value_2 = np.prod(self.array_2.shape, dtype=np.int64)

Numpy automatically selects a 32 bit integer type if the values used for array construction are small enough. During multiplication they are not automatically cast to 64 bit.