numpy array to list conversion issue

Question

For some reason, evalRow(list(array([0, 1, 0, 0, 0]))) and evalRow([0, 1, 0, 0, 0]) give different results. However if I use magicConvert (here to debug this) instead of list to go from numpy array to list it works as expected:

def magicConvert(a):
  ss = str(list(a))[1:-1]
  return map(int, ss.split(","))

# You don't actually need to read these functions, just here to reproduce the error:
from itertools import *
def evalRow(r):
  grouped = map(
    lambda (v, l): (v, len(tuple(l))),
    groupby(chain([2], r, [2])))
  result = 0
  for player in (1, -1):
    for (pre, mid, post) in allTuples(grouped, 3):
      if mid[0] == player:
        result += player * streakScore(mid[1], (pre[0] == 0) + (post[0] == 0))
  return result

def streakScore(size, blanks):
  return 0 if blanks == 0 else (
    100 ** (size - 1) * (1 if blanks == 1 else 10))

def allTuples(l, size):
  return map(lambda i: l[i : i + size], xrange(len(l) - size + 1))

Answer 1

The difference in the behaviour is due to the fact that doing list(some_array) returns a list of numpy.int64, while, doing the conversion via the string representation (or equivalently using the tolist() method) returns a list of python's ints:

In [21]: import numpy as np

In [22]: ar = np.array([1,2,3])

In [23]: list(ar)
Out[23]: [1, 2, 3]

In [24]: type(list(ar)[0])
Out[24]: numpy.int64

In [25]: type(ar.tolist()[0])
Out[25]: builtins.int

I believe the culprit is the 100 ** (size - 1) part of your code:

In [26]: 100 ** (np.int64(50) - 1)
Out[26]: 0

In [27]: 100 ** (50 - 1)
Out[27]: 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

In [28]: type(100 ** (np.int64(50) - 1))
Out[28]: numpy.int64

What you see is the int64 overflowing, hence the result of the exponentiation are essentially "random", while python's ints have unlimited range and give the correct result.

To summary:

• If you want to convert between numpy and python data types use the proper methods, in this case array.tolist()
• Remember that numpys data types have limited range, hence you should check for overflows and expect strange results in other situations. If you do not use the proper methods for conversion you might end up using numpy data types when you didn't expect (as in this case).
• Never assume it's a bug in python/numpy/a very widely used library. The chances to find a bug in such trivial cases in such well-tested and widely used softwares is really small. If the program gives unexpected results, 99.999% of the times it's because you are doing something wrong. So, before blaming on others try to check step by step what your program is doing.