Python: is the iteration of the multidimensional array super slow?

Question

I have to iterate all items in two-dimensional array of integers and change the value (according to some rule, not important).

I'm surprised how significant difference in performance is there between python runtime and C# or java runtime. Did I wrote totally wrong python code (v2.7.2)?

import numpy
a = numpy.ndarray((5000,5000), dtype = numpy.int32)
for x in numpy.nditer(a.T):
    x = 123

>python -m timeit -n 2 -r 2 -s "import numpy; a = numpy.ndarray((5000,5000), dtype=numpy.int32)" "for x in numpy.nditer(a.T):" "  x = 123"
2 loops, best of 2: 4.34 sec per loop

For example the C# code performs only 50ms, i.e. python is almost 100 times slower! (suppose the matrix variable is already initialized)

for (y = 0; y < 5000; y++)
for (x = 0; x < 5000; x++)
    matrix[y][x] = 123;

Answer 1

Yep! Iterating through numpy arrays in python is slow. (Slower than iterating through a python list, as well.)

Typically, you avoid iterating through them directly.

If you can give us an example of the rule you're changing things based on, there's a good chance that it's easy to vectorize.

As a toy example:

import numpy as np

x = np.linspace(0, 8*np.pi, 100)
y = np.cos(x)

x[y > 0] = 100

However, in many cases you have to iterate, either due to the algorithm (e.g. finite difference methods) or to lessen the memory cost of temporary arrays.

In that case, have a look at Cython, Weave, or something similar.