Numpy mean 'inplace'

Question

I have a line of code that looks like this:

te_succ_rate = np.mean(np.argmax(test_y, axis=1) == self.predictor(test_x))

where test_y is a numpy array of arrays and self.predictor(test_x) returns a numpy array. The whole line of code returns the percentage of subarrays in test_y that has a max value equal to the value in the corresponding position in the array returned from self.predictor(test_x).

The problem is that for large sizes of test_y and test_x, it runs out of memory. It works fine for 10 000, but not 60 000.

Is there a way to avoid this?

I tried this:

tr_res = []
for start, end in zip(range(0, len(train_x), subsize), range(subsize, len(train_x), subsize)):
    tr_res.append(self.predictor(train_x[start:end]))
tr_res = np.asarray(tr_res)
tr_res = tr_res.flatten()
tr_succ_rate = np.mean(np.argmax(train_y, axis=1) == tr_res)

But it does not work as the result is somehow 0 (which is not correct).

Answer 1

Level 1:

Though this isn't an answer for doing it inline, it may still be an answer to your problem:

You sure you're running out of memory from the mean and not the argmax?

Each additional dimension in test_y will be storing an extra N number of whatever datatype you're working with. Say you have 5 dimensions in your data, you'll have to store 5N values (presumably floats). The results of your self.predictor(test_x) will take a 6th N of memory. The temporary array that is the answer to your conditional is a 7th N. I don't actually know what the memory usage of np.mean is, but I assume it's not another N. But for arguments sake, let's say it is. If you inline just np.mean, you'll only save up to an N of memory, while you already need 7N worth.

So alternatively, try pulling out your np.argmax(test_y, axis=1) into an intermediate variable in a previous step and don't reference test_y again after calculating the argmax so test_y gets garbage collected. (or do whatever python 3 does to force deletion of that variable) That should save you the number of dimensions of your data minus 1 N of memory usage. (you'll be down to around 3N or up to 4N memory usage, which is better than you could have achieved by in-lining just np.mean.

I made the assumption that running self.predictor(test_x) only takes 1N. If it takes more, then pulling that out into its own intermediate variable in the same way will also help.

Level 2:

If that still isn't enough, still pull out your np.argmax(test_y, axis=1) and the self.predictor(test_x) into their own variables, then iterate across the two arrays yourself and do the conditional and aggregation yourself. Something like:

sum = 0.
n = 0
correct_ans = np.argmax(test_y, axis=1)
returned_ans = self.predictor(test_x)
for c, r in zip(correct_ans, returned_ans):
    if c == r:
        sum += 1
    n += 1
avg = sum / n

(not sure if zip is the best way to do this. np probably has a more efficient way to do the same thing. This is the second thing you tried, but accumulating the aggregates without storing an additional array) That way, you'll also save the need to store the temporary list of booleans resulting from your conditional.

If that still isn't enough, you're going to have to fundamentally change how you're storing your actual and target results, since the issue becomes you not being able to fit just the target and results into memory.