Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

balance numpy array with over-sampling

please help me finding a clean way to create a new array out of existing. it should be over-sampled, if the number of example of any class is smaller than the maximum number of examples in the class. samples should be taken from the original array (makes no difference, whether randomly or sequentially)

let's say, initial array is this:

[  2,  29,  30,   1]
[  5,  50,  46,   0]
[  1,   7,  89,   1]
[  0,  10,  92,   9]
[  4,  11,   8,   1]
[  3,  92,   1,   0]

the last column contains classes:

classes = [ 0,  1,  9]

the distribution of the classes is the following:

distrib = [2, 3, 1]

what i need is to create a new array with equal number of samples of all classes, taken randomly from the original array, e.g.

[  5,  50,  46,   0]
[  3,  92,   1,   0]
[  5,  50,  46,   0] # one example added
[  2,  29,  30,   1]
[  1,   7,  89,   1]
[  4,  11,   8,   1]
[  0,  10,  92,   9]
[  0,  10,  92,   9] # two examples
[  0,  10,  92,   9] # added
like image 758
funkifunki Avatar asked Apr 30 '14 15:04

funkifunki


People also ask

How much should I oversample by?

Choosing an oversampling rate 2x or more instructs the algorithm to upsample the incoming signal thereby temporarily raising the Nyquist frequency so there are fewer artifacts and reduced aliasing. Higher levels of oversampling results in less aliasing occurring in the audible range.

Does NumPy do lazy evaluation?

WeldNumpy is a Weld-enabled library that provides a subclass of NumPy's ndarray module, called weldarray, which supports automatic parallelization, lazy evaluation, and various other optimizations for data science workloads.

Is appending to NumPy array efficient?

Appending to numpy arrays is very inefficient. This is because the interpreter needs to find and assign memory for the entire array at every single step. Depending on the application, there are much better strategies. If you know the length in advance, it is best to pre-allocate the array using a function like np.

How do I arrange in NumPy ascending order?

Use numpy. sort() function to sort the elements of NumPy array in an ordered sequence. The parameter arr is mandatory. If you execute this function on a one-dimensional array, it will return a one-dimensional sorted array containing elements in ascending order.


2 Answers

You can use the imbalanced-learn package:

import numpy as np
from imblearn.over_sampling import RandomOverSampler

data = np.array([
    [  2,  29,  30,   1],
    [  5,  50,  46,   0],
    [  1,   7,  89,   1],
    [  0,  10,  92,   9],
    [  4,  11,   8,   1],
    [  3,  92,   1,   0]
])

ros = RandomOverSampler()

# fit_resample expects two arguments: a matrix of sample data and a vector of
# sample labels. In this case, the sample data is in the first three columns of 
# our array and the labels are in the last column
X_resampled, y_resampled = ros.fit_resample(data[:, :-1], data[:, -1])

# fit_resample returns a matrix of resampled data and a vector with the 
# corresponding labels. Combine them into a single matrix
resampled = np.column_stack((X_resampled, y_resampled))

print(resampled)

Output:

[[ 2 29 30  1]
 [ 5 50 46  0]
 [ 1  7 89  1]
 [ 0 10 92  9]
 [ 4 11  8  1]
 [ 3 92  1  0]
 [ 3 92  1  0]
 [ 0 10 92  9]
 [ 0 10 92  9]]

The RandomOverSampler offers different sampling strategies, but the default is to resample all classes except the majority class.

like image 199
ThisSuitIsBlackNot Avatar answered Oct 04 '22 03:10

ThisSuitIsBlackNot


The following code does what you are after:

a = np.array([[  2,  29,  30,   1],
              [  5,  50,  46,   0],
              [  1,   7,  89,   1],
              [  0,  10,  92,   9],
              [  4,  11,   8,   1],
              [  3,  92,   1,   0]])

unq, unq_idx = np.unique(a[:, -1], return_inverse=True)
unq_cnt = np.bincount(unq_idx)
cnt = np.max(unq_cnt)
out = np.empty((cnt*len(unq),) + a.shape[1:], a.dtype)
for j in xrange(len(unq)):
    indices = np.random.choice(np.where(unq_idx==j)[0], cnt)
    out[j*cnt:(j+1)*cnt] = a[indices]

>>> out
array([[ 5, 50, 46,  0],
       [ 5, 50, 46,  0],
       [ 5, 50, 46,  0],
       [ 1,  7, 89,  1],
       [ 4, 11,  8,  1],
       [ 2, 29, 30,  1],
       [ 0, 10, 92,  9],
       [ 0, 10, 92,  9],
       [ 0, 10, 92,  9]])

When numpy 1.9 is released, or if you compile from the development branch, then the first two lines can be condensed into:

unq, unq_idx, unq_cnt = np.unique(a[:, -1], return_inverse=True,
                                  return_counts=True)

Note that, the way np.random.choice works, there is no guarantee that all rows of the original array will be present in the output one, as the example above shows. If that is needed, you could do something like:

unq, unq_idx = np.unique(a[:, -1], return_inverse=True)
unq_cnt = np.bincount(unq_idx)
cnt = np.max(unq_cnt)
out = np.empty((cnt*len(unq) - len(a),) + a.shape[1:], a.dtype)
slices = np.concatenate(([0], np.cumsum(cnt - unq_cnt)))
for j in xrange(len(unq)):
    indices = np.random.choice(np.where(unq_idx==j)[0], cnt - unq_cnt[j])
    out[slices[j]:slices[j+1]] = a[indices]
out = np.vstack((a, out))

>>> out
array([[ 2, 29, 30,  1],
       [ 5, 50, 46,  0],
       [ 1,  7, 89,  1],
       [ 0, 10, 92,  9],
       [ 4, 11,  8,  1],
       [ 3, 92,  1,  0],
       [ 5, 50, 46,  0],
       [ 0, 10, 92,  9],
       [ 0, 10, 92,  9]])
like image 23
Jaime Avatar answered Oct 04 '22 04:10

Jaime