Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Finding unusual value in an array, list

I have sales statistic data in array form to calc standard deviation or average from this data.

stats = [100, 98, 102, 100, 108, 23, 120] 

let said +-20% differential is normal situation, 23 is obviously a special case.

what's the best algorithm (in any language, pseudo or any principle) to find this unusual value?

like image 426
aifarfa Avatar asked May 09 '12 05:05

aifarfa


2 Answers

You could convert them to Z-scores and look for outliers.

>>> import numpy as np
>>> stats = [100, 98, 102, 100, 108, 23, 120]
>>> mean = np.mean(stats)
>>> std = np.std(stats)
>>> stats_z = [(s - mean)/std for s in stats]
>>> np.abs(stats_z) > 2
array([False, False, False, False, False,  True, False], dtype=bool)
like image 54
wim Avatar answered Sep 24 '22 22:09

wim


Compute the average and standard deviation. Treat any value more than X standard deviations from the average as "unusual" (where X will probably be somewhere around 2.5 to 3.0 or so).

There are quite a few variations of this theme. If you need something that's really statistically sound, you might want to look into some of them -- they can eliminate things like defending the arbitrary choice of (say) 2.7 standard deviations as the dividing line.

like image 30
Jerry Coffin Avatar answered Sep 24 '22 22:09

Jerry Coffin