Need to calculate standard deviation from an array using bash and awk?

Question

Guys I'm new to awk and I'm struggling with awk command to find the standard deviation.

I have got the mean using the following:

echo ${GfieldList[@]} | awk 'NF {sum=0;for (i=1;i<=NF;i++)sum+=$i; print "Mean= " sum / NF; }'

Standard Deviation formula is:

sqrt((1/N)*(sum of (value - mean)^2))

I have found the mean using the above formula

Can you guys help me with the awk command for this one?

Answer 1

An alternate formula for the standard deviation is the square root of the quantity: (the mean square minus the square of the mean). This is used below:

$ echo 20 21 22 | awk 'NF {sum=0;ssq=0;for (i=1;i<=NF;i++){sum+=$i;ssq+=$i**2}; print "Std Dev=" (ssq/NF-(sum/NF)**2)**0.5}'
Std Dev=0.816497

Notes:

• In awk, NF is the number of "fields" on a line. In our case, every field is a number, so NF is the number of numbers on a given line.

• ssq is the sum of the squares of each number on the line. Thus, ssq/NF is the mean square.

• sum is the sum of the numbers on the line. Thus sum/NF is the mean and (sum/NF)**2 is the square of the mean.

• As per the formular, then, the standard deviation is (ssq/NF-(sum/NF)**2)**0.5.

The awk code

• NF

  This serves as a condition: the statements which follow will only be executed if the number of fields on this line, NF, evaluates to true, meaning non-zero. In other words, this condition will cause empty lines to be skipped.

• sum=0;ssq=0;

  This initializes sum and ssq to zero. This is only needed if there is more than one line of input.

• for (i=1;i<=NF;i++){sum+=$i;ssq+=$i**2}

  This puts the sum of all the numbers in sum and the sum of the square of the numbers in ssq.

• print "Std Dev=" (ssq/NF-(sum/NF)**2)**0.5

  This prints out the standard deviation.