Average of multiple files in shell

Question

I want to calculate the average of 15 files:- ifile1.txt, ifile2.txt, ....., ifile15.txt. Number of columns and rows of each file are same. Part of the data looks as

ifile1.txt      ifile2.txt       ifile3.txt
3  5  2  2 .    1  2  1  3 .    4  3  4  1 .
1  4  2  1 .    1  3  0  2 .    5  3  1  5 .
4  6  5  2 .    2  5  5  1 .    3  4  3  1 .
5  5  7  1 .    0  0  1  1 .    4  3  4  0 .
.  .  .  . .    .  .  .  . .    .  .  .  . .  

I would like to find over a new file which will show the average of these 15 fils.

ofile.txt
2.66   3.33  2.33 2      . (i.e. average of 3 1 4, average of 5 2 3 and so on)
2.33   3.33  1    2.66   .
3      5     4.33 1.33   .
3      2.33  4    0.66   .
.      .     .    .      .

I was trying with following, but getting error

awk'{for (i=1; i<=NF; i++)} rows=FNR;cols=NF} END 
{for (i=1; i<=rows; i++){for (j=1; j<=cols; j++) 
s+=$i;print $0,s/NF;s=0}}' ifile* > ofile.txt

Answer 1

As written:

awk'{for (i=1; i<=NF; i++)} rows=FNR;cols=NF} END
…

you get 'command not found' as the error because you must leave a space between awk and the script inside the quotes. When you fix that, you start getting into problems because there are two } and only one { on the first line of the script.

When you get around to tackling the problem, you're going to need a 2D array, indexed by line number and column number, summing the values from the files. You'll also need to know the number of files processed, and the number of columns. You can then arrange to iterate over the 2D array in the END block.

awk 'FNR == 1 { nfiles++; ncols = NF }
     { for (i = 1; i < NF; i++) sum[FNR,i] += $i
       if (FNR > maxnr) maxnr = FNR
     }
     END {
         for (line = 1; line <= maxnr; line++)
         {
             for (col = 1; col < ncols; col++)
                  printf "  %f", sum[line,col]/nfiles;
             printf "\n"
         }
     }' ifile*.txt

Given the three data files from the question:

ifile1.txt

3 5 2 2
1 4 2 1
4 6 5 2
5 5 7 1

ifile2.txt

1 2 1 3
1 3 0 2
2 5 5 1
0 0 1 1

ifile3.txt

4 3 4 1
5 3 1 5
3 4 3 1
4 3 4 0

The script I showed produces:

  2.666667  3.333333  2.333333
  2.333333  3.333333  1.000000
  3.000000  5.000000  4.333333
  3.000000  2.666667  4.000000

If you want to control the number of decimal places to 2, then use %.2f in place of %f.

Answer 2

$ { head -n1 ifile1.txt; paste ifile*.txt;} | awk 'NR==1{d=NF; next;} {for (i=1;i<=d;i++) {s=0; for (j=i;j<=NF;j+=d) s+=$j; printf "%.2f%s",s/(NF/d),j==NF+d?"\n":"\t";}}'
2.67    3.33    2.33    2.00
2.33    3.33    1.00    2.67
3.00    5.00    4.33    1.33
3.00    2.67    4.00    0.67

This script computes each row and prints the results before moving on to the next row. Because of this, the script does not need to hold all the data in memory at once. This is important if the data files are large.

How it works

• { head -n1 ifile1.txt; paste ifile*.txt;}

  This prints just the first line of ifile1.txt. Then, the paste command causes it to print the first row of all files merged, then the second row merged, and so on:

  $ paste ifile*.txt
  3  5  2  2      1  2  1  3      4  3  4  1
  1  4  2  1      1  3  0  2      5  3  1  5
  4  6  5  2      2  5  5  1      3  4  3  1
  5  5  7  1      0  0  1  1      4  3  4  0
  

• |

  The pipe symbol causes the output of the above commands to be sent as input to awk. Addressing each of the awk commands in turn:

• NR==1{d=NF; next;}

  For the first row, we save the number of columns in variable d. Then, we skip the rest of the commands and start over on the next line of input.

• for (i=1;i<=d;i++) {s=0; for (j=i;j<=NF;j+=d) s+=$j; printf "%.2f%s",s/(NF/d),j==NF+d?"\n":"\t";}

  This adds up the numbers from the respective files and prints the average.

As a multiline script:

{
    head -n1 ifile1.txt
    paste ifile*.txt
} | 
awk '
    NR==1 {d=NF; next;}

    {
        for (i=1;i<=d;i++)
        {
            s=0; for (j=i;j<=NF;j+=d)
                s+=$j;
            printf "%.2f%s",s/(NF/d),j==NF+d?"\n":"\t";
        }
    }

Answer 3

You need to save the sum the fields into an array when you're reading the original files. You can't access $0 and i in the END block, since there's no input line then.

awk '{rows=FNR; cols=NF; for (i = 1; i <= NF; i++) { total[FNR, i] += $i }}
     FILENAME != lastfn { count++; lastfn = FILENAME }
     END { for (i = 1; i <= rows; i++) { 
                for (j =  1; j <= cols; j++) {
                    printf("%s ", total[i, j]/count)
                }
                printf("\n")
            }
        }' ifile* > ofile.txt