Redirecting printf to file in awk

Question

I have a simple bash script.

It's purpose is to monitor http access log file (test.log) and output to a file (out.log) the updated hit rate:

  stdbuf -o0 tail -f test.log | awk -F'[ "]+' '{
  ipcount[$1]++;
  print "test" > "out.log"; #Truncate out.log
  for (i in ipcount) {
        printf "%15s - %d\n", i, ipcount[i] >> "out.log";
        printf "%15s - %d\n", i, ipcount[i] }
}'

The main logic works. my only problem is the redirection to "out.log" that doesn't seem to work. The last printf outputs the expected result to the standard output. But the other two printf do not output anything to "out.log", and I cannot figure out why. out.log has all permissions (777)

Answer 1

This should work for you:

tail -f test.log | awk -F'[ "]+' -v out_file="out.log" '{
    val_count[$1]++
    print "" > out_file

    for (i in val_count) {
        printf "%15s - %d\n", i, val_count[i] >> out_file
        printf "%15s - %d\n", i, val_count[i]
    }

    close(out_file)
}'

(Note: I moved the output file definition to the command line to hopefully reduce repetition.)

Your original version has one fatal issue: print "" > "out.log" only truncates out.log the first time it's called. All subsequent calls to it will simply append to it because it's already open. As a secondary issue, awk likes to buffer output, so the contents will only be flushed intermittently.

To fix this, we need to close the file after each iteration. This forcefully flushes the output to out.log and forces the > redirection to re-truncate the file on the next iteration. If you didn't need to truncate each iteration, a simple fflush(out_file) would suffice.

---

To illustrate the issue more clearly...

This results in an output.txt that has multiple lines because it is truncated just once (the first iteration):

ls -l | awk '{ print "This file has many lines" > "output.txt"; }'

This results in an output.txt with a single output line because it is truncated multiple times:

ls -l | awk '{ print "This file has one line" > "output.txt"; close("output.txt"); }'