BASH: How to write values generated by a for loop to a file quickly

Question

I have a for loop in bash that writes values to a file. However, because there are a lot of values, the process takes a long time, which I think can be saved by improving the code.

nk=1152
nb=24

for k in $(seq 0 $((nk-1))); do
    for i in $(seq 0 $((nb-1))); do
        for j in $(seq 0 $((nb-1))); do
            echo -e "$k\t$i\t$j" 
        done
    done
done > file.dat

I've moved the output action to after the entire loop is done rather than echo -e "$k\t$i\t$j" >> file.dat to avoid opening and closing the file many times. However, the speed the script writes to the file is still rather slow, ~ 10kbps.

Is there a better way to improve the IO?

Many thanks Jacek

Answer 1

It looks like the seq calls are fairly punishing since that is a separate process. Try this just using shell math instead:

for ((k=0;k<=$nk-1;k++)); do
    for ((i=0;i<=$nb-1;i++)); do
        for ((j=0;j<=$nb-1;j++)); do
            echo -e "$k\t$i\t$j" 
        done
    done
done > file.dat

It takes just 7.5s on my machine.

Another way is to compute the sequences just once and use them repeatedly, saving a lot of shell calls:

    nk=1152
    nb=24

    kseq=$(seq 0 $((nk-1)))
    bseq=$(seq 0 $((nb-1)))

    for k in $kseq; do
        for i in $bseq; do
            for j in $bseq; do
                echo -e "$k\t$i\t$j" 
            done
        done
    done > file.dat

This is not really "better" than the first option, but it shows how much of the time is spent spinning up instances of seq versus actually getting stuff done.

Bash isn't always the best for this. Consider this Ruby equivalent which runs in 0.5s:

#!/usr/bin/env ruby

nk=1152
nb=24

nk.times do |k|
  nb.times do |i|
    nb.times do |j|
      puts "%d\t%d\t%d" % [ k, i, j ]
    end
  end
end