Read data from pipe and write to standard out with a delay in between. Must handle binary files too

Question

I have been trying for about an hour now to find an elegant solution to this problem. My goal is basically to write a bandwidth control pipe command which I could re-use in various situations (not just for network transfers, I know about scp -l 1234). What I would like to do is:

1. Delay for X seconds.
2. Read Y amount (or less than Y if there isn't enough) data from pipe.
3. Write the read data to standard output.

Where:

• X could be 1..n.
• Y could be 1 Byte up to some high value.

My problem is:

• It must support binary data which Bash can't handle well.

Roads I've taken or at least thought of:

• Using a while read data construct, it filters all white characters in the encoding your using.
• Using dd bs=1 count=1 and looping. dd doesn't seem to have different exit codes for when there were something in if and not. Which makes it harder to know when to stop looping. This method should work if I redirect standard error to a temporary file, read it to check if something was transfered (as it's in the statistics printed on stderr) and repeat. But I suspect that it's extremely slow if used on large amounts of data and if it's possible I'd like to skip creating any temporary files.

Any ideas or suggestions on how to solve this as cleanly as possible using Bash?

Answer 1

may be pv -qL RATE ?

   -L RATE, --rate-limit RATE
          Limit the transfer to a maximum of RATE  bytes  per  second.   A
          suffix of "k", "m", "g", or "t" can be added to denote kilobytes
          (*1024), megabytes, and so on.

Answer 2

It's not much elegant but you can use some redirection trick to catch the number of bytes copied by dd and then use it as the exit condition for a while loop:

while [ -z "$byte_copied" ] || [ "$byte_copied" -ne 0 ]; do 
    sleep $X; 
    byte_copied=$(dd bs=$Y count=1 2>&1 >&4 | awk '$2 == "byte"{print $1}'); 
done 4>&1

However, if your intent is to limit the transfer throughput, I suggest you to use pv.