wget hangs with -r and -O -

Question

This is a VERY strange wget behavior. I'm on debian 7.2.

wget -r -O - www.blankwebsite.com

hangs forever. And I mean it hangs, it isn't searching through the internet, I can verify it with a strace. If I do this:

while read R
do
   wget -r -O - www.blankwebsite.com
done < smallfile

with smallfile containing a single line, the command exits in a few seconds.

I tried also with

wget -r -O - localhost/test.html

with an empty test.html file, same results. To me, it sounds like a bug.
Everything runs fine changing -O - with -O myfile or removing -r.
I used -O - because I was passing output to grep.
Could anyone explain that? Have you seen anything similar?

Answer 1

Of course:

 wget -r -O file www.blankwebsite.com

works, but the BUG is that:

 wget -r -O - www.blankwebsite.com

hangs!

The same problem is if you create a FIFO

mkfifo /tmp/myfifo
wget -r -O /tmp/myfifo www.blankwebsite.com

wget, when called with -r option, will try to find HTML "a href=..." tags reading the output file. Since the output file is a FIFO or stdout (ex. HYPHEN char '-') it is not able to find any tag and waits for INPUT. Then you will have a wget process waintg forever on a read system call.

To resolve this you can: 1) Patch wget to handle this case 2) Patch wget to not allow "-r -O -" combination... (Just check that the argument of '-O' is a regular file) 3) Use a workaround like:

TMPFILE=$(mktemp /tmp/wget.XXXXXX)
wget -r -O $TMPFILE www.blankwebsite.com
grep STRING $TMPFILE
rm $TMPFILE