Menu
I'd like to download a web pages while supplying URLs from stdin. Essentially one process continuously produces URLs to stdout/file and I want to pipe them to wget or curl. (Think about it as simple web crawler if you want).
This seems to work fine:
tail 1.log | wget -i - -O - -q
But when I use 'tail -f' and it doesn't work anymore (buffering or wget is waiting for EOF?):
tail -f 1.log | wget -i - -O - -q
Could anybody provide a solution using wget, curl or any other standard Unix tool? Ideally I don't won't want to restart wget in the loop, just keep it running downloading URLs as they come.
729
Differences Between wget and cURLWget is a simple transfer utility, while curl offers so much more. Curl provides the libcurl library, which can be expanded into GUI applications. Wget, on the other hand, is a simple command-line utility. Wget supports fewer protocols compared to cURL.
Unlike curl , the wget command is solely for the retrieval of information from a remote server. By default, the information received is saved with the same name as in the provided URL. You can specify one or more specific DNS servers to use when utilizing wget to access a remote server.
wget 's major strong side compared to curl is its ability to download recursively. wget is command line only. There's no lib or anything, but curl 's features are powered by libcurl. curl supports FTP , FTPS , HTTP , HTTPS , SCP , SFTP , TFTP , TELNET , DICT , LDAP , LDAPS , FILE , POP3 , IMAP , SMTP , RTMP and RTSP .
GNU Wget (or just Wget, formerly Geturl, also written as its package name, wget) is a computer program that retrieves content from web servers. It is part of the GNU Project. Its name derives from "World Wide Web" and "get." It supports downloading via HTTP, HTTPS, and FTP.
What you need to use is xargs. E.g.
tail -f 1.log | xargs -n1 wget -O - -q
Use xargs which converts stdin to argument.
tail 1.log | xargs -L 1 wget
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
