Menu
I've been trying to create a simple script that will take a list of queries from a .txt file, append the main url variable, then scrape the content and output it to a text file.
Here's what I have so far:
#!/bin/bash url="example.com/?q=" for i in $(cat query.txt); do content=$(curl -o $url $i) echo $url $i echo $content >> output.txt done list:
images news stuff other error log:
curl: (6) Could not resolve host: other; nodename nor servname provided, or not known example.com/?q= other If I use this command straight from the command line I get some output into the file:
curl -L http://example.com/?q=other >> output.txt Ultimately I would like the output to be:
fetched: http://example.com/?q=other content: the output of the page followed by the next query in the list. 
757
cURL, which stands for client URL, is a command line tool that developers use to transfer data to and from a server. At the most fundamental, cURL lets you talk to a server by specifying the location (in the form of a URL) and the data you want to send.
The curl command transfers data to or from a network server, using one of the supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE). It is designed to work without user interaction, so it is ideal for use in a shell script.
Execute the following command to insert the file's name, followed by a newline, followed by the text Loops Rule! into each file: for FILE in *; do echo -e "$FILE\nLoops Rule\!" > $FILE; done.
The basic syntax of a for loop is: for <variable name> in <a list of items>;do <some command> $<variable name>;done; The variable name will be the variable you specify in the do section and will contain the item in the loop that you're on.
Use more quotes !
Try this instead :
url="example.com/?q=" for i in $(cat query.txt); do content="$(curl -s "$url/$i")" echo "$content" >> output.txt done You've got nested quotes, try something like this:
#!/bin/bash url=https://www.google.fr/?q= while read query do content=$(curl "{$url}${query}") echo $query echo $content >> output.txt done < query.txt If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
