Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Shell script using curl to loop through urls

Tags:

bash

loops

curl

I've been trying to create a simple script that will take a list of queries from a .txt file, append the main url variable, then scrape the content and output it to a text file.

Here's what I have so far:

#!/bin/bash  url="example.com/?q=" for i in $(cat query.txt); do     content=$(curl -o $url $i)     echo $url $i     echo $content >> output.txt done 

list:

images news stuff other 

error log:

curl: (6) Could not resolve host: other; nodename nor servname provided, or not known example.com/?q= other 

If I use this command straight from the command line I get some output into the file:

curl -L http://example.com/?q=other >> output.txt 

Ultimately I would like the output to be:

fetched:    http://example.com/?q=other content:    the output of the page  followed by the next query in the list. 
like image 757
Mena Ortega Avatar asked Apr 21 '13 12:04

Mena Ortega


People also ask

What does cURL do with a URL?

cURL, which stands for client URL, is a command line tool that developers use to transfer data to and from a server. At the most fundamental, cURL lets you talk to a server by specifying the location (in the form of a URL) and the data you want to send.

Can we use cURL command in shell script?

The curl command transfers data to or from a network server, using one of the supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE). It is designed to work without user interaction, so it is ideal for use in a shell script.

How do you iterate through a shell script?

Execute the following command to insert the file's name, followed by a newline, followed by the text Loops Rule! into each file: for FILE in *; do echo -e "$FILE\nLoops Rule\!" > $FILE; done.

How do you loop a shell script in Linux?

The basic syntax of a for loop is: for <variable name> in <a list of items>;do <some command> $<variable name>;done; The variable name will be the variable you specify in the do section and will contain the item in the loop that you're on.


2 Answers

Use more quotes !

  • http://mywiki.wooledge.org/Quotes
  • http://mywiki.wooledge.org/Arguments
  • http://wiki.bash-hackers.org/syntax/words

Try this instead :

url="example.com/?q=" for i in $(cat query.txt); do     content="$(curl -s "$url/$i")"     echo "$content" >> output.txt done 
like image 98
Gilles Quenot Avatar answered Sep 19 '22 03:09

Gilles Quenot


You've got nested quotes, try something like this:

#!/bin/bash  url=https://www.google.fr/?q= while read query do     content=$(curl "{$url}${query}")     echo $query     echo $content >> output.txt done < query.txt 
like image 26
David George Avatar answered Sep 19 '22 03:09

David George