Bash curl and variable in the middle of the url

Question

I would need to read certain data using curl. I'm basically reading keywords from file

while read line do     curl 'https://gdata.youtube.com/feeds/api/users/'"${line}"'/subscriptions?v=2&alt=json' \          > '/home/user/archive/'"$line" done < textfile.txt 

Anyway I haven't found a way to form the url to curl so it would work. I've tried like every possible single and double quoted versions. I've tried basically:

'...'"$line"'...' "..."${line}"..." '...'$line'...' 

and so on.. Just name it and I'm pretty sure that I've tried it.

When I'm printing out the URL in the best case it will be formed as:

 /subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE 

or something similar. If you know what could be the cause of this I would appreciate the information. Thanks!

Answer 1

It's not a quoting issue. The problem is that your keyword file is in DOS format -- that is, each line ends with carriage return & linefeed (\r\n) rather than just linefeed (\n). The carriage return is getting read into the line variable, and included in the URL. The giveaway is that when you echo it, it appears to print:

/subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE" 

but it's really printing:

https://gdata.youtube.com/feeds/api/users/KEYWORD FROM FILE /subscriptions?v=2&alt=json 

...with just a carriage return between them, so the second overwrites the first.

So what can you do about it? Here's a fairly easy way to trim the cr at the end of the line:

cr=$'\r' while read line do     line="${line%$cr}"     curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \          > "/home/user/archive/$line" done < textfile.txt