Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using curl in a bash script and getting curl: (3) Illegal characters found in URL

Tags:

linux

bash

curl

awk

So I have a very simple bash script that is curl'ing to an auth server for a header. The header url is written to a var and then used in the next curl call. When using the var set in the first curl call I am getting "curl: (3) Illegal characters found in URL". I am able to echo the var and all looks good, I am even able to reset the var (in my example below) and it works.

The Bash script

URL=$(curl -i -X GET -H "X-Auth-User: MyUserna,e" -H "X-Auth-Key: MyAPIKey" "https://urlToAuthServer.tld/auth/v1.0/" | grep "X-Storage-Url:" | awk '{print $2}')  curl -X GET -H "X-Auth-Token: MyAuthTok" "${URL}/folder/myfile.txt" -o ./myfile.txt 

When running the above example I get:

curl: (3) Illegal characters found in URL 

The URL var looks like this (no illegal chars)

https://somesecureurl.com/auth/AUTH_67383834-45245453-g34g34t5-34534 

When I do this in terminal it works:

$ URL=$(curl -i -X GET -H "X-Auth-User: MyUserna,e" -H "X-Auth-Key: MyAPIKey" "https://urlToAuthServer.tld/auth/v1.0/" | grep "X-Storage-Url:" | awk '{print $2}')

$ echo $URL

https://somesecureurl.com/auth/AUTH_67383834-45245453-g34g34t5-34534

Now I copy and paste the string and reasign it to URL like so (again all in terminal):

>$ URL="https://somesecureurl.com/auth/AUTH_67383834-45245453-g34g34t5-34534" >$ curl -X GET -H "X-Auth-Token: MyAuthTok" "${URL}/folder/myfile.txt" -o ./myfile.txt 

It works.

So why am I getting the "curl: (3) Illegal characters found in URL" error in the first example?

Update I ran this: printf %s "$URL" | xxd

Here is the output (addressed changed up you get the idea)

0000000: 6874 7470 733a 2f2f 6461 6c30 352e 6f62  https://server.ob 0000010: 6a65 6374 7374 6f72 6167 652e 736f 6674  jectstorage.lite 0000020: 6c61 7965 722e 6e65 742f 7631 2f41 5554  sabers.com/v1/AUT 0000030: 485f 6665 3235 3339 3434 2d38 6433 322d  H_aE2563981-7d32- 0000040: 3432 3138 2d61 6566 632d 6665 6638 3465  4201-bdoi-fef94a 0000050: 6166 3331 6232 0d                        ag11c8. 
like image 587
gregwinn Avatar asked Jan 26 '16 16:01

gregwinn


People also ask

Can you use curl in bash script?

If you've ever sat in front of a terminal, typed 'curl', pasted the URL of something you want to download, and hit enter, cool! You're going to be killing it with curl in bash scripts in no time. Here you will learn how to use curl in bash scripts and important tips and tricks for automation.

How do you curl in bash?

To make a GET request using Curl, run the curl command followed by the target URL. Curl automatically selects the HTTP GET request method unless you use the -X, --request, or -d command-line option. The target URL is passed as the first command-line option.

Could not resolve host curl 3 URL using bad Illegal format or missing URL?

Resolution. The error curl: (3) URL using bad/illegal format or missing URL could be caused by a character issue with the passwords. Characters such as @ or & or other symbols may be problematic on the command line. To fix this issue, add double quotes around your URL.

Why curl command is not working Linux?

We might have come across errors like “curl: command not found” while working in the terminal. This type of error comes due to only one reason: the relevant package is not installed. Curl is a very popular data transfer command-line utility used for downloading and uploading data from or to the server.


2 Answers

The $URL contains a \r (CR) at the end (0d). Remove it with

URL=${URL%$'\r'} 

before using it with curl.

like image 158
choroba Avatar answered Oct 11 '22 04:10

choroba


I had the same problem, but it was caused by me having created the script through Windows, which meant that the end of line had an unnecessary \r.

A simple dos to unix conversion fixed it.

dos2unix <scriptname> 
like image 30
ianiver9 Avatar answered Oct 11 '22 04:10

ianiver9