How to grep curl -I header information

Question

I'm trying to get the redirect link from a site by using curl -I then grep to "location" and then sed out the location text so that I am left with the URL.

But this doesn't work. It outputs the URL to screen and doesn't put it

test=$(curl -I "http://www.redirectURL.com/" 2> /dev/null | grep "location" | sed -E 's/location:[ ]+//g')
echo "1..$test..2"

Which then outputs:

..2http://www.newURLfromRedirect.com/bla

What's going on?

Answer 1

As @user353852 points out, you have a carriage return character in you output from curl that is only apparent when you try to echo any character after it. The less pager shows this up as ^M

You can use sed to remove "control characters", like in this example:

% test=$(curl -I "http://www.redirectURL.com/" 2>|/dev/null | awk '/^Location:/ { print $2 }' | sed -e 's/[[:cntrl:]]//') && echo "1..${test}..2"
1..http://www.redirecturl.com..2

Notes:

I used awk rather than your grep [...] | sed approach, saving one process.

For me, curl returns the location in a line starting with 'Location:' (with a capital 'L'), if your version is really reporting it with a lowercase 'l', then you may need to change the regular expression accordingly.

Answer 2

the "Location" http header starts with a capital L, try replacing that in your command.

UPDATE

OK, I have run both lines separately and each runs fine, except that it looks like the output from the curl command includes some control chars which is being captured in the variable. When this is later printed in the echo command, the $test variable is printed followed by carriage return to set the cursor to the start of the line and then ..2 is printed over the top of 1..

Check out the $test variable in less:

echo 1..$test..2 | less

less shows:

1..http://www.redirectURL.com/^M..2

where ^M is the carriage return character.