Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

curl 302 redirect not working (command line)

Tags:

bash

curl

scrape

In the browser, navigating to this URL initiates a 302 (moved temporarily) request which in turn downloads a file.

http://www.targetsite.com/target.php/?event=download&task_id=123

When I view what is actually happening via Chrome network tools I see that the redirect is going to a dynamically generated path that cancels itself immediately after download. In other words, even if I know that full path I will not have time to manually call it.

So, how in using the command line can I mimic the browser actions?

I tried

curl --cookies bin/cookies.txt -O -L "http://www.targetsite.com/target.php/?event=download&task_id=123" --compressed

but this just returns gibberish. The goal of this is to programmatically download this file without having to navigate to the site. Unfortunately I cannot share the site here as it is behind a log-in.

like image 475
user2029890 Avatar asked Jan 03 '14 13:01

user2029890


People also ask

How do I enable redirects in curl?

To follow redirect with Curl, use the -L or --location command-line option. This flag tells Curl to resend the request to the new address. When you send a POST request, and the server responds with one of the codes 301, 302, or 303, Curl will make the subsequent request using the GET method.

How do you fix a 302 redirect?

How to fix it? You should only use 302 redirects where the redirection is temporary and content will come back to the original URL soon. Check the reported URLs. Where the redirection is permanent, change the redirection to 301 (Moved Permanently).

Does curl follow redirects by default?

In curl's tradition of only doing the basics unless you tell it differently, it does not follow HTTP redirects by default. Use the -L, --location option to tell it to do that. When following redirects is enabled, curl will follow up to 50 redirects by default.

Does a 302 automatically redirect?

Does a 302 automatically redirect? Yes. Since the browser is handling the communication between the server and you, it will automatically redirect to the new resource location. However, the same communication will occur with the new resource, too.


1 Answers

You need to supply the -L or --location option in order to enable curl to follow HTTP redirects.

Quoting from man curl:

   -L, --location           (HTTP/HTTPS)  If  the server reports that the requested page has           moved to a different location (indicated with a Location: header           and  a  3XX  response code), this option will make curl redo the           request on the new place. If used together with -i, --include or           -I, --head, headers from all requested pages will be shown. When           authentication is used, curl only sends its credentials  to  the           initial  host.  If a redirect takes curl to a different host, it           won't be able to intercept the user+password. See  also  --loca‐           tion-trusted  on how to change this. You can limit the amount of           redirects to follow by using the --max-redirs option.            When curl follows a redirect and the request is not a plain  GET           (for example POST or PUT), it will do the following request with           a GET if the HTTP response was 301, 302, or 303. If the response           code  was  any  other  3xx code, curl will re-send the following           request using the same unmodified method. 
like image 170
devnull Avatar answered Sep 20 '22 14:09

devnull