cuRL returnin 404 error

Question

I`m using cuRL to get some data from remote server... The response is in JSON format.. This is my code:

$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER , 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($ch, CURLOPT_URL, 'http://www.myaddress.com/mypage.php');
curl_setopt($ch, CURLOPT_POSTFIELDS, array("id" => $id));
$return = curl_exec($ch);
curl_close($ch);

If I access the link in the browser the page load OK, but if I access through the cuRL return a 404 error...

Answer 1

I can guess a few things that it can be checked from the server side, to show the error.

1) As it is stated in other answers, be sure to set all the necessary headers, you can check them e.g. by firebug, as it is shown in here,

or you can get the headers by php get_headers function. to set it use

curl_setopt($ch, CURLOPT_HTTPHEADER, array("HeaderName: HeaderValue"));

2) When you open a page in the browser(excluding form submit with post method) it makes a get request, instead of post, so if in the server side it is checked $_GET, then your post request will not be considered.

3) If you sure that it should be a post request(say, it is a form submit), then the following can be a problem: some forms can have hidden fields, that again are being checked in the server, and if they are not set, error can be returned. So, you should look at the source code of the form and add them(if there are any) to your post parameters.

4) if you are submitting a form, be sure to set the submit button with its name and value as well, because similar to hidden fields, this can be checked as well.

5) Cookies can be a problem as well, because by default browser has it , and curl does not. To to able to set and read cookies use this code

// set cookie 
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
    
// use cookie
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);

here, $cookie_file path to the cookies file. Do not know in linux or mac, but in windows be sure to use absolute path to the cookie file.

6) Also, you can set the referer by

curl_setopt($ch, CURLOPT_REFERER, 'http://www.myaddress.com/mypage.php');

EDIT: In case of ajax request you might want to add a header X-Requested-With with value as XMLHttpRequest

Answer 2

It's possible the server check the HTTP Header, it's the case in the majority of case.

So add the same HTTP Header of your browser, verify with Firebug :

curl_setopt($ch, CURLOPT_HTTPHEADER, array('SomeName: SomeValue'));

Answer 3

Probably there is something else the browser is sending your cURL code is not. You can use any of the tools other folks have suggested, Firebug, Wireshark, Fiddler, etc, etc.

What you need to do is add missing pieces to your request to match the browser as closely as possible in the cURL request until the remote page responds with a 200.

I notice you're doing a POST. In many cases what happens with your browser is you visit a page with a GET request. A session is initialized on the remote site and a cookie is saved in your browser with the session id.

This cookie then needs to be supplied along with subsequent POST requests. PHP cURL has many options to support cookies. There may be other requirements such as CSRF tokens and so forth.

Again, reverse-engineering is the key.