How to use Curl with HEADERS?

Question

I tried doing something like this but it doesn't work!

$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, "http://google.com/"); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('GET /search?q=kk HTTP/1.1
Host: www.google.de
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.2) Gecko/20100101 Firefox/6.0.2
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Proxy-Connection: Close
Cookie: PREF=ID=2bb051bfbf00e95b:U=c0bb6046a0ce0334:
Cache-Control: max-age=0
Connection: Close
'));
$response = curl_exec($ch); 
curl_close($ch);   



    echo $response;

Also, Is it possible to make the entire request with just headers without setting the URL? I mean without this?

curl_setopt($ch, CURLOPT_URL, "http://google.com/"); 

Thanks!

Answer 1

I got it to work.

1) Change header Host: www.google.de to Host: www.google.com

Motivation: the host specified in the Host header should match the URL's host exactly.

2) Use "www.google.com" instead of "google.com"

Motivation: search requests made to google.com will not retrieve search results. You will be told to go to www.google.com.

3) Set the full URL into CURLOPT_URL, not just the hostname. E.g. change the CURLOPT_URL to curl_setopt($ch, CURLOPT_URL, "http://www.google.com/search?q=kk");

Motivation: correct usage of cURL API.

4) Remove GET /search?q=kk HTTP/1.1 from CURLOPT_HTTPHEADER -- it's misplaced.

Motivation: correct usage of cURL API.

5) The response will be gzip or deflate compressed. To stop this, remove the Accept-Encoding: gzip, deflate request header.

Motivation: if you tell Google you're capable of receiving a compressed response, they will send you one. Decompressing an HTTP response is an extra step that you may not want to undertake. It may be easier to deal with the response if it is in an uncompressed text form.

Answer 2

To add onto what the other posters have said, you also cannot stick a GET command in the CURLOPT_HTTPHEADER array, because that's specified in the other cURL options. cURL is meant to be operated using the curl_setopt function; you cannot bypass it by putting your HTTP message in the headers section. For instance, to ensure that your command is an HTTP GET operation, you set CURLOPT_HTTPGET to TRUE (although by default cURL will send a GET until you change it to something else).

To address your question of why you can't get to the right URL, that's because you need to specify the entire pathname in CURLOPT_URL, not just the host. So, you should really be writing curl_setopt($ch, CURLOPT_URL, "http://google.de/search?q=kk HTTP/1.1"); to set the URL.

Further, I have no idea why you put Connection: Close in he HTTP headers for a GET request. In that header, you're telling Google that you're closing the connection you have.; this is handled by curl_close($ch);, so forget about that header. In fact, half the items in your HTTP headers have no place being there. For instance, why are you sending a cookie in a request to get search results? Make sure you know what each header does before you send it out. Otherwise, you have absolutely no way to tell if you are sending the correct headers or not.