How do I CURL www.google.com - it keeps redirecting me to .co.uk

Question

I am using CURL to check for the existence of a URL (HEAD request) but when I test it with www.google.com, it redirects me to www.google.co.uk - probably because my server is UK-based.

Is there a way you can stop this from happening? I don't want to remove the CURLOPT_FOLLOWLOCATION option as this is useful for 301 redirects etc.

Part of my code is below;

$ch = curl_init();

    // set URL and other appropriate options
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_MAXREDIRS, 5);
    curl_setopt($ch, CURLOPT_NOBODY, true);
    curl_setopt($ch, CURLOPT_AUTOREFERER, true);
    curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 4);
    curl_setopt($ch, CURLOPT_TIMEOUT, 4);

    $output = curl_exec($ch);

    // get data     
$data = curl_getinfo($ch);

$data['url'] contains www.google.co.uk when I set $url as www.google.com

Answer 1

You need to use curl with a cookie that simulate a similar behavior in a browser.

When you visit google.com from England it redirects you to google.co.uk, however there is a link on that page titled "go to google.com" that lets you go back to google.com and stay there. It uses a cookie to remember your site preferences.

For example, here are the cookies that I have after doing this (using firefox):

Answer 2

Try accessing www.google.com/ncr, it'll avoid the redirect to the .co.uk (or any other national) page.

Answer 3

Another option is to use simply encrypted.google.com. That won't redirect.

Answer 4

A bit of a hack, but how about using an IP address? http://216.239.59.147/ http://66.102.7.104/