Scraping a site that wants a cookie

Question

I'm trying to scrape a remote website. I'm using PHP Curl, and my code is good. I know that because I've used Fiddler, Tamper Data, etc to get my code just right. But, it still wasn't working. So, I tried something that I should have tested hours ago:

I turned cookies off in my browser. Sure enough, I can't search the remote site now. Their code requires a cookie to search for the products I want to scrape.

Is there a way to fake/spoof/circumvent so I can scrape the site? Can I tell CURL to use a cookie I download? I don't even know the right question to ask.

Answer 1

Take a look at the various curl_setopt parameters for cookies.

You can use CURLOPT_COOKIE to manually set cookies, or use CURLOPT_COOKIEJAR and a file on disk to actually store and persist cookies across multiple requests.

However, you probably only need a session cookie, which the manual says are supported by default -- as long as you use the same curl instance for each request. If you're making a new curl instance for each request, those instances won't share cookies.