Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Cloudflare won't cache page via CURL request

One.php makes two separate CURL requests to two.php. I want Cloudflare to cache the first response, and to serve that cache in the second response. But for some reason, Cloudflare is not caching either response. If I then visit two.php using my browser, it's still not cached. If I then refresh two.php in my browser, I finally receive a cache HIT. How do I trigger caching on Cloudflare via CURL requests?

Bare Bones Test:

./one.php

header("Cache-Control: no-store");
$url = "http://www.example.com/two.php";

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);

sleep(1);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_exec($ch); // prints current timestamp, see two.php
curl_close($ch);

./two.php

header( 'Cache-Control: public, max-age=10' );
print( time()."<br>\r\n" );

Extended Test:

I tried setting the User-Agent header and using Cloudflare's cfduid cookie. I tried issuing the CURL requests on and between two different websites on two different servers. I tried HTTP and HTTPS. I tried running one.php from direct.example.com (which bypasses cloudflare).

./one.php

header("Cache-Control: no-store");

$url = "http://www.example.com/target.php";
$request_headers = array(
  'Connection: keep-alive',
  'Upgrade-Insecure-Requests: 1',
  'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
  'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
  'Accept-Encoding: compressed',
  'Accept-Language: en-US,en;q=0.9',
);

// Fetch CFDUID Cookie
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
$response_headers = curl_exec($ch);
curl_close($ch);

// Extract Cookies
preg_match_all( '/^Set-Cookie:\s*(.+?);?\s*$/mi', $response_headers, $cookies);
$cookies = implode( ';', $cookies[1] );

// Setup First request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );

// Print First Response
echo( "First Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );

// Pause
sleep(1);

// Setup Second request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_COOKIE, $cookies );

// Record CURL Request Info
curl_setopt($ch, CURLOPT_VERBOSE, true);
$verbose = fopen('php://temp', 'w+');
curl_setopt($ch, CURLOPT_STDERR, $verbose);

// Print Second Response
echo( "Second Response:<br>\r\n<pre>" );
curl_exec($ch);
curl_close($ch);
echo( "</pre>" );

// Show CURL Request Info
rewind($verbose);
$verboseLog = stream_get_contents($verbose);
echo "CURL Verbose Info:<br>\r\n<pre>", htmlspecialchars($verboseLog), "</pre>\r\n";

./two.php

header("Cache-Control: public, max-age=10");
header( "set-cookie: test1=true; test2=true" );
echo( time() . "\r\n" );
var_dump( getallheaders() );

Extended Test Output (Censored):

First Response:
1536190748
array(25) {
  ["Content-Type"]=>
  string(0) ""
  ["Content-Length"]=>
  string(1) "0"
  ["X-Original-Url"]=>
  string(20) "/target.php"
  ["Was-Default-Hostname"]=>
  string(33) "*****.azurewebsites.net"
  ["X-Site-Deployment-Id"]=>
  string(15) "*****"
  ["Disguised-Host"]=>
  string(21) "www.*****.com"
  ["X-Arr-Log-Id"]=>
  string(36) "c157*****e003"
  ["Is-Service-Tunneled"]=>
  string(1) "0"
  ["Client-Ip"]=>
  string(21) "*.*.*.175:18968"
  ["X-Waws-Unencoded-Url"]=>
  string(20) "/target.php"
  ["Cf-Connecting-Ip"]=>
  string(12) "*.*.*.33"
  ["Upgrade-Insecure-Requests"]=>
  string(1) "1"
  ["Cf-Visitor"]=>
  string(17) "{"scheme":"http"}"
  ["X-Forwarded-Proto"]=>
  string(4) "http"
  ["Cf-Ray"]=>
  string(20) "455c*****-ATL"
  ["X-Forwarded-For"]=>
  string(35) "*.*.*.33, *.*.*.175:18968"
  ["Cf-Ipcountry"]=>
  string(2) "US"
  ["User-Agent"]=>
  string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
  ["Max-Forwards"]=>
  string(2) "10"
  ["Host"]=>
  string(21) "www.*****.com"
  ["Cookie"]=>
  string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
  ["Accept-Language"]=>
  string(14) "en-US,en;q=0.9"
  ["Accept-Encoding"]=>
  string(4) "gzip"
  ["Accept"]=>
  string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
  ["Connection"]=>
  string(10) "Keep-Alive"
}

Second Response:
1536190749
array(25) {
  ["Content-Type"]=>
  string(0) ""
  ["Content-Length"]=>
  string(1) "0"
  ["X-Original-Url"]=>
  string(20) "/target.php"
  ["Was-Default-Hostname"]=>
  string(33) "*****.azurewebsites.net"
  ["X-Site-Deployment-Id"]=>
  string(15) "*****"
  ["Disguised-Host"]=>
  string(21) "www.*****.com"
  ["X-Arr-Log-Id"]=>
  string(36) "588f*****3a02"
  ["Is-Service-Tunneled"]=>
  string(1) "0"
  ["Client-Ip"]=>
  string(21) "*.*.*.175:19820"
  ["X-Waws-Unencoded-Url"]=>
  string(20) "/target.php"
  ["Cf-Connecting-Ip"]=>
  string(12) "*.*.*.33"
  ["Upgrade-Insecure-Requests"]=>
  string(1) "1"
  ["Cf-Visitor"]=>
  string(17) "{"scheme":"http"}"
  ["X-Forwarded-Proto"]=>
  string(4) "http"
  ["Cf-Ray"]=>
  string(20) "455c8*****-ATL"
  ["X-Forwarded-For"]=>
  string(35) "*.*.*.33, *.*.*.175:19820"
  ["Cf-Ipcountry"]=>
  string(2) "US"
  ["User-Agent"]=>
  string(115) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36"
  ["Max-Forwards"]=>
  string(2) "10"
  ["Host"]=>
  string(21) "www.*****.com"
  ["Cookie"]=>
  string(279) "__cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com"
  ["Accept-Language"]=>
  string(14) "en-US,en;q=0.9"
  ["Accept-Encoding"]=>
  string(4) "gzip"
  ["Accept"]=>
  string(85) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
  ["Connection"]=>
  string(10) "Keep-Alive"
}

CURL Verbose Info:
* Hostname www.*****.com was found in DNS cache
*   Trying *.*.*.46...
* TCP_NODELAY set
* Connected to www.*****.com (*.*.*.46) port 80 (#0)
> GET /target.php HTTP/1.1
Host: www.*****.com
Cookie: __cfduid=d2be*****0748; expires=Thu, 05-Sep-19 23:39:08 GMT; path=/; domain=.*****.com; HttpOnly;test1=true; test2=true;ARRAffinity=91cd*****4f3f;Path=/;HttpOnly;Domain=www.*****.com
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/*.*.*.106 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: compressed
Accept-Language: en-US,en;q=0.9

< HTTP/1.1 200 OK
< Date: Wed, 05 Sep 2018 23:39:10 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: public, max-age=10
< Vary: Accept-Encoding
< Set-Cookie: test1=true; test2=true
< X-Powered-By: PHP/5.6.37
< X-Powered-By: ASP.NET
< CF-Cache-Status: MISS
< Server: cloudflare
< CF-RAY: 455c*****-ATL
< 
* Connection #0 to host www.*****.com left intact
like image 920
skibulk Avatar asked Sep 03 '18 13:09

skibulk


1 Answers

Requests to CloudFlare are geographically distributed. Chances are your server is hitting a different CloudFlare edge location than you are (there are 152 right now), so you don't benefit from its calls.

like image 164
ceejayoz Avatar answered Oct 02 '22 22:10

ceejayoz