What to do with extra HTTP header from proxy?

Question

Our environment requires the use of an outbound proxy for offsite services. Normally this isn't a problem. In this case with Twilio, the extra header returned breaks the client.

Outgoing headers:

POST /2010-04-01/Accounts/FOO/SMS/Messages.json HTTP/1.1
Authorization: Basic FOO==
User-Agent: twilio-php/3.10.0
Host: api.twilio.com
Accept: */*
Accept-Charset: utf-8
Content-Type: application/x-www-form-urlencoded
Content-Length: 108

Response Headers:

HTTP/1.0 200 Connection established

HTTP/1.1 201 Created
Server: nginx
Date: Thu, 06 Jun 2013 14:39:24 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 551
Connection: close
X-Powered-By: PHP/5.3.11

I can only assume the proxy is adding the extra HTTP header.

The Twilio client does check for:

list($head, $body) = ($parts[0] == 'HTTP/1.1 100 Continue') 

As I understand it, there are times or versions of curl that will automatically add an Expect header in the request, and the HTTP 100 would be returned in the response, but in this case it is not, and the response is 200 Connection established. For what it's worth adding an empty Expect: or an Expect:bacon didn't change the results.

I'd really prefer not to hack on the Twilio client too much here, and I especially would like to avoid just adding a || $parts[0] == 'HTTP/1.0 200 Connection established' as it seems like that's messy.

Is it possible to send a request header in the that will suppress/hide the extra header? Or, a curl option I'm not seeing to ignore it?

The outbound proxy is Linux/Squid

Answer 1

The proxy issue is something a lot of scripts are facing. The preferred solution I can find on the internet is to simply add the following lines of code.

<?php
// cURL automatically handles Proxy rewrites, remove the "HTTP/1.0 200 Connection established" string
if (false !== stripos($response, "HTTP/1.0 200 Connection established\r\n\r\n")) {
  $response = str_ireplace("HTTP/1.0 200 Connection established\r\n\r\n", '', $response);
}
?>

Now to add this to the twilio client would be a bit messy indeed. Luckily you can use namespaces to recreate native functions. See the following example.

<?php
namespace FakeCurl;

//create curl_exec function with same name, but its created in the FakeCurl namespace now.
function curl_exec($ch) {
  //execute the actual curl_exec function in the main namespace
  $response =  \curl_exec($ch);

  // cURL automatically handles Proxy rewrites, remove the "HTTP/1.0 200 Connection established" string
  if (false !== stripos($response, "HTTP/1.0 200 Connection established\r\n\r\n")) {
    $response = str_ireplace("HTTP/1.0 200 Connection established\r\n\r\n", '', $response);
  } 

  return "ADDED TO RESPONSE\r\n\r\n".$response;
}

//make a regular curl request, no alterations.

$curl = curl_init();
curl_setopt_array( $curl, array(
    CURLOPT_HEADER => true,
    CURLOPT_NOBODY => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_URL => 'http://stackoverflow.com' ) );
$response = curl_exec( $curl );
curl_close( $curl );

echo '<pre>'.$response.'</pre>';

?>

Output

ADDED TO RESPONSE

HTTP/1.1 200 OK
Cache-Control: public, max-age=11
Content-Length: 191951
Content-Type: text/html; charset=utf-8
Expires: Wed, 12 Jun 2013 07:09:02 GMT
Last-Modified: Wed, 12 Jun 2013 07:08:02 GMT
Vary: *
X-Frame-Options: SAMEORIGIN
Date: Wed, 12 Jun 2013 07:08:49 GMT

So to use with the twilio client, you need to put at the very top of your script the following:

<?php
namespace FakeCurl;
function curl_exec($ch) {
  $response =  \curl_exec($ch);

  // cURL automatically handles Proxy rewrites, remove the "HTTP/1.0 200 Connection established" string
  if (false !== stripos($response, "HTTP/1.0 200 Connection established\r\n\r\n")) {
    $response = str_ireplace("HTTP/1.0 200 Connection established\r\n\r\n", '', $response);
  } 

  return $response;
}

include("twilio.php");
?>

Should the namespace option fail for some reason, I would add a simple function outside the twilio client like.

<?php
function fixProxyResponse($response) {
  // cURL automatically handles Proxy rewrites, remove the "HTTP/1.0 200 Connection established" string
  if (false !== stripos($response, "HTTP/1.0 200 Connection established\r\n\r\n")) {
    $response = str_ireplace("HTTP/1.0 200 Connection established\r\n\r\n", '', $response);
  } 

  return $response;
}

And then alter the twilio script TinyHttp.php and change only the following line (~linenr 63)

if ($response = curl_exec($curl)) {
  $parts = explode("\r\n\r\n", $response, 3);
  list($head, $body) = ($parts[0] == 'HTTP/1.1 100 Continue')

to

if ($response = curl_exec($curl)) {
  $parts = explode("\r\n\r\n", fixProxyResponse($response), 3);
  list($head, $body) = ($parts[0] == 'HTTP/1.1 100 Continue')

Answer 2

Some very late clarification. When you connect to an SSL/TLS server through a proxy, the proxy establishes a tunnel, using HTTP CONNECT. This is covered in RFC2817 and the expired tunneling spec, and not RFC2616.

The original tunneling spec required the proxy to return a '200 Connection Established' to the client once it had successfully connected to the server, which is what you're seeing. This is potentially followed by more headers, and then an empty line, before the connection becomes transparent and you get the actual response from the server. So, you get two sets of headers. RFC 2817 relaxes this, and allows any 2xx response to a CONNECT request.

Which means, in short, that you can't rely on detecting and stripping out a single header line using the php code above. There may be more than one line, and the first line may not have a 200 code, and and may not include the 'connection established' string. You have to be prepared to detect two full sets of headers.

cURL stripped out the initial connect response until 7.11.1 in 2004, but now sends everything back to the client. See here for details.