Under what conditions are HTTP request headers removed by proxies?

Question

I'm looking at various methods of RESTfully versioning APIs, and there are three major contenders. I believe I've all but settled on using X-API-Version. Putting that debate aside, one of the arguments against using that header, and custom headers in general, is that you can't control when headers are manipulated by proxy servers. I'm curious about what real-world examples there are of this, when it happens on the internet at large, or when it might be used on an intranet or server cluster, or when it might occur in any other situation.

Answer 1

The Guidelines for Web Content Transformation Proxies 1.0 is pretty much the definitive guide to understanding and predicting standards-compliant proxy server behavior. In terms of your question, the Proxy Forwarding of Request portion of the document might be especially helpful.

Each proxy software package and their individual configurations will be vary but, HTTP proxies are generally expected to follow the W3C Guidelines. Here are some highlights.

4.1 Proxy Forwarding of Request:

Other than to convert between HEAD and GET proxies must not alter request methods.

If the request contains a Cache-Control: no-transform directive, proxies must not alter the request other than to comply with transparent HTTP behavior defined in [RFC 2616 HTTP] sections section 14.9.5 and section 13.5.2 and to add header fields as described in 4.1.6 Additional HTTP Header Fields below.

4.1.3 Treatment of Requesters that are not Web browsers

Before altering aspects of HTTP requests and responses proxies need to take account of the fact that HTTP is used as a transport mechanism for many applications other than "Traditional Browsing". Increasingly browser based applications involve exchanges of data using XMLHttpRequest (see 4.2.8 Proxy Decision to Transform) and alteration of such exchanges is likely to cause misoperation.

4.1.5 Alteration of HTTP Header Field Values

Other than the modifications required by [RFC 2616 HTTP] proxies should not modify the values of header fields other than the User-Agent, Accept, Accept-Charset, Accept-Encoding, and Accept-Language header fields and must not delete header fields (see 4.1.5.5 Original Header Fields).

Other than to comply with transparent HTTP operation, proxies should not modify any request header fields unless one of the following applies:

• the user would be prohibited from accessing content as a result of the server responding that the request is "unacceptable" (see 4.2.4 Server Rejection of HTTP Request);

• the user has specifically requested a restructured desktop experience (see 4.1.5.3 User Selection of Restructured Experience);

• the request is part of a sequence of requests comprising either included resources or linked resources on the same Web site (see 4.1.5.4 Sequence of Requests).

These circumstances are detailed in the following sections.

Note:

It is emphasized that requests must not be altered in the presence of Cache-Control: no-transform as described under 4.1.2 no-transform directive in Request.

The URI referred to in the request plays no part in determining whether or not to alter HTTP request header field values. In particular the patterns mentioned in 4.2.8 Proxy Decision to Transform are not material.

4.1.6 Additional HTTP Header Fields

Irrespective of the presence of a no-transform directive:

• proxies should add the IP address of the initiator of the request to the end of a comma separated list in an X-Forwarded-For HTTP header field;

• proxies must (in accordance with RFC 2616) include a Via HTTP header field (see 4.1.6.1 Proxy Treatment of Via Header Field).

  ---

There is also lots of information regarding the alteration of response headers and being able to detect those changes.

As for web service REST API versioning, there is a very lucid and useful SO thread at Best practices for API versioning? that should provide a wealth of helpful insight.

I hope all of this helps. Take care.

Answer 2

This isn't an answer per se, but rather a mention of real-world scenario.

My current environment uses a mixed CAS/AD solution in order to allow SSO across several different platforms (classic ASP, ASP.NET, J2EE, you name it).

Recently we identified some issues - part of the solution involves aggregating Auth tokens to HTTP headers whenever necessary to propagate credentials. One specific solution, making considerable heavy usage of cookies, was chained with an nginx implementation, whose HTTP header limit was set to 4KiB. If the cookie payload went over 2KiB, it would start leaking out headers.

Consequently, applications that had some sort of state/scope control being coordinated via HTTP headers (session cookies included) suddenly started behaving erratically.

On an interesting, related note, REST services using URL versioning (http://server/api/vX.X/resource, for example) were unaffected.