Python requests: POST request dropping Authorization header

Question

I'm trying to make an API POST request using the Python requests library. I am passing through an Authorization header but when I try debugging, I can see that the header is being dropped. I have no idea what's going on.

Here's my code:

access_token = get_access_token()
bearer_token = base64.b64encode(bytes("'Bearer {}'".format(access_token)), 'utf-8')
headers = {'Content-Type': 'application/json', 'Authorization': bearer_token}
data = '{"FirstName" : "Jane", "LastName" : "Smith"}'
response = requests.post('https://myserver.com/endpoint', headers=headers, data=data)

As you can see above, I manually set the Authorization header in the request arguments, but it is missing the actual request's headers: {'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.4.3 CPython/2.7.9 Linux/4.1.19-v7+'}.

An additional piece of information is that if I change the POST request to a GET request, the Authorization header passes through normally!

Why would this library be dropping the header for POST requests and how do I get this to work?

Using v2.4.3 of the requests lib and Python 2.7.9

Answer 1

TLDR

The url you are requesting redirects POST requests to a different host, so the requests library drops the Authoriztion header in fear of leaking your credentials. To fix that you can override the responsible method in requests' Session class.

Details

In requests 2.4.3, the only place where reqeuests removes the Authorization header is when a request is redirected to a different host. This is the relevant code:

if 'Authorization' in headers:
    # If we get redirected to a new host, we should strip out any
    # authentication headers.
    original_parsed = urlparse(response.request.url)
    redirect_parsed = urlparse(url)

    if (original_parsed.hostname != redirect_parsed.hostname):
        del headers['Authorization']

In newer versions of requests, the Authorization header will be dropped in additional cases (for example if the redirect is from a secure to a non-secure protocol).

So what probably happens in your case, is that your POST requests get redirected to a different host. The only way you can provide authentication for a redirected host using the requests library, is through a .netrc file. Sadly that will only allow you to use HTTP Basic Auth, which doesn't help you much. In that case, the best solution is probably to subclass requests.Session and override this behavior, like so:

from requests import Session

class NoRebuildAuthSession(Session):
    def rebuild_auth(self, prepared_request, response):
        """
        No code here means requests will always preserve the Authorization
        header when redirected.
        Be careful not to leak your credentials to untrusted hosts!
        """

session = NoRebuildAuthSession()
response = session.post('https://myserver.com/endpoint', headers=headers, data=data)

Edit

I have opened a pull-request to the requests library on github to add a warning when this happens. It has been waiting for a second approval to be merged (three months already).