Python Requests POST doing a GET?

Question

I am using Python 2.7.5, Django 1.7, requests 2.4.1, and doing some simple testing. However, it seems like when I call requests.post, the method is doing a GET instead.

My code, talking to a RESTful API. Note that the POST command works via Hurl.it with this payload and endpoint:

def add_dummy_objective(self):
    """
    To the bank
    """
    payload = {
        'displayName': {
            'text': self._test_objective
        },
        'description': {
            'text': 'For testing of API Middleman'
        },
        'genusTypeId': 'DEFAULT'
    }
    obj_url = self.host + self.bank_id + '/objectives/?proxyname=' + self._admin_key
    req = requests.post(obj_url, data=json.dumps(payload), headers=self.headers)
    return req.json()

I am setting the headers to json:

self.headers = {
    'Content-Type'  : 'application/json'
}

Instead of creating a new objective (as expected with a POST), I get a list of objectives back (what I would expect with a GET). Using pdb, I see:

(Pdb) req.request
<PreparedRequest [GET]>
(Pdb) req.request.method
'GET'

How did this get flipped? I have used the Python requests library before with no issues, so I'm not sure if I am missing something obvious or if (with newer versions of Django / Requests) I have to set another parameter? Is this a caching issue? Any tips for debugging? I have tried re-installing requests, and rolling back Django to 1.6.5, but nothing works...must be simple. -- Thanks!

====== UPDATE 1 ========

Just consolidating some of the debug info that Martijn suggested here:

(Pdb) requests.post.__name__
'post'

Stepping into the requests/api.py > post() definition:

(Pdb) l
 88         :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
 89         :param \*\*kwargs: Optional arguments that ``request`` takes.
 90         """
 91         import pdb
 92         pdb.set_trace()
 93  ->     return request('post', url, data=data, **kwargs)

Drilling down into the request() method:

(Pdb) method
'post'
(Pdb) l
 43           >>> req = requests.request('GET', 'http://httpbin.org/get')
 44           <Response [200]>
 45         """
 46         import pdb
 47         pdb.set_trace()
 48  ->     session = sessions.Session()
 49         return session.request(method=method, url=url, **kwargs)

One more layer, in session.request:

(424)request()
-> method = builtin_str(method)
(Pdb) method
'post'
(Pdb) l
419             :param cert: (optional) if String, path to ssl client cert file (.pem).
420                 If Tuple, ('cert', 'key') pair.
421             """
422             import pdb
423             pdb.set_trace()
424  ->         method = builtin_str(method)
425
426             # Create the Request.
427             req = Request(
428                 method = method.upper(),
429                 url = url,

Stepping down to the end of the method, where the request is actually made, my "prep" is a POST, but my resp is a GET:

(Pdb) prep
<PreparedRequest [POST]>
(Pdb) n
-> return resp
(Pdb) resp
<Response [200]>
(Pdb) resp.request
<PreparedRequest [GET]>
(Pdb) l
449                 'allow_redirects': allow_redirects,
450             }
451             send_kwargs.update(settings)
452             resp = self.send(prep, **send_kwargs)
453
454  ->         return resp
455
456         def get(self, url, **kwargs):
457             """Sends a GET request. Returns :class:`Response` object.
458
459             :param url: URL for the new :class:`Request` object.

Answer 1

To be clear, whenever requests receives a redirect (with a certain status code) we have to perform certain transformations on the request.

In cases like this, when you see something very unexpected the best debugging tips are to retry your request but with allow_redirects=False. This will immediately return the 30x response. Alternatively, you can also check r.history to see if there are any 30x responses that were followed. In this case you probably would have seen something like

>>> r.request.method
'GET'
>>> r.history
[<Response [302]>,]
>>> r.history[0].request.method
'POST'

We know that doing this can cause unexpected behaviour for users (as it just did to you) but it's the only correct way to operate on the web.

I hope this helps you understand why this happened beyond the fact that it was a redirect, and hopefully it gives you and others tools to debug this in the future.

Answer 2

Thanks to Martijn for some debugging tips! The problem was the RESTful API was redirecting me from http:// to https://, which caused the library to return the "second" request (GET)...