Simulating ajax request with python using requests lib

Question

Why does request not download a response for this webpage?

#!/usr/bin/python

import requests

headers={ 'content-type':'application/x-www-form-urlencoded; charset=UTF-8',
     'Accept-Encoding': 'gzip, deflate',
     'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0',
     'Referer' : 'http://sportsbeta.ladbrokes.com/football',
    }

payload={'N': '4294966750',
     'facetCount_156%23327': '12',
     'facetCount_157%23325': '8',
     'form-trigger':'moreId',
     'moreId':'156%23327',
     'pageId':'p_football_home_page',
     'pageType':'EventClass',
     'type':'ajaxrequest'
     }

url='http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController'

r = requests.post(url, data=payload, headers=headers)

These are the parameters of the POST that I see in Firebug, and there the response received back contains a list (of football leagues), yet when I run my python script like this I get nothing.

(you can see the request in Firefox by clicking the See All in the competitions section of the left hand nav bar of link and looking at the XHR in Firebug. The Firebug response shows the HTML body as expected.)

Anyone any ideas? Will my handling of the % symbols in the payload be causing any trouble at all?

EDIT: Attempt using session

from requests import Request, Session

#turn post string into dict: 
def parsePOSTstring(POSTstr):
    paramList = POSTstr.split('&')
    paramDict = dict([param.split('=') for param in paramList])
    return paramDict

headers={'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0',
     'Referer' : 'http://sportsbeta.ladbrokes.com/football'
    }

#prep the data (POSTstr copied from Firebug raw source)
POSTstr = "moreId=156%23327&facetCount_156%23327=12&event=&N=4294966750&pageType=EventClass&
          pageId=p_football_home_page&type=ajaxrequest&eventIDNav=&removedSelectionNav=&
          currentSelectedId=&form-trigger=moreId"
payload = parsePOSTstring(POSTstr)

#end url
url='http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController'

#start a session to manage cookies, and visit football page first so referer agrees
s = Session()
s.get('http://sportsbeta.ladbrokes.com/football')
#now visit disired url with headers/data
r = s.post(url, data=payload, headers=headers)

#print output
print r.text #this is empty

Working curl

curl 'http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController'
-H 'Cookie: JSESSIONID=DE93158F07E02DD3CC1CC32B1AA24A9E.ecomprodsw015;
    geoCode=FRA; 
    FLAGS=en|en|uk|default|ODDS|0|GBP;
    ECOM_BETA_SPORTS=1;
    PLAYED=4%7C0%7C0%7C0%7C0%7C0%7C0'
-H 'Referer: http://sportsbeta.ladbrokes.com/football'
-H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) 
    Gecko/20100101 Firefox/27.0'  
--data 'facetCount_157%23325=8&moreId=156%23327&
        facetCount_156%23327=12&event=&
        N=4294966750&
        pageType=EventClass&pageId=p_football_home_page&
        type=ajaxrequest&eventIDNav=&
        removedSelectionNav=&currentSelectedId=&
        form-trigger=moreId' --compressed

Yet this curl works.

Answer 1

Here's the smallest working example that I can come up with:

from requests import Session

session = Session()

# HEAD requests ask for *just* the headers, which is all you need to grab the
# session cookie
session.head('http://sportsbeta.ladbrokes.com/football')

response = session.post(
    url='http://sportsbeta.ladbrokes.com/view/EventDetailPageComponentController',
    data={
        'N': '4294966750',
        'form-trigger': 'moreId',
        'moreId': '156#327',
        'pageType': 'EventClass'
    },
    headers={
        'Referer': 'http://sportsbeta.ladbrokes.com/football'
    }
)

print response.text

You just weren't decoding the percent-encoded POST data properly, so # was being represented as %23 in the actual POST data (e.g. 156%23327 should've been 156#327).