Using Python Requests: Sessions, Cookies, and POST

Question

I am trying to scrape some selling data using the StubHub API. An example of this data seen here:

https://sell.stubhub.com/sellapi/event/4236070/section/null/seatmapdata

You'll notice that if you try and visit that url without logging into stubhub.com, it won't work. You will need to login first.

Once I've signed in via my web browser, I open the URL which I want to scrape in a new tab, then use the following command to retrieve the scraped data:

r = requests.get('https://sell.stubhub.com/sellapi/event/4236070/section/null/seatmapdata') 

However, once the browser session expires after ten minutes, I get this error:

<FormErrors> <FormField>User Auth Check</FormField> <ErrorMessage> Either is not active or the session might have expired. Please login again. </ErrorMessage> 

I think that I need to implement the session ID via cookie to keep my authentication alive and well.

The Requests library documentation is pretty terrible for someone who has never done this sort of thing before, so I was hoping you folks might be able to help.

The example provided by Requests is:

s = requests.Session()  s.get('http://httpbin.org/cookies/set/sessioncookie/123456789') r = s.get("http://httpbin.org/cookies")  print r.text # '{"cookies": {"sessioncookie": "123456789"}}' 

I honestly can't make heads or tails of that. How do I preserve cookies between POST requests?

Answer 1

I don't know how stubhub's api works, but generally it should look like this:

s = requests.Session() data = {"login":"my_login", "password":"my_password"} url = "http://example.net/login" r = s.post(url, data=data) 

Now your session contains cookies provided by login form. To access cookies of this session simply use

s.cookies 

Any further actions like another requests will have this cookie