access https site python (urllib2, requests)

Question

I am trying to access the following site https://electionbuddy.com/elections/4322/results

I have tried using handler = urllib2.HTTPBasicAuthHandler(password_mgr) etc and requests, but the webapp just sends back a page saying i dont have permission (no where for the user/pass to go). I tried firebug, but couldn't get anything obvious out of it. I can edit and attach if needed.

How could i go about logging in to such a site? As mentioned by Burhan, this case is where you need to "login save the cookies sent and pass them sent on the next request to the results page".

To follow up on that - why doesn't the following work

import requests
payload = {'user_username': 'xxxxxxxxx', 'user_password': 'xxxxxxx'}
r = requests.post("https://electionbuddy.com/login", data=payload)
r = requests.get('https://electionbuddy.com/elections/xxxx/results', cookies=r.cookies)
print r.text

There seems to be an authentication token, different from the session id. Not really sure how to go about doing this though...

Thanks

Answer 1

I assume this site makes a session cookie for you when you login to it. Try first logging in and see if if throws you a cookie, and if so give the cookie to the page you need. Are you familiar with cookielib?

from urllib2 import *
import cookielib

cj = cookielib.CookieJar()
opener = build_opener(
             HTTPHandler(), HTTPSHandler(), HTTPErrorProcessor(), 
             HTTPRedirectHandler(), HTTPCookieProcessor(cj))
params = urllib.urlencode(dict(USER=user, PASSWORD=pw, action='Login'))

# Response here will have cookies, use info() to show
response = opener.open(loginurl, params)
print response.info()

# cookies automatically sent to the page you wanted
response2 = opener.open(your_url)