Sending a POST with mechanize and requests.

Question

I am trying to send a POST using mechanize however my code won't work sometimes(and I know why). I have used mechanize, twill and requests. With mechanize and twills it is working and with requests it is not. Probably I am doing it in wrong way.

My mechanize code. The following works :

#!/usr/bin/env python
import sys
import urllib
import mechanize
from mechanize import ParseResponse, urlopen, urljoin
response = urlopen(url)
forms = ParseResponse(response, backwards_compat=False)
form = forms[0]
form["username"] = "avi"
form["password"] = "stackoverflow"
urlopen(form.click())

My twill code. It also works :

import os
import twill
from twill.commands import *
out = open(os.devnull,"w")
twill.set_output(out)
go(url)
formvalue("1", "username", "avi")
formvalue("1", "password", "stackoverflow")
submit()
go(url2) #some protected page
content = show()
print 'content is',content[:100]

From what I understand, mechanize or twill first gets the page, fills the form and sends the form. But problem here is, sometimes target page displays random page, without any form. In that case I get an error, obviously because there is no form to show. I don't want to handle this error, because I already know the post URL. even though some random page is displayed, after I click next, same form is loaded. The POST url, fields all remain same. So I wanted to send the POST request directly, since I already know details everything required. Here is my code, based on :

browser = mechanize.Browser()
parameters = {"username" : "avi",
          "password" : "stackoverflow",
         }  
data = urllib.urlencode(parameters)
browser.open(post_url,data)
cool = browser.open(post_url + '%s' % data).read()
print cool

I get an error :

urllib2.URLError: <urlopen error [Errno 61] Connection refused>

I tried same thing with requests. But I am getting same error as above. Here is the code :

import requests
from requests import session
payload = { 'username': 'avi','password': 'stackoverflow'}
url1 = 'http://example.com/login.php'
url2 = 'http://example.com/protected.php'

with session() as c:
    c.post(url1, data=payload)
    c.get(url2)

So I poked here and there, read few more code available online and I think it is not working because to prevent CSRF. So I went to same page with twill, did showforms and saw there is some value with token :

Here is what I want :

1. Do POST with mechanize and requests (without downloading the page first)
2. How to handle CSRF tokens ?
3. How to debug 'urlopen error [Errno 61] Connection refused'

Answer 1

I've used the code below with success:

params = {u'user_login': self.USER, u'password':self.PASSWORD}
data = urllib.urlencode(params)
request = mechanize.Request( loginURL )
response = mechanize.urlopen(request, data=data)

To handle the CSRF token issue, I added the line below to my controller:

skip_before_filter :verify_authenticity_token, :only => [:create]

To keep the session between requests, I use a cookiesJar. But you can retrieve the CSRF token from your request and keep it in your application, adding it the each new request.