Using mechanize to login to a webpage

Question

This is my first experience in programming with Python and I'm trying to log in to this webpage. After searching around I found that many people suggested using mechanize. Just to be sure that I setup things correctly before I get to code I downloaded the mechanize zip from the website and had my python script in the unzipped mechanize folder.

I have this code so far using different examples I've found:

import mechanize

theurl = 'http://voyager.umeres.maine.edu/Login'
mech = mechanize.Browser()
mech.open(theurl)

mech.select_form(nr=0)
mech["userid"] = "MYUSERNAME"
mech["password"] = "MYPASSWORD"
results = mech.submit().read()

f = file('test.html', 'w')
f.write(results) 
f.close()

From looking at the source of the webpage I believe the userid/password are the correct names for the form. When I run the script in IDLE I get a bunch of errors including a time out error and a robot error. The full traceback: I'm not exactly sure what I should expect either even if the code works. The login is for my school email which has class folders as well. My end game for what i'm trying to accomplish is once I log into my account I wanted to parse some folders for information and store them in a file that can be later converted in to json or RSS feed, but this is much further down the road with a much better understanding of Python just trying to give a more clear idea of what I want to accomplish.

Answer 1

The problem is that Mechanize is respecting the robots.txt

You must turn it off.

Solution:

mech = mechanize.Browser()
// needs to be set before you call open
mech.set_handle_robots(False)

Edit: it appears that the site is using some sort of additional POST values that are generated via Javascript. This maybe a pain to recreate yourself, check the source of the page to see what's going on. Actual POST values being sent:

challenge   [a14b1f67-11edcc01]
charset UTF-8
login   Login
origurl /Login/
password    
savedpw 0
sha1    3f77d1e8c2ab0470ef8005a85f5f9c0d7aeedba6
userid  sdsads