Fill form values in a web page via a Python script (not testing)

Question

I need to fill form values on a target page then click a button via Python. I've looked at Selenium and Windmill, but these are testing frameworks - I'm not testing. I'm trying to log into a 3rd party website programatically, then download and parse a file we need to insert into our database. The problem with the testing frameworks is that they launch instances of browsers; I just want a script I can schedule to run daily to retrieve the page I want. Any way to do this?

Answer 1

You are looking for Mechanize

Form submitting sample:

import re from mechanize import Browser  br = Browser() br.open("http://www.example.com/") br.select_form(name="order") # Browser passes through unknown attributes (including methods) # to the selected HTMLForm (from ClientForm). br["cheeses"] = ["mozzarella", "caerphilly"]  # (the method here is __setitem__) response = br.submit()  # submit current form 

Answer 2

Have a look on this example which use Mechanize: it will give the basic idea:

#!/usr/bin/python import re  from mechanize import Browser br = Browser()  # Ignore robots.txt br.set_handle_robots( False ) # Google demands a user-agent that isn't a robot br.addheaders = [('User-agent', 'Firefox')]  # Retrieve the Google home page, saving the response br.open( "http://google.com" )  # Select the search box and search for 'foo' br.select_form( 'f' ) br.form[ 'q' ] = 'foo'  # Get the search results br.submit()  # Find the link to foofighters.com; why did we run a search? resp = None for link in br.links():     siteMatch = re.compile( 'www.foofighters.com' ).search( link.url )     if siteMatch:         resp = br.follow_link( link )         break  # Print the site content = resp.get_data() print content