How can I input data into a webpage to scrape the resulting output using Python?

Question

I am familiar with BeautifulSoup and urllib2 to scrape data from a webpage. However, what if a parameter needs to be entered into the page before the result that I want to scrape is returned?

I'm trying to obtain the geographic distance between two addresses using this website: http://www.freemaptools.com/how-far-is-it-between.htm

I want to be able to go to the page, enter two addresses, click "Show", and then extract the "Distance as the Crow Flies" and "Distance by Land Transport" values and save them to a dictionary.

Is there any way to input data into a webpage using Python?

Answer 1

Take a look at tools like mechanize or scrape:

• http://pypi.python.org/pypi/mechanize
• http://stockrt.github.com/p/emulating-a-browser-in-python-with-mechanize/

• http://www.ibm.com/developerworks/linux/library/l-python-mechanize-beautiful-soup/

• http://zesty.ca/scrape/

Packt Publishing has an article on that matter, too:

• http://www.packtpub.com/article/web-scraping-with-python

Answer 2

Yes! Try mechanize for this kind of Web screen-scraping task.