Using Python to enter data into form and get data from results page

Question

I am using Python 2.7 on a 32 bit Windows machine.

I am trying to enter species data into http://explorer.natureserve.org and retrieve the results, but am having difficulty understanding how to do it. Needless to say I am relatively new to Python.

I have the following code:

import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("http://explorer.natureserve.org")
assert "NatureServe" in driver.title
SciName = driver.find_element_by_name('searchSciOrCommonName')
SciName.send_keys("Arabis georgiana")
SciName.send_keys(Keys.RETURN) 
assert "No results found." not in driver.page_source

The above works, but now I need to select the element Arabis georgiana on the results page, which will take me to another page. How do I get the results page back into Python and redirect to the page that I actually want?

Answer 1

You need to set the searchSciOrCommonName field value this way:

br.form = list(br.forms())[0]
br.form['searchSciOrCommonName'] = 'butterfly'
response = br.submit()

Then, you can parse the HTML response via, for example, BeautifulSoup:

from bs4 import BeautifulSoup

soup = BeautifulSoup(response)

for item in soup.select('table[border="1"] > tr i')[1:]:
    print(item.text.strip())

which would print:

Aglais io
Callophrys mossii hidakupa
Callophrys mossii marinensis
Cercyonis pegala incana
...
Psora nipponica
Flowering Plants
Asclepias tuberosa