Fill input of type text and press submit using python

Question

I have this html:

<input type="text" class="txtSearch">
<input type="submit" value="Search" class="sbtSearch">

What I need is to write in the text field and then click on submit using python. The input tags are not inside Form. How I could do that?

Answer 1

You shouldn't have to actually populate the fields and 'click' submit. You can simulate the submission and get the desired results.

Use BeautifulSoup and urllib alongside firebug in Firefox. Watch the network traffic with firebug, and get the post parameters from the HTTP POST that the submit is doing. Create a dict and url-encode it. Pass it alongside your url request.

For example:

from BeautifulSoup import BeautifulSoup
import urllib

post_params = {
    param1 : val1,
    param2 : val2,
    param3 : val3
        }
post_args = urllib.urlencode(post_params)

url = 'http://www.website.com/'
fp = urllib.urlopen(url, post_args)
soup = BeautifulSoup(fp)

The parameter vals will change according to what you're attempting to submit. Make appropriate accommodations in your code.

Answer 2

Here's a selenium solution if you actually need to populate the fields. You would typically only need this for testing purposes, though.

from selenium import webdriver

webpage = r"https://www.yourwebsite.com/" # edit me
searchterm = "Hurricane Sandy" # edit me

driver = webdriver.Chrome()
driver.get(webpage)

sbox = driver.find_element_by_class_name("txtSearch")
sbox.send_keys(searchterm)

submit = driver.find_element_by_class_name("sbtSearch")
submit.click()

Answer 3

UPDATED 2019 answer. This code also takes care of the HTTP 403 Forbidden errors.

import urllib.request as urlRequest
import urllib.parse as urlParse

url = "https://yoururl.com"
values = {"name": "value"}

# pretend to be a chrome 47 browser on a windows 10 machine
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}

# encode values for the url
params = urlParse.urlencode(values).encode("utf-8")

# create the url
targetUrl = urlRequest.Request(url=url, data=params, headers=headers)

# open the url
x  = urlRequest.urlopen(targetUrl)

# read the response
respone = x.read()
print(respone)