Submitting to a web form using python

Question

I have seen questions like this asked many many times but none are helpful

Im trying to submit data to a form on the web ive tried requests, and urllib and none have worked

for example here is code that should search for the [python] tag on SO:

import urllib import urllib2  url = 'http://stackoverflow.com/'  # Prepare the data values = {'q' : '[python]'} data = urllib.urlencode(values)  # Send HTTP POST request req = urllib2.Request(url, data) response = urllib2.urlopen(req)  html = response.read()  # Print the result print html 

yet when i run it i get the html soure of the home page

here is an example of using requests:

import requests  data= {     'q': '[python]'     } r = requests.get('http://stackoverflow.com', data=data)  print r.text 

same result! i dont understand why these methods arent working i've tried them on various sites with no success so if anyone has successfully done this please show me how!

Thanks so much!

Answer 1

If you want to pass q as a parameter in the URL using requests, use the params argument, not data (see Passing Parameters In URLs):

r = requests.get('http://stackoverflow.com', params=data) 

This will request https://stackoverflow.com/?q=%5Bpython%5D , which isn't what you are looking for.

You really want to POST to a form. Try this:

r = requests.post('https://stackoverflow.com/search', data=data) 

This is essentially the same as GET-ting https://stackoverflow.com/questions/tagged/python , but I think you'll get the idea from this.