Making HTTP POST request

Question

I'm trying to make a POST request to retrieve information about a book. Here is the code that returns HTTP code: 302, Moved

import httplib, urllib params = urllib.urlencode({     'isbn' : '9780131185838',     'catalogId' : '10001',     'schoolStoreId' : '15828',     'search' : 'Search'     }) headers = {"Content-type": "application/x-www-form-urlencoded",            "Accept": "text/plain"} conn = httplib.HTTPConnection("bkstr.com:80") conn.request("POST", "/webapp/wcs/stores/servlet/BuybackSearch",              params, headers) response = conn.getresponse() print response.status, response.reason data = response.read() conn.close() 

When I try from a browser, from this page: http://www.bkstr.com/webapp/wcs/stores/servlet/BuybackMaterialsView?langId=-1&catalogId=10001&storeId=10051&schoolStoreId=15828 , it works. What am I missing in my code?

EDIT: Here's what I get when I call print response.msg

302 Moved Date: Tue, 07 Sep 2010 16:54:29 GMT Vary: Host,Accept-Encoding,User-Agent Location: http://www.bkstr.com/webapp/wcs/stores/servlet/BuybackSearch X-UA-Compatible: IE=EmulateIE7 Content-Length: 0 Content-Type: text/plain; charset=utf-8 

Seems that the location points to the same url I'm trying to access in the first place?

EDIT2:

I've tried using urllib2 as suggested here. Here is the code:

import urllib, urllib2  url = 'http://www.bkstr.com/webapp/wcs/stores/servlet/BuybackSearch' values = {'isbn' : '9780131185838',           'catalogId' : '10001',           'schoolStoreId' : '15828',           'search' : 'Search' }   data = urllib.urlencode(values) req = urllib2.Request(url, data) response = urllib2.urlopen(req) print response.geturl() print response.info() the_page = response.read() print the_page 

And here is the output:

http://www.bkstr.com/webapp/wcs/stores/servlet/BuybackSearch Date: Tue, 07 Sep 2010 16:58:35 GMT Pragma: No-cache Cache-Control: no-cache Expires: Thu, 01 Jan 1970 00:00:00 GMT Set-Cookie: JSESSIONID=0001REjqgX2axkzlR6SvIJlgJkt:1311s25dm; Path=/ Vary: Accept-Encoding,User-Agent X-UA-Compatible: IE=EmulateIE7 Content-Length: 0 Connection: close Content-Type: text/html; charset=utf-8 Content-Language: en-US Set-Cookie: TSde3575=225ec58bcb0fdddfad7332c2816f1f152224db2f71e1b0474c866f3b; Path=/ 

Answer 1

Their server seems to want you to acquire the proper cookie. This works:

import urllib, urllib2, cookielib  cookie_jar = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_jar)) urllib2.install_opener(opener)  # acquire cookie url_1 = 'http://www.bkstr.com/webapp/wcs/stores/servlet/BuybackMaterialsView?langId=-1&catalogId=10001&storeId=10051&schoolStoreId=15828' req = urllib2.Request(url_1) rsp = urllib2.urlopen(req)  # do POST url_2 = 'http://www.bkstr.com/webapp/wcs/stores/servlet/BuybackSearch' values = dict(isbn='9780131185838', schoolStoreId='15828', catalogId='10001') data = urllib.urlencode(values) req = urllib2.Request(url_2, data) rsp = urllib2.urlopen(req) content = rsp.read()  # print result import re pat = re.compile('Title:.*') print pat.search(content).group()  # OUTPUT: Title:&nbsp;&nbsp;Statics & Strength of Materials for Arch (w/CD)<br /> 

Answer 2

You might want to use the urllib2 module which should handle redirects better. Here's an example of POSTING with urllib2.