Python urllib2 Basic Auth Problem

Question

Update: based on Lee's comment I decided to condense my code to a really simple script and run it from the command line:

import urllib2 import sys  username = sys.argv[1] password = sys.argv[2] url = sys.argv[3] print("calling %s with %s:%s\n" % (url, username, password))  passman = urllib2.HTTPPasswordMgrWithDefaultRealm() passman.add_password(None, url, username, password) urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))  req = urllib2.Request(url) f = urllib2.urlopen(req) data = f.read() print(data) 

Unfortunately it still won't generate the Authorization header (per Wireshark) :(

I'm having a problem sending basic AUTH over urllib2. I took a look at this article, and followed the example. My code:

passman = urllib2.HTTPPasswordMgrWithDefaultRealm() passman.add_password(None, "api.foursquare.com", username, password) urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))  req = urllib2.Request("http://api.foursquare.com/v1/user")     f = urllib2.urlopen(req) data = f.read() 

I'm seeing the following on the Wire via wireshark:

GET /v1/user HTTP/1.1 Host: api.foursquare.com Connection: close Accept-Encoding: gzip User-Agent: Python-urllib/2.5  

You can see the Authorization is not sent, vs. when I send a request via curl: curl -u user:password http://api.foursquare.com/v1/user

GET /v1/user HTTP/1.1 Authorization: Basic =SNIP= User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3 Host: api.foursquare.com Accept: */* 

For some reason my code seems to not send the authentication - anyone see what I'm missing?

thanks

-simon

Answer 1

The problem could be that the Python libraries, per HTTP-Standard, first send an unauthenticated request, and then only if it's answered with a 401 retry, are the correct credentials sent. If the Foursquare servers don't do "totally standard authentication" then the libraries won't work.

Try using headers to do authentication:

import urllib2, base64  request = urllib2.Request("http://api.foursquare.com/v1/user") base64string = base64.b64encode('%s:%s' % (username, password)) request.add_header("Authorization", "Basic %s" % base64string)    result = urllib2.urlopen(request) 

Had the same problem as you and found the solution from this thread: http://forums.shopify.com/categories/9/posts/27662

Answer 2

(copy-paste/adapted from https://stackoverflow.com/a/24048772/1733117).

First you can subclass urllib2.BaseHandler or urllib2.HTTPBasicAuthHandler, and implement http_request so that each request has the appropriate Authorization header.

import urllib2 import base64  class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler):     '''Preemptive basic auth.      Instead of waiting for a 403 to then retry with the credentials,     send the credentials if the url is handled by the password manager.     Note: please use realm=None when calling add_password.'''     def http_request(self, req):         url = req.get_full_url()         realm = None         # this is very similar to the code from retry_http_basic_auth()         # but returns a request object.         user, pw = self.passwd.find_user_password(realm, url)         if pw:             raw = "%s:%s" % (user, pw)             auth = 'Basic %s' % base64.b64encode(raw).strip()             req.add_unredirected_header(self.auth_header, auth)         return req      https_request = http_request 

Then if you are lazy like me, install the handler globally

api_url = "http://api.foursquare.com/" api_username = "johndoe" api_password = "some-cryptic-value"  auth_handler = PreemptiveBasicAuthHandler() auth_handler.add_password(     realm=None, # default realm.     uri=api_url,     user=api_username,     passwd=api_password) opener = urllib2.build_opener(auth_handler) urllib2.install_opener(opener)