Accessing a web page with basic auth on python

Question

I'm trying to connect a web page with mechanize but I'm getting a http 401 error.

Here's my code;

import base64, mechanize

url = "http://www.dogus.edu.tr/dusor/FrmMain.aspx"
user = "user"
pwd = "pwd"

br = mechanize.Browser()
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

br.add_password(url, user, pwd)
#br.addheaders.append(('Authorization', 'Basic %s' % base64.encodestring('%s:%s' % (user, pwd))))
print br.open(url).read()

Both add_password and addheaders are not working. Is it because I never specified a realm? How can I get what realm is that web page using? The username and password that I'm using are correct, as I can login using chrome with those credentials.

Answer 1

The site you are using as a sample page requires the NTLM authentication. You can see this by looking at the returned HEADER fields. For example curl -I http://www.dogus.edu.tr/dusor/FrmMain.aspx returns:

HTTP/1.1 401 Unauthorized
Content-Length: 1293
Content-Type: text/html
Server: Microsoft-IIS/7.0
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
Date: Mon, 07 Apr 2014 21:24:09 GMT

The line WWW-Authenticate: NTLM says, which authentication method is used. I think the answer to this question Use python mechanize to log into pages with NTLM authentication will help you.