How can I read the contents of an URL with Python?

Question

The following works when I paste it on the browser:

http://www.somesite.com/details.pl?urn=2344 

But when I try reading the URL with Python nothing happens:

 link = 'http://www.somesite.com/details.pl?urn=2344'  f = urllib.urlopen(link)             myfile = f.readline()    print myfile 

Do I need to encode the URL, or is there something I'm not seeing?

Answer 1

To answer your question:

import urllib  link = "http://www.somesite.com/details.pl?urn=2344" f = urllib.urlopen(link) myfile = f.read() print(myfile) 

You need to read(), not readline()

EDIT (2018-06-25): Since Python 3, the legacy urllib.urlopen() was replaced by urllib.request.urlopen() (see notes from https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen for details).

If you're using Python 3, see answers by Martin Thoma or i.n.n.m within this question: https://stackoverflow.com/a/28040508/158111 (Python 2/3 compat) https://stackoverflow.com/a/45886824/158111 (Python 3)

Or, just get this library here: http://docs.python-requests.org/en/latest/ and seriously use it :)

import requests  link = "http://www.somesite.com/details.pl?urn=2344" f = requests.get(link) print(f.text) 

Answer 2

For python3 users, to save time, use the following code,

from urllib.request import urlopen  link = "https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html"  f = urlopen(link) myfile = f.read() print(myfile) 

I know there are different threads for error: Name Error: urlopen is not defined, but thought this might save time.