Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Get webpage contents with Python?

I'm using Python 3.1, if that helps.

Anyways, I'm trying to get the contents of this webpage. I Googled for a little bit and tried different things, but they didn't work. I'm guessing that this should be an easy task, but...I can't get it. :/.

Results of urllib, urllib2:

>>> import urllib2 Traceback (most recent call last):   File "<pyshell#0>", line 1, in <module>     import urllib2 ImportError: No module named urllib2 >>> import urllib >>> urllib.urlopen("http://www.python.org") Traceback (most recent call last):   File "<pyshell#2>", line 1, in <module>     urllib.urlopen("http://www.python.org") AttributeError: 'module' object has no attribute 'urlopen' >>>  

Python 3 solution

Thank you, Jason. :D.

import urllib.request page = urllib.request.urlopen('http://services.runescape.com/m=hiscore/ranking?table=0&category_type=0&time_filter=0&date=1519066080774&user=zezima') print(page.read()) 
like image 371
Andrew Avatar asked Dec 03 '09 22:12

Andrew


People also ask

Can Python get data from website?

Web scraping is the process of collecting and parsing raw data from the Web, and the Python community has come up with some pretty powerful web scraping tools.


2 Answers

If you're writing a project which installs packages from PyPI, then the best and most common library to do this is requests. It provides lots of convenient but powerful features. Use it like this:

import requests response = requests.get('http://hiscore.runescape.com/index_lite.ws?player=zezima') print (response.status_code) print (response.content) 

But if your project does not install its own dependencies, i.e. is limited to things built-in to the standard library, then you should consult one of the other answers.

like image 70
Jonathan Hartley Avatar answered Sep 29 '22 05:09

Jonathan Hartley


Because you're using Python 3.1, you need to use the new Python 3.1 APIs.

Try:

urllib.request.urlopen('http://www.python.org/') 

Alternately, it looks like you're working from Python 2 examples. Write it in Python 2, then use the 2to3 tool to convert it. On Windows, 2to3.py is in \python31\tools\scripts. Can someone else point out where to find 2to3.py on other platforms?

Edit

These days, I write Python 2 and 3 compatible code by using six.

from six.moves import urllib urllib.request.urlopen('http://www.python.org') 

Assuming you have six installed, that runs on both Python 2 and Python 3.

like image 25
Jason R. Coombs Avatar answered Sep 29 '22 06:09

Jason R. Coombs