How to extract ids and classes from a webpage using python?

Question

This is my code so far :

import urllib2
with urllib2.urlopen("https://quora.com") as response:
    html = response.read()

I am new to Python and somehow I am successful in fetching the webpage, now how to extract ids and classes from the webpage?

Answer 1

A better way to do so would be using the BeautifulSoup (bs4) web-scraping library, and requests.

After having installed both using pip, you can start as so:

import requests 
from bs4 import BeautifulSoup

r = requests.get("http://quora.com")
soup = BeautifulSoup(r.content, "html.parser")

To find an element with a specific id:

soup.find(id="your_id")

To find all elements with the "Answer" class:

soup.find_all(class_="Answer")

You can then use .get_text() to remove the html tags and use python string operations to organize your data.