Menu
In Python, I want the user to enter a URL in the console prompt (take input and store it in some variable), For example, if the webpage contains this HTML:
<html>
<head>
</head>
<body>
<div>
<h1 class="class_one">First heading</h1>
<p>Some text</p>
<div class="class_two">
<div class="class_three">
<div class="class_one">
<center class="class_two">
<h3 class="class_three">
</h3>
</center>
<center>
<h3 class="find_first_class">
Some text
</h3>
</center>
</div>
</div>
</div>
<div class="class_two">
<div class="class_three">
<div class="class_one">
<center class="class_two">
<h2 class="find_second_class">
</h2>
</center>
</div>
</div>
</div>
</div>
</body>
</html>
Then the CSV should contain rows for each and every class that's in the HTML of the webpage (since classes can appear more than once, so we could have multiple rows for any given class).
Now, I want to generate XPath for all the classes present on the page. What i have written so far is :
import urllib2
from bs4 import BeautifulSoup
result = {}
user_url_list = raw_input("Please enter your urls separated by spaces : \n")
url_list = map(str, user_url_list.split())
for url in url_list:
try:
page = urllib2.urlopen(url)
soup = BeautifulSoup(page, 'html.parser')
user_class_list = raw_input("Please enter the classes to parse for " + url + " separated by spaces : \n")
class_list = map(str, user_class_list.split())
for find_class in class_list:
try:
name_box = soup.find(attrs={'class': find_class})
print(xpath_soup(name_box))
break
except:
print("There was some error getting the xpath of class : " + find_class + " for url : " + url + "\n..trying next class now \n")
continue
except:
print(url + " is not valid, please enter correct full url \n")
continue
print(result)
229
asked Sep 07 '26 14:09
Here is the try/except logic Orhan mentioned. lxml parses the document it is passed and can reference the elements via xpath and extract the classes. After that it is just a simple check if they occur in the desired classes. lxml also allows the reconstruction of the initial xpath via ElementTree.
import csv
import requests
from lxml import etree
target_url = input('Which url is to be scraped?')
page = '''
<html>
<head>
</head>
<body>
<div>
<h1 class="class_one">First heading</h1>
<p>Some text</p>
<div class="class_two">
<div class="class_three">
<div class="class_one">
<center class="class_two">
<h3 class="class_three">
</h3>
</center>
<center>
<h3 class="find_first_class">
Some text
</h3>
</center>
</div>
</div>
</div>
<div class="class_two">
<div class="class_three">
<div class="class_one">
<center class="class_two">
<h2 class="find_second_class">
</h2>
</center>
</div>
</div>
</div>
</div>
</body>
</html>
'''
#response = requests.get(target_url)
#document = etree.parse(response.content)
classes_list = ['find_first_class', 'find_second_class']
expressions = []
document = etree.fromstring(page)
for element in document.xpath('//*'):
try:
ele_class = element.xpath("@class")[0]
print(ele_class)
if ele_class in classes_list:
tree = etree.ElementTree(element)
expressions.append((ele_class, tree.getpath(element)))
except IndexError:
print("No class in this element.")
continue
with open('test.csv', 'w') as f:
writer = csv.writer(f, delimiter=',')
writer.writerows(expressions)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With