Hi, I followed and understood this article on how to read content from sites and it worked perfectly: geeksforgeeks.org:Reading selected webpage content using Python Web Scraping
But when I changed my code to work with another site it doesn't return any value. I am trying to get those Value1 and Value2 etc.. as shown below.
Please Note: It's Legal to read content from that web page.
import requests
from bs4 import BeautifulSoup
# the target we want to open
url='https://hackerone.com/directory?offers_bounties=true&asset_type=URL&order_direction=DESC&order_field=started_accepting_at'
#open with GET method
resp=requests.get(url)
#http_respone 200 means OK status
if resp.status_code==200:
print("Successfully opened the web page")
print("The news are as follow :-\n")
# we need a parser,Python built-in HTML parser is enough .
soup=BeautifulSoup(resp.text,'html.parser')
# l is the list which contains all the text i.e news
l=soup.find("tr","spec-directory-entry daisy-table__row fade fade--show")
#now we want to print only the text part of the anchor.
#find all the elements of a, i.e anchor
for i in l:
print(i.text)
else:
print("Error")
Here's the website source code:
<tr class="spec-directory-entry daisy-table__row fade fade--show">
<a href="/livestream" class="daisy-link spec-profile-name">Value1</a>
<tr class="spec-directory-entry daisy-table__row fade fade--show">
<a href="/livestream" class="daisy-link spec-profile-name">Value2</a>
<tr class="spec-directory-entry daisy-table__row fade fade--show">
.
.
.
JavaScript needed to render the webpage contents. Using prerenderio service is an easy/light way to get the data you're looking for from the page.
import requests
from bs4 import BeautifulSoup
# the target we want to open
# changed to use prerenderio service
url='http://service.prerender.io/https://hackerone.com/directory?offers_bounties=true&asset_type=URL&order_direction=DESC&order_field=started_accepting_at'
#open with GET method
resp=requests.get(url)
#http_respone 200 means OK status
if resp.status_code==200:
print("Successfully opened the web page")
print("The news are as follow :-\n")
# we need a parser,Python built-in HTML parser is enough .
soup=BeautifulSoup(resp.text,'html.parser')
# l is the list which contains all the text i.e news
l=soup.find("tr","spec-directory-entry daisy-table__row fade fade--show")
#now we want to print only the text part of the anchor.
#find all the elements of a, i.e anchor
for i in l:
print(i.text)
else:
print("Error")
The returned data from the above code:
Successfully opened the web page
The news are as follow :-
LivestreamManaged
04 / 2019
73
$100
$150-$250
Edited: Responding to Ahmad's comment
Here's the code to only get the values for "Livestream" table row.
import requests
from bs4 import BeautifulSoup
# the target we want to open
# changed to use prerenderio service
url='http://service.prerender.io/https://hackerone.com/directory?offers_bounties=true&asset_type=URL&order_direction=DESC&order_field=started_accepting_at'
#open with GET method
resp=requests.get(url)
#http_respone 200 means OK status
if resp.status_code==200:
print("Successfully opened the web page")
print("The news are as follow :-\n")
# we need a parser,Python built-in HTML parser is enough .
soup=BeautifulSoup(resp.text,'html.parser')
# l is the list which contains all "tr" tags
l=soup.findAll("tr","spec-directory-entry daisy-table__row fade fade--show")
# looping through the list of table rows
for i in l:
# checking if the current row is for 'Livestream'
if i.find('a').text == 'Livestream':
# printing the row's values except the first "td" tag
for e in i.findAll('td')[1:]:
print(e.text)
else:
print("Error")
Result:
Successfully opened the web page
The news are as follow :-
04 / 2019
73
$100
$150-$250
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With