Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Clicking "Load More" on webpages by using python

I am trying to load the whole content of this link (clicking on المزيد)

I tried so many tutorials like this one here, which teaches how to work with a similar issue (Infinite Scrolling Pages).

My issue is that I couldn't manage to specify the load more class on this page to click it. But, if I am not mistaken, it exists in this part of the webpage source code:

<ng-include src="'loader.html'" class="ng-scope">
    <div class="loading-div ng-scope ng-hide" ng-show="!loadingMoreData">
        <div class="spinner">
            <div class="bounce1"></div>
            <div class="bounce2"></div>
            <div class="bounce3"></div>
        </div>
    </div>
</ng-include>
<div class="block-hed block-link clearfix" data-ng-show="isMoreDisplayed" data-ng-click="getMoreMaterials()">
                    <h4 class="link-border-color"><a href="javascript:void(0)">المزيد </a></h4>
                </div>

I do not necessarily need to implement any function like "click()" or "perform()". Any way to show the whole content under load more button is considered.


by the way, this is my code so far:

from selenium import webdriver
browser = webdriver.Chrome("/home/aziz/anaconda3/lib/python3.6/site-packages/chromedriver/chromedriver")
am = browser.get("https://sabq.org/%D8%A7%D9%84%D9%85%D9%85%D9%84%D9%83%D8%A9/%D8%A5%D9%82%D8%AA%D8%B5%D8%A7%D8%AF")

UPDATE I used this link here to solve the issue, but none of the tutorials worked.

find_element_by_id
find_element_by_name
find_element_by_xpath
find_element_by_link_text
find_element_by_partial_link_text
find_element_by_tag_name
find_element_by_class_name
find_element_by_css_selector
like image 815
Abdulaziz Al Jumaia Avatar asked Oct 15 '22 13:10

Abdulaziz Al Jumaia


2 Answers

I finally managed to solve the problem. I just added this line after my code

while True:
    browser.find_elements_by_link_text('المزيد')[1].click()

and the page started infinitely loading all of the articles. I didn't really know that المزيد itself is clickable. I thought there is a link included in it.

like image 172
Abdulaziz Al Jumaia Avatar answered Oct 30 '22 12:10

Abdulaziz Al Jumaia


The correct/better way to do it is to use WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)
while True:
  element = wait.until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, 'المزيد')))
  element.click()
like image 43
Tarun Lalwani Avatar answered Oct 30 '22 11:10

Tarun Lalwani