Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can't exhaust the load more button to unveil all the headlines

I've tried to keep clicking on more button located at the bottom of a webpage (in it's landing page) to unveil all the headlines. The thing is when I execute my script, It only click once and then stop. How can I keep clicking on that button until there is no more option to click?

Link to that website

This is my script so far:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://www.newsnow.co.uk/h/Sport/Football/Championship/Transfer+News"

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get(link)

while True:
    try:
        loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "[class^='hl_more']")))
        driver.execute_script("arguments[0].scrollIntoView();",loadmore)
        loadmore.click()
    except Exception: break
driver.quit()
like image 256
SIM Avatar asked Dec 03 '25 00:12

SIM


1 Answers

Try below code to simulate required behavior:

header = driver.find_element_by_id("phead")
driver.execute_script('arguments[0].style.position = "absolute";', header)
while True:
    try:    
        loadmore = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.hl_more.bg_tween:not(.nfloading)")))
        driver.execute_script("arguments[0].scrollIntoView();",loadmore)
        loadmore.click()
    except: break
like image 171
Andersson Avatar answered Dec 05 '25 12:12

Andersson



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!