Using Python + Selenium to click on next page with while loop

Question

I'm trying to move through different pages on a website called iens. I'm using selenium + python to click on "volgende" (which means "next" in dutch), but I want my program to keep on clicking on next until there are no more pages left using a while loop. So in this case I want my program to end at page 23. Right now I'm able to get to page 2, by closing the cookie pop up message, waiting until it's closed en then clicking on the "Volgende" button.

My code looks like this:

 from selenium.webdriver.support.ui import WebDriverWait
 from selenium import webdriver
 from selenium.webdriver.common.by import By
 from selenium.webdriver.support import expected_conditions as EC

 chrome_path = '/Users/user/Downloads/chromedriver'
 driver = webdriver.Chrome(chrome_path)
 driver.get('https://www.iens.nl/restaurant+utrecht')

 #wait for cookie message
 close_icon = WebDriverWait(driver, 5, 
 0.25).until(EC.visibility_of_element_located([By.CSS_SELECTOR, 
 '.cookiePolicy-close']))

 close_icon.click()

 #wait for cookie message to disappear
 WebDriverWait(driver, 5, 
 0.25).until(EC.invisibility_of_element_located([By.CSS_SELECTOR, 
 '.cookiePolicy-close']))

 click_icon = WebDriverWait(driver, 5, 
 0.25).until(EC.visibility_of_element_located([By.LINK_TEXT, 
 'Volgende']))
 click_icon.click()

The website is called https://www.iens.nl/restaurant+utrecht

Thanks in advance!

Answer 1

Because of automatic scrolling each time you switch to next page, webdriver tries to click Next button, but some other element receives click. You can use this solution:

while True:
    try:
        click_icon = WebDriverWait(driver, 5, 0.25).until(EC.visibility_of_element_located([By.LINK_TEXT, 'Volgende']))
        click_icon.click()
        WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'main:not([style*="margin-top"])')))
    except:
        break

Another simple solution (only if your goal is to reach last page) is to get last page without clicking Next all the time:

driver.find_elements_by_xpath('//div[@class="pagination"]/ul/li/a')[-2].click()