Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to open the option items of a select tag (dropdown) in different tabs/windows?

I'm trying to scrape this website using Python and Selenium, it requires you to select a date from drop-down box then click search to view the planning applications.

URL: https://services.wiltshire.gov.uk/PlanningGIS/LLPG/WeeklyList.

I have the code working to select the first index of the drop-down box and press search. How would I open multiple windows for all the date options in the drop-down box or go through them one by one so I can scrape it?

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options


options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome('/Users/weaabduljamac/Downloads/chromedriver', 
chrome_options=options)

url = 'https://services.wiltshire.gov.uk/PlanningGIS/LLPG/WeeklyList'
driver.get(url)

select = Select(driver.find_element_by_xpath('//*[@id="selWeek"]'))
select.select_by_index(1)

button = driver.find_element_by_id('csbtnSearch')
button.click()

app_numbers = driver.find_element_by_xpath('//*[@id="form1"]/table/tbody/tr[1]/td[1]/a').text
print(app_numbers)

Drop-down box HTML:

<select class="formitem" id="selWeek" name="selWeek">
   <option selected="selected" value="2018,31">Week commencing Monday 30 July 2018</option>
   <option value="2018,30">Week commencing Monday 23 July 2018</option>
   <option value="2018,29">Week commencing Monday 16 July 2018</option>
   <option value="2018,28">Week commencing Monday 9 July 2018</option>
   <option value="2018,27">Week commencing Monday 2 July 2018</option>
   <option value="2018,26">Week commencing Monday 25 June 2018</option>
   <option value="2018,25">Week commencing Monday 18 June 2018</option>
   <option value="2018,24">Week commencing Monday 11 June 2018</option>
   <option value="2018,23">Week commencing Monday 4 June 2018</option>
   <option value="2018,22">Week commencing Monday 28 May 2018</option>
</select>
like image 535
Abdul Jamac Avatar asked Aug 07 '18 09:08

Abdul Jamac


1 Answers

As per your question you won't be able to open multiple windows for different drop-down options as the <options> tags doesn't contains any href attribute. They will always render the new page in the same browser window.

However to select a date from the Dropdown and then click() Search to view the planning applications you can use the following solution:

  • Code Block:

    from selenium import webdriver
    from selenium.webdriver.support.ui import Select
    from selenium.webdriver.chrome.options import Options
    
    options = Options()
    options.add_argument('--headless')
    options.add_argument("start-maximized")
    options.add_argument('disable-infobars')
    driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
    url = 'https://services.wiltshire.gov.uk/PlanningGIS/LLPG/WeeklyList'
    driver.get(url)
    
    select = Select(driver.find_element_by_xpath("//select[@class='formitem' and @id='selWeek']"))
    list_options = select.options
    for item in range(len(list_options)):
        select = Select(driver.find_element_by_xpath("//select[@class='formitem' and @id='selWeek']"))
        select.select_by_index(str(item))
        driver.find_element_by_css_selector("input.formbutton#csbtnSearch").click()
        print(driver.find_element_by_xpath('//*[@id="form1"]/table/tbody/tr[1]/td[1]/a').text)
        driver.get(url)
    driver.quit()
    
  • Console Output:

    18/06760/FUL
    18/07187/LBC
    18/06843/FUL
    18/06705/FUL
    18/06449/FUL
    18/05534/FUL
    18/06030/DEM
    18/05784/FUL
    18/05914/LBC
    18/05241/FUL
    

trivia

To scrape all the links you need to replace:

find_element_by_xpath('//*[@id="form1"]/table/tbody/tr[1]/td[1]/a')

with:

find_elements_by_xpath('//*[@id="form1"]/table/tbody/tr[1]/td[1]/a')
like image 112
undetected Selenium Avatar answered Sep 18 '22 21:09

undetected Selenium