Selecting an element by XPath for Selenium Web Scraping

Question

I am trying to select a 'price' from a web page using Selenium (Python). The logic behind the web page is that there are two types of prices - 'regular price' and 'sale price' and they have different XPaths.

The conundrum I have is that I am able to select the parent element which contains both 'regular price' and 'sale price' (that is, if sale price exists for a particular product). I then try to apply a 'Try' and 'Except' to it, so that if the 'sale price' exists - grab that price, otherwise - grab the 'regular price'. However, I do not appear to be able to force it to only look for a 'sale price' within the selected parent element, rather it starts from the beginning of the page with the result that if there is a single 'sale price' on the page -it would apply it to all products.

I have set out the relevant extractors of the code plus the HTML below.

prices = self.driver.find_elements_by_xpath('//*[contains(@class,"row item-block checkout-item isc-productContainer")]')
all_prices_list=[]
for i in prices:
    try:
        sale_price=self.driver.find_element_by_xpath('.//*[contains(@class,"row item-block checkout-item isc-productContainer")]/div[4]/isc-product-price-pdp/span/span/span[1]/span[2]')
        all_prices_list.append(sale_price.text)
    except:
        reg_price=self.driver.find_element_by_xpath('.//*[contains(@class,"row item-block checkout-item isc-productContainer")]/div[4]/isc-product-price-pdp/span/span/span[2]')
        all_prices_list.append(reg_price.text)

The relevant snippet of the HTML code is as follows.

<span class="sale-price" id="salePrice">
                    <span class="price-title">Sale</span>
                    <span class="price-sale" ng-bind="vm.getActualPrice(product)">£0.98</span>
                </span>
<span class="regPrice" id="regPrice">
                    <span class="price-title">Reg:</span>
                    <span class="price-old" ng-bind="vm.getRegularPrice(product)">£0.99</span>
                </span>

There is a a way to only look for 'regular' or 'sale prices' within the selected 'prices' element?

Answer 1

Looks like you are attempting to find sale_price and reg_price inside the parent element but actually not doing so.
So, try the following:

prices = self.driver.find_elements_by_xpath('//*[contains(@class,"row item-block checkout-item isc-productContainer")]')
all_prices_list=[]
for i in prices:
    try:
        sale_price=i.find_element_by_xpath('.//*[contains(@class,"row item-block checkout-item isc-productContainer")]/div[4]/isc-product-price-pdp/span/span/span[1]/span[2]')
        all_prices_list.append(sale_price.text)
    except:
        reg_price=i.find_element_by_xpath('.//*[contains(@class,"row item-block checkout-item isc-productContainer")]/div[4]/isc-product-price-pdp/span/span/span[2]')
        all_prices_list.append(reg_price.text)

Additionally, your locators could be improved

Answer 2

Here is an alternative solution:

using the index of the products
prices = driver.find_elements_by_xpath('//*[contains(@class,"row item-block checkout-item isc-productContainer")]')
all_prices_list = []
for i in range(0,len(prices)):
    try:
        sale_price = driver.find_elements_by_xpath("//span[@ng-bind='vm.getActualPrice(product)']")[i]
        all_prices_list.append(sale_price.text)
    except:
        reg_price = driver.find_elements_by_xpath("//span[@ng-bind='vm.getRegularPrice(product)']")[i]
        all_prices_list.append(reg_price.text)

print (all_prices_list)