Rselenium: down_arrow not working in dropdown menus

Question

It would be awesome if you could help me with the following:

I am using Rselenium and Firefox to explore the following website: https://hcupnet.ahrq.gov/#setup

I am stuck by not being able to scroll down on dropdown menus using the down_arrow, nor using window.scrollBy(0,1200). My objective is to get to a diagnosis down the list that is not visible at first and select it.

The full x-paths used are provided in the code, but if you prefer exploring via your browser the clicks are:

1. Create new analysis
2. Community
3. Single year
4. Arizona
5. Counties
6. Diagnosis Procedure
7. Diagnosis CSS
8. Choose your diagnosis dropdown

I am finding the problem in the choose your diagnosis dropdown. My objective is to select a diagnosis like "27 Cancer of ovary", which is down on the list and not visible at first.

# Stackoverflow question
library(RSelenium)
library(wdman)
library(rvest)
library(tidyverse)

# Open docker -------------------------------------------------------------
remDr <- rsDriver(port = 4443L,
                  browser = "firefox")

remDr2 <- remDr[["client"]]

# Navigate landing website ------------------------------------------------

# Set window size
remDr2$setWindowSize(1280L, 1024L)

# Open landing website
remDr2$navigate("https://hcupnet.ahrq.gov/#setup") #Entering our URL gets the browser to navigate to the page
Sys.sleep(2)

# Function to explore xpaths ----------------------------------------------
click_xpath <- function(xpath){
  webElem <- remDr2$findElement(using = 'xpath', value = xpath )
  remDr2$mouseMoveToLocation(webElement = webElem)
  remDr2$click(1)
  Sys.sleep(2)
}


# Landing Page ------------------------------------------------------------

# Landing page paths
landing_xpaths <- c(
  '/html/body/div[1]/div/section[1]/div[3]/div[1]/button[1]', # 1. Create new analysis,
  '//*[@id="DS_COMM"]', # 2. Press community
  '//*[@id="YEAR_SINGLE"]', #3. Press single year,
  '/html/body/div[2]/div/div/div/div/div/div[2]/div[3]/section/div/div/button/span[1]', #4. Press state dropdown,
  '/html/body/div[6]/div/ul/li[2]/a', #5. Press Arizona
  '//*[@id="CL_COUNTY"]', #6. Press county
  '//*[@id="DP"]', # 7. Press diagnosis procedure
 '/html/body/div[2]/div/div/div/div/div/div[2]/div[6]/section/div/div/button/span[1]', #8 Press dropdown category
  '/html/body/div[6]/div/ul/li[2]/a', #9. Press diagnosis (CSS)
  '/html/body/div[2]/div/div/div/div/div/div[2]/div[7]/section/div/div/button/span[1]' ) #10. Press Diagnosis dropdown

map(landing_xpaths,click_xpath)

# Problem ------------------------------------------------------------

# I am not able to select diagnoses that are not visible, nor I am able to use the down_arrow to go down on the list.

###################################################
# Selecting: Not working
###################################################

click_xpath(" //span[@class='text'][contains(text(),'27 Cancer of ovary')]/parent::a ")

 Error:      Summary: MoveTargetOutOfBounds
     Detail: Target provided for a move action is out of bounds.
     class: org.openqa.selenium.interactions.MoveTargetOutOfBoundsException
     Further Details: run errorDetails method 


###################################################
# Scrolling down: Not working
###################################################
septicemia <- remDr2$findElement(using = 'xpath', value = '/html/body/div[6]/div/ul/li[1]/a')

# Notice that none works
septicemia$sendKeysToElement(list(key = "down_arrow"))
septicemia$sendKeysToElement(list(key = "down_arrow"))
septicemia$executeScript("window.scrollBy(0,1200)")

Answer 1

The problem is, the year dropdown isn't exactly a dropdown. Or at least you can't interact with it like one.

This is what the source looks like: As you can see, it's a series of div and list item tags that make it look like a dropdown.

So how I would go about selecting an option is click the main "dropdown" element to display the list of options

//button[@class='btn dropdown-toggle btn-default']

And now we click the parent element of the option we want (the a tag in this case). For example, if we want to select 2012:

//span[@class='text'][contains(text(),'2012')]/parent::a

---

In response to your update, you have a couple options to select values in that dropdown.

1. Use the search box that's present.

//div[@class='bs-searchbox']/input

Send the desired selection to that input field, then send the enter key to select the first value that's displayed. Note that you'll have to make the dropdown visible first to do this. You could also just click the element that's now visible because the search narrowed down the results.

2. Use executeScript to click on the element that's not visible.

webElem <- remDr2$findElement(using = 'xpath', value = "//span[contains(text(),'27 Cancer of ovary')]/parent::a")
remDr2$executeScript("arguments[0].click();", list(webElem))