how to handle elements that load after ajax request in puppeteer

Question

I'm trying to do web scraping using puppeteer. The element I need to handle loads lately. When I click on the search button the result loads in AJAX and I need to pick the element I am trying to pick is in the search results but not in the initial load of the page. The page screenshot it is producing contains search results too and if it output the HTML source I can see the element there too. but not sure why I cannot pick it.

Answer 1

You can use await page.waitForSelector(cssSelector); to ask Puppeteer to wait for any element to be displayed in the UI before continuing on to further steps in your script. By default, the timeout for the wait is 30 seconds but you can set it to any timeout you wish.

So in your case I would:

• Enter your search text into the search bar.
• Click on the search button (this will execute your AJAX call to load the results).
• Use await page.waitForSelector(cssSelector); to ask Puppeteer to wait until some element you are sure will be displayed in the UI after executing the search is visible.
• Now that Puppeteer has registered the element as visible, you know that any actions you wish to perform on it will also execute correctly.

What you might find happens, if you don't use that waitForSelector() call is that the element is displayed but Puppeteer will timeout, for example, if you wish to execute a click command on an element. This is because the timeouts for click events (and other Puppeteer events which interact with elements) is very short and sometimes the script (especially in headless mode) can move to the next instruction too quickly to allow for the UI to update fast enough to keep up.

So by adding the additional waitForSelector calls, you're also making your scripts much more robust. Especially when data is being generated dynamically as they are in your case.