Accessing DOM dynamically updated from Chrome extension

Question

I am using a Chrome extension to crawl data from one site. There is a 'load more' div on the site which loads the next 30 records on click. I use the code below to click and load the data:

Content.js

setTimeout(function(){
  var activityTab = document.getElementsByClassName("loadMore")[0];
  activityTab.click();
}, 5000);

It loads the next 30 records and generates a new 'load more' div, but I cannot click on this new dynamically loaded 'load more' div. How I can access dynamically generated HTML in the original site in JavaScript? I can only access data that is available in DOM of the original site on page load.

Answer 1

Naively this is a way to achieve what you want:

function doSomething(startIndex, items) {
    for (let i = startIndex; i < items.length; i++) {
        //Crawl the data
    }
}

var selector = ".myfancystuff"; //This is just test data, change this selector to one which is more appropriate to your case
var items = document.querySelectorAll(selector);
var activityTab = document.getElementsByClassName("loadMore")[0];
activityTab.click();
setTimeout(function() {
    var newItems = document.querySelectorAll(selector);
    if (newItems.length > items.length) {
        doSomething(items.length, items = newItems);
        activityTab.click();
    }
}, 1000);

Of course, you can do it much more elegantly in most of the cases, that is, but more information is needed about that site in that case.