This is the analyzed page https://www.diretta.it/.
In this page the content of the following days is loaded dynamically with the js without changing the URL of the site (you can try it at the top right of the table).
Using puppeteer, with the following code
await page.goto ('https://www.diretta.it/');
it loads the contents of today's page. Is there a way to load the page with tomorrow's content? i have to scrape information from the matches of the following days
the function in js executable from terminal for change day is:
> set_calendar_date ('1')
Puppeteer pages have a handy evaluate() function that lets you execute JavaScript in the Chrome window. The evaluate() function is the most flexible way to interact with Puppeteer, because it lets you control Chrome using browser APIs like document.
evaluate() method. Evaluates a function in the page's context and returns the result. If the function passed to page. evaluteHandle returns a Promise, the function will wait for the promise to resolve and return its value.
Puppeteer lets you automate the testing of your web applications. With it, you can run tests in the browser and then see the results in real-time on your terminal. Puppeteer uses the WebDriver protocol to connect with the browser and simulate user interaction with HTML elements or pages.
What you are looking for is the page.evaluate()
function.
This function lets you run any JS function in the page context.
In simpler terms, running page.evaluate()
is akin to opening Dev tools and writing set_calendar_date('1')
there directly.
Here is a working snippet, don't hesitate to pass {headless: false}
to puppeteer.launch()
if you want to see it working with your own eyes.
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.diretta.it/');
await page.evaluate(() => {
set_calendar_date ('1');
});
await page.waitFor(500); //Wait a bit for the website to refresh contents
//Updated table is now available
})();
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With