I have a website I would like to click a button on then scrape the website using python the html code between the button is:
<span id="exchange-testing" class="exchange-input nav-link" data track="&lid=testing&lpos=site_settings" data-value="testing">Testing</span>
Is this possible? I am able to scrape all the data I need from the page but I need to click the button first.
Any help would be appreciated
Basically, you have two options:
high-level approach: automate a real browser using selenium
or, in other words, make the browser repeat all the user actions needed to get to the page with the desired data.
low-level approach: when you click the button, investigate what is happening under the hood - explore the "Network" tab of browser developer tools and see what requests are being made. Then, simulate them in your scraper. Here, you may consider using tools like requests
, mechanize
for making requests, handling scraping sessions, submitting forms etc and tools like BeautifulSoup
, lxml.html
for html parsing. Also, Scrapy
web-scraping framework is a must see.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With