Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Click button on website then scrape web page

I have a website I would like to click a button on then scrape the website using python the html code between the button is:

 <span id="exchange-testing" class="exchange-input nav-link" data track="&amp;lid=testing&amp;lpos=site_settings" data-value="testing">Testing</span>

Is this possible? I am able to scrape all the data I need from the page but I need to click the button first.

Any help would be appreciated

like image 582
user3325317 Avatar asked Nov 09 '14 00:11

user3325317


1 Answers

Basically, you have two options:

  • high-level approach: automate a real browser using selenium or, in other words, make the browser repeat all the user actions needed to get to the page with the desired data.

  • low-level approach: when you click the button, investigate what is happening under the hood - explore the "Network" tab of browser developer tools and see what requests are being made. Then, simulate them in your scraper. Here, you may consider using tools like requests, mechanize for making requests, handling scraping sessions, submitting forms etc and tools like BeautifulSoup, lxml.html for html parsing. Also, Scrapy web-scraping framework is a must see.

like image 183
alecxe Avatar answered Sep 28 '22 08:09

alecxe