Menu
Can I use Watir to scrape data from a website (AJAX used) but on a linux server without monitor? (linode.com) ?
814
Watir (Web Application Testing in Ruby), pronounced as "Water" is an open source tool developed using Ruby which helps in automating web application no matter which language the application is written. The browsers supported are Internet Explorer, Firefox, Chrome, Safari, and Edge.
Watir-WebDriver (Watir is short for Web Application Testing in Ruby) is a Ruby gem which allows you to automate your browser (make it click a button, submit a form, wait for some text to appear before continuing, and so on).
There are several ways to do this:
Use HtmlUnit, either Celerity or watir-webdriver (through the remote Selenium2/WebDriver server).
Use a real browser + a virtual X server (Xvfb). I'd recommend using watir-webdriver's Firefox driver and the Headless gem for a simple way to control this from Ruby.
This is basically a tradeoff between speed and realism. Personally I'd go with #2 if the site has any complex JavaScript or invalid HTML, but both approaches could be worth investigation.
For the future, I'm keeping an eye on this project, which looks like a terrific idea.
178
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
