web scraping vaadin python

Question

I'm trying to scrape a site created using vaadin using python. This is the code I use:

requests.get('http://rnb.osim.ro/?pn=').text

but this is the result which contains no useful information:

    <noscript>
    You have to enable javascript in your browser to use an application built with Vaadin.
   </noscript>
  </div>
  <script type="text/javascript" src="./VAADIN/vaadinBootstrap.js"></script>
  <script type="text/javascript">//<!

Do you know how I can get the data I need from a vaadin site?

Answer 1

This is happening because requests can't really execute the JavaScript inside the website. As you might think, requests is just a request and not a browser that can handle JS and work with frameworks for the front end (i.e. Angular, React, Ajax). To scrape this modern and robust websites I personally recommend to use scrapy library. It's designed specially for scraping and can handle with the JavaScript a little bit. And even if it couldn't, you can still use the selenium web driver to fully emulate a browser.