How to call JavaScript function using BeautifulSoup and Python

Question

I am performing web scraping to grab data from a website as part of my project. I can make the request and grab the data which is present in the dom. However, some data is getting rendered on javascript onClick function.

One way could be, using the selenium to click on the link (which calls the javascript function) and grab the rendered data, but this process is time-consuming, and I don't want to open the browser.

Is there any way other than selenium to achieve this?

Website: http://catalog.fullerton.edu/preview_entity.php?catoid=16&ent_oid=1849

In the courses section of this webpage, all the courses are hyperlinks, and as soon as someone clicks on the courses, a javascript method gets called. I need the data which gets rendered after the javascript function call.

Answer 1

You can't do this using BeautifulSoup alone. This module was created to scrape HTML (Hyper Text Markup Language) not JavaScript, CSS or any other web language.

It can extract between <script></script> tags (which will be quite useful) but beyond this BeautifulSoup is not what you need.

To call a JavaScript functions you will need a headless browser such as PhantomJS or Selenium. There have also been attempts to parse JavaScript as well as using regex (which is not a good idea) and using other methods (recommended) some methods are described in this question and may be useful.