Menu
I am trying to use the Requests framework with python (http://docs.python-requests.org/en/latest/) but the page I am trying to get to uses javascript to fetch the info that I want.
I have tried to search on the web for a solution but the fact that I am searching with the keyword javascript most of the stuff I am getting is how to scrape with the javascript language.
Is there anyway to use the requests framework with pages that use javascript?
250
Answer: No, Python cannot replace Javascript. In fact, the two languages complement each other. Javascript is used as a client-side scripting language, whereas Python is mostly used as a server-side scripting language.
Alternatively, we could also use BeautifulSoup on the rendered HTML (see below). However, the awesome point here is that we can create the connection to this webpage, render its JavaScript, and parse out the resultant HTML all in one package!
Good news: there is now a requests module that supports javascript: https://pypi.org/project/requests-html/
from requests_html import HTMLSession session = HTMLSession() r = session.get('http://www.yourjspage.com') r.html.render() # this call executes the js in the page As a bonus this wraps BeautifulSoup, I think, so you can do things like
r.html.find('#myElementID').text which returns the content of the HTML element as you'd expect.
96
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
