get the executed javascript content in python [duplicate]

Question

Is there any way to get the executed javascript contents from a webpage? I have tried requests + BeautifulSoup, mechanize, these yield me with "source code" of the webpage and not the executed javascript. For example, this website :- http://listen.tidal.com/login

As you can see, in the source code, there is un-execute JS, but, when you inspect the element, you'll see the executed code.

Now, is there any way I could get that EXECUTED code in python? Hints please, because I have tried emulating a browser using mechanize and it does the same like reuqests. Thank You

Answer 1

In fact, JavaScript engine is needed for execution of javascript. Python is a language with its own interpreter(compiler!) to execute python code. These are two different technology. So if you want to execute javascript from python, python must have api or sort of bindings that interacts with the engine executes javascript. Fortunately python has interactivity with several JS Engine for implementing web related works(testing etc.). This interoperable JS can be divided into two groups as below-

1. Browser without Graphical User Interface(GUI) aka Headless browser: e.g. PhantomJS a Webkit rendering engine based headless browser, SlimerJS a Gecko rendering engine based headless browser for more see here. You can interoperate with PhantomJS with selenium ( a glue between python and PhantomJS) or you can use PyQt and use python to run JS like here.
2. Browser with Graphical User Interface(GUI): e.g. Firefox, Chromium, Safari etc. In this case also you can execute JS through selenium python.

Simple execution example of JS in selenium python as below-

from selenium import webdriver
#define driver- firefox, chrome  or phantomjs etc.
driver = webdriver.Firefox()
#Open the url
driver.get('https://www.google.com')
#see how javascript simple alert is being executed
driver.execute_script("alert('hello world');")
#close the driver  i.e. closing opened Firefox instance!
driver.close()

Answer 2

Just to highlight - Python doesn't execute your Js code, but runtime does. Here is an example of the python module that picks available runtime and evaluates code for you.

Look at PyExecJS, here you can find some examples, but take into account that it might not contain any browser APIs like DOM, Html5 Api, etc. It's mostly based on js engine capabilities.

Another big question, what is the reason to evaluate code in the python?