Python: Login to webpage and click button without Selenium

Question

I am writing a python script to connect to a website, login and then click a button at a given time in the following page. I managed to do that with Selenium, but I would like a solution without the need of opening a browser.

I am trying to do it with Requests. I can login and navigate to the protected page, but I struggling to push the button:

import requests

payload = {
    'username': 'MyUsername',
    'password': 'MyPassword'
}

with requests.Session() as s:
    p = s.post("login_URL", data=payload)
    print (p.text)

    r = s.get("A_protected_URL")
    print (r.text)

The button I would like to press is in the "A_protected_URL" and it is defined as follow:

<script type="text/javascript">
function SubmitForm(form) // Button clicked
{
    form.Submit.disabled = true;
    form.Submit.value = "Please wait...";
    return true;
 }
</script>
<div id="submit_button" >
<form action="https://mydummywebsite.com/submitted" 
 method="post" onsubmit="return SubmitForm(this);" >
<input type="submit" class="sub-process-button" name="Submit" value="Submit">
</form>

What is the correct way to send a request to a button that executes the Javascript? Thanks

Answer 1

Don't try to "click" anything, just mimic what the button would do. In this case the button submits a form. That means it will perform a POST request (method="post") to the action URL (action="https://mydummywebsite.com/submitted"). All the fields from the form will be send as the content of the request, but in this case there are no additional fields in the form.

import requests

payload = {
    'username': 'MyUsername',
    'password': 'MyPassword'
}

with requests.Session() as s:
    p = s.post("login_URL", data=payload)
    print (p.text)

    r = s.post("https://mydummywebsite.com/submitted")
    print (r.text)

Answer 2

What you're asking isn't really possible unless someone has implemented a full browser purely in Python, which I doubt. The reason you need something like Selenium is because Selenium is providing Python bindings around an actual web browser (such as Chromium). So the code that you write in Python to interact with Selenium is being translated into a lower level API interaction with the browser (probably Chromium) itself.

Browsers are incredibly complex and responsible for, among other things, rendering your HTML, CSS, JavaScript. Once Chromium receives the HTML you give it, it will have to load its own JavaScript engine to interpret your JavaScript and run it. requests, the Python library, has no knowledge of any of this. It's just interacting with raw HTML.

So if you are crafting HTML and need the embedded JavaScript to be simulated to see how it behaves in a browser, the most realistic option is to just run the HTML through a browser. Selenium is a good choice for that. If you don't actually care about the JavaScript itself and you just want to make sure that your Python server is doing the right thing, you can mimic the POST request as another answer suggests. Let me break it down:

Your Options

• You are writing a Python web server
  • Do you need to test the behavior of the JavaScript in the HTML you return? You should probably use Selenium.
  • Do you need to test the behavior of your webserver, making sure it returns the correct HTML responses to certain requests? You should just mimic GET/POST etc. requests directly with the requests library
• You are trying to write something, anything, to interact with a real-world web page because you are trying to automate some activity
  • Does the web page rely on interaction with JavaScript? You are probably going to need to write your automation in JavaScript instead of Python
  • Does the web page rely purely on HTTP GET/POST/DELETE etc that you can mimic with requests? You could just use Python requests, but in my experience, it's less painful to write JavaScript to interact with web pages.