Python-requests, how to login to website

Question

I am attempting to scrape this website but it requires a login. I am struggling to successfully log in through the use of the requests library in python.

Looking through the form in the html, there are no hidden values and while intercepting http requests in the console, the login post request for forms contains username:"username here" password:"password here".

I have also attempted adjusting the headers, as I read that that some servers might deny access to non browser header types.

Here are my attempts

import requests
from fake_useragent import UserAgent

ua = UserAgent()
headers = {"User-Agent": str(ua.chrome)}

payload = {"username": "username",
           "password": "password"
          }

login = requests.get("https://scsctennis.gametime.net/auth", 
headers=headers)

response = requests.post("https://scsctennis.gametime.net/auth", 
data=payload, cookies=login.cookies, headers=headers)

print(response.text)

and as well

import requests
from fake_useragent import UserAgent

ua = UserAgent()
headers = {"User-Agent": str(ua.chrome)}

payload = {"username": "username",
           "password": "password"
          }

s = requests.session()
login = s.get("https://scsctennis.gametime.net/auth", headers=headers)

response = s.post("https://scsctennis.gametime.net/auth", data=payload, 
headers=headers)

print(response.text)

One thing I have noticed, after the post request, if I attempt to view the cookie - print(response.cookies) there is no cookie, but for the get request, print(login.cookies) there is a cookie.

I have followed and read through this blog and the requests documentation, and have gone through many stackoverflow posts. Any help would be appreciated, thanks.

Edit You are right, it posts to "https://scsctennis.gametime.net/auth/json-index" Here is changed up code with recommendations.

import requests

# headers = {'x-requested-with': 'XMLHttpRequest'}
headers = {"Content-Type": "application/x-www-form-urlencoded; charset=UTF-
8"}

payload = {"username": "username",
           "password": "password"
          }

s = requests.session()
login = s.get("https://scsctennis.gametime.net/auth/json-index", 
headers=headers)
print(login.text)
response = s.post("https://scsctennis.gametime.net/auth/json-index", 
data=payload, headers=headers)
print(response.text)

The response of each respective print statement:

{"code":505,"msg":"The username or password was not recognized. Please check the spelling and try again."}

{"code":202,"msg":"The username or password was not recognized. Please check the spelling and try again.","isStaff":false,"user":{"name":"Vuk"}}

I receive the 505 message simply by visiting the url, not posting to it.

202 message is when I post to the url, however the username/password are correct but it says they are wrong. Not sure why? The "isStaff":false,"user":{"name":"Vuk"} response is correct, as that is my name that is associated with the attempted login credentials, and I am not a staff member.

Any thoughts on how to proceed?

Last Edit: Successfully got it. Thanks for catching that I wasn't posting to correct url! Turns out, the 202 message above is successful. It recognizes my name as belonging to the login credentials, but they just choose to display whatever message. After the post request, if I use a get request to my desired page, I receive a good response. Thanks!

import requests


payload = {"username": "username",
           "password": "password"
           }

s = requests.session()

response = s.post("https://scsctennis.gametime.net/auth/json-index", 
data=payload)
print(response.text)
stuff = s.get("http://scsctennis.gametime.net/scheduling/index/jsoncourtdata/sport/1/date/2017-12-25")` 

print(stuff.text)

Answer 1

I see the form posts credentials to "https://scsctennis.gametime.net/auth/json-index" and get the json in response.

Can you post into this endpoint instead of the one you posted?

Posting fake credentials to this endpoint:

curl "https://scsctennis.gametime.net/auth/json-index" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" -H "Cookie: gametime=ba3725642c5b55fe1123dec46e45e3a7" --data "username=test&passwo
rd=test"

returns error like {"code":505,"msg":"The username or password was not recognized. Please check the spelling and try again."}