I've created a script to get the html elements from a target page by sending two https requests subsequently. My script can does the thing flawlessly. However, I had to copy the four values from chrome dev tools to fill in the four keys within payload
in order to send the final http requests to reach the target page. This is the starting link and following are the description as to how I could reach the target page.
Find Hotel
button (no need to change dates if chek-out
date is by default at least one day longer than check-in
date).Book Now
button just above it. Now, it should lead you to the target page automatically.Enter Guest Details
, parse the html elements from there I've tried with (working one):
import requests
from bs4 import BeautifulSoup
url = 'https://booking.discoverqatar.qa/SearchHandler.aspx?'
second_url = 'https://booking.discoverqatar.qa/PassengerDetails.aspx?'
params = {
'Module':'H','txtCity':'','hdnCity':'2947','txtHotel':'','hdnHotel':'',
'fromDate':'05/11/2019','toDate':'07/11/2019','selZone':'','minSelPrice':'',
'maxSelPrice':'','roomConfiguration':'2|0|','noOfRooms':'1',
'hotelStandardArray':'63,60,54,50,52,51','CallFrom':'','DllNationality':'-1',
'HdnNoOfRooms':'-1','SourceXid':'MTEzNzg=','mdx':''
}
payload = {
'CallFrom':'MToxNjozOCBQTXxCMkN8MToxNjozOCBQTQ==',
'Btype':'MToxNjozOCBQTXxBfDE6MTY6MzggUE0=',
'PaxConfig':'MToxNjozOCBQTXwyfDB8MnwwfHwxOjE2OjM4IFBN',
'usid':'MToxNjozOCBQTXxoZW54dmkzcWVnc3J3cXpld2lsa2ZwMm18MToxNjozOCBQTQ=='
}
with requests.Session() as s:
r = s.get(url,params=params,headers={"User-agent":"Mozilla/5.0"})
res = s.get(second_url,params=payload,headers={
"User-agent":"Mozilla/5.0",
"Referer":r.url
})
soup = BeautifulSoup(res.text,'lxml')
print(soup)
In the above script I've copied and pasted the value of CallFrom
,Btype
,PaxConfig
and usid
from dev tools to use within payload
.
How can I fill in the values automatically to be used within payload?
Params sent to the second request is Base64 encoded, after decode they are:
'CallFrom':'1:16:38 PM|B2C|1:16:38 PM',
'Btype':'1:16:38 PM|A|1:16:38 PM',
'PaxConfig':'1:16:38 PM|2|0|2|0||1:16:38 PM',
'usid':'1:16:38 PM|henxvi3qegsrwqzewilkfp2m|1:16:38 PM'
At first glance, you already notice they are in patterns of:
$date|$param|$date
Where $date
is current time in the format of utc_ts_now.strftime("%I:%M:%S %p")
.
For $param
section of these four parameters, I guess it should be fixed for CallFrom
and Btype
, usid
is the session key, you can find it easily in the previous response.
PaxConfig
is guest counts, it's related to roomConfiguration
you sent in the first request.
To automate the second request, you would generate the decoded value for each parameter first, then encode them with Base64
.
Update:
#!/usr/bin/env python3.7
import base64
from datetime import datetime
import requests
def first_request(session, params):
url = 'https://booking.discoverqatar.qa/SearchHandler.aspx'
r = session.get(url, params=params)
return r
def second_request(session, params):
url = 'https://booking.discoverqatar.qa/PassengerDetails.aspx'
r = session.get(url, params=params)
return r
def main():
params1 = {
'Module': 'H',
'txtCity': '',
'hdnCity': '2947',
'txtHotel': '',
'hdnHotel': '',
'fromDate': '05/11/2019',
'toDate': '07/11/2019',
'selZone': '',
'minSelPrice': '',
'maxSelPrice': '',
'roomConfiguration': '2|0|',
'noOfRooms': '1',
'hotelStandardArray': '63,60,54,50,52,51',
'CallFrom': '',
'DllNationality': '-1',
'HdnNoOfRooms': '-1',
'SourceXid': 'MTEzNzg=',
'mdx': ''
}
session = requests.Session()
_ = first_request(session, params1)
asp_session = session.cookies.get("ASP.NET_SessionId")
params2 = {
# Could related to options "Available" / "On Request"
"Btype": "A",
# Try out other guest counts to make sure
"PaxConfig": params1["roomConfiguration"] * 2,
"CallFrom": "B2C",
"usid": asp_session
}
date = datetime.utcnow().strftime("%I:%M:%S %p")
for k, v in params2.items():
v = "|".join([date, v, date])
v = base64.b64encode(bytes(v, "utf-8")).decode("utf-8")
params2[k] = v
r = second_request(session, params2)
print(r.text)
if __name__ == '__main__':
main()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With