How could I query the result without selenium on Python or Ruby

Question

I'm trying to track the tickets trend

currently, I'm using selenium to simulate submitting forms.

as you know, the selenium is slow and consume much more memory.

However, when you submit the form, it will redirect you to a new url http://makeabooking.flyscoot.com/Flight/Select

Therefore, I don't have the idea how could I do this without the selenium.

Because I couldn't change the form of query like this http://makeabooking.flyscoot.com/Flight/from={TPE}&to={NYK}&date={2015-10-12} to fetch the result.

Any idea to do this with Ruby or Python with SSL proxy and HTTP proxy support ?

sample website: http://www.flyscoot.com/index.php/en/

Answer 1

You can get the curl requests from chrome easily and use it by:

F12 > Network > request > Right Click > Copy As cURL 

curl 'http://makeabooking.flyscoot.com/Flight/Select' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,tr;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8' -H 'Referer: http://www.flyscoot.com/index.php/en/' -H 'Cookie: optimizelyEndUserId=oeu1444666692081r0.12463579000905156; __utmt=1; granify.lasts@1345=1444666699786; ASP.NET_SessionId=lql5yzv1l3yatkh1lcumg2e5; dotrez=1209262602.20480.0000; optimizelySegments=%7B%222335550040%22%3A%22gc%22%2C%222344180004%22%3A%22referral%22%2C%222354350067%22%3A%22false%22%2C%222355380121%22%3A%22none%22%7D; optimizelyBuckets=%7B%223025070068%22%3A%223020800213%22%7D; __utma=185425846.733949751.1444666694.1444666694.1444666694.1; __utmb=185425846.2.10.1444666694; __utmc=185425846; __utmz=185425846.1444666694.1.1.utmcsr=stackoverflow.com|utmccn=(referral)|utmcmd=referral|utmcct=/questions/33084039/how-could-i-query-the-result-without-selenium-on-python-or-ruby; granify.uuid=68b0d8e8-d068-40d8-9068-3098e870b858; granify.session@1345=1444666699786; granify.flags@1345=8; _gr_ep_sent=1; _gr_er_sent=1; granify.session_init@1345=2; optimizelyPendingLogEvents=%5B%5D' -H 'Connection: keep-alive' -H 'X-FirePHP-Version: 0.0.6' -H 'Cache-Control: max-age=0' --compressed

If you can set the headers and cookies info correctly you can use Python requests. If you want to convert it to the Python requests, you can use the this link. By this way you can simulate the browser. See the pyton requests:

cookies = {
    'optimizelyEndUserId': 'oeu1444666692081r0.12463579000905156',
    '__utmt': '1',
    'granify.lasts@1345': '1444666699786',
    'ASP.NET_SessionId': 'lql5yzv1l3yatkh1lcumg2e5',
    'dotrez': '1209262602.20480.0000',
    'optimizelySegments': '%7B%222335550040%22%3A%22gc%22%2C%222344180004%22%3A%22referral%22%2C%222354350067%22%3A%22false%22%2C%222355380121%22%3A%22none%22%7D',
    'optimizelyBuckets': '%7B%223025070068%22%3A%223020800213%22%7D',
    '__utma': '185425846.733949751.1444666694.1444666694.1444666694.1',
    '__utmb': '185425846.2.10.1444666694',
    '__utmc': '185425846',
    '__utmz': '185425846.1444666694.1.1.utmcsr=stackoverflow.com|utmccn=(referral)|utmcmd=referral|utmcct=/questions/33084039/how-could-i-query-the-result-without-selenium-on-python-or-ruby',
    'granify.uuid': '68b0d8e8-d068-40d8-9068-3098e870b858',
    'granify.session@1345': '1444666699786',
    'granify.flags@1345': '8',
    '_gr_ep_sent': '1',
    '_gr_er_sent': '1',
    'granify.session_init@1345': '2',
    'optimizelyPendingLogEvents': '%5B%5D',
}

headers = {
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'en-US,en;q=0.8,tr;q=0.6',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Referer': 'http://www.flyscoot.com/index.php/en/',
    'Connection': 'keep-alive',
    'X-FirePHP-Version': '0.0.6',
    'Cache-Control': 'max-age=0',
}

requests.get('http://makeabooking.flyscoot.com/Flight/Select', headers=headers, cookies=cookies)

If you save the result, you can see that result is as done via browser (open stack.html):

r = requests.get('http://makeabooking.flyscoot.com/Flight/Select', headers=headers, cookies=cookies
f = open("stack1.html", "w")
f.write(r.content)