Python 3 urllib.request send cookie, fetch results

Question

I am trying to retrieve the 2nd page of these results:

http://raceresults.sportstats.ca/display-results.xhtml?raceid=451

If I click on page 2 at the bottom, it goes to the 2nd page, but the URL stays the same. If I look at the http headers I can see this cookie:

Set-Cookie: sportstats_preferences="{\"raceId\":451,\"firstRow\":40,
\"category\":\"All Categories\",\"chronosStep\":\"INSTRUCTIONS
\",\"facebookLoggedIn\":false,\"twitterLoggedIn\":false,\"fbServiceId
\":0,\"twServiceId\":0,\"unit\":1}"; Version=1; Max-Age=2592000; 
Expires=Sat, 04-Apr-2015 14:30:28 GMT

I can see this is different than the first page in that firstRow is being set to 40.

I am attempting to get this 2nd page in Python 3 with the following code:

#!/usr/bin/env python
import urllib.request
opener = urllib.request.build_opener()
cookie = 'sportstats_preferences="{{\\"raceId\\":451,\\"firstRow\\":40,\\"category\\":\\"All Categories\\",\\"chronosStep\\":\\"INSTRUCTIONS\\",\\"facebookLoggedIn\\":false,\\"twitterLoggedIn\\":false,\\"fbServiceId\\":0,\\"twServiceId\\":0,\\"unit\\":1}}"; Version=1; Max-Age=2592000; Expires=Sat, 04-Apr-2015 04:18:36 GMT'
opener.addheaders = [('Cookie', cookie)]
f = opener.open(url).read().decode("utf-8")
for line in f.splitlines():
    print(line)

But that still just returns the results from the first page. Am I going about this the right way? Any ideas how I can get the results of the 2nd page?

Answer 1

Your best option may be to use Selenium and the corresponding python package. Selenium allows you to use python to open and automatically control a web browser. This would allow you to interact with their next page buttons, and read the results in a python script.

http://www.seleniumhq.org/

https://pypi.python.org/pypi/selenium