Python requests isn't giving me the same HTML as my browser is

Question

I am grabbing a Wikia page using Python requests. There's a problem, though: the requests request isn't giving me the same HTML as my browser is with the very same page.

For comparison, here's the page Firefox gets me, and here's the page requests fetches (download them to view - sorry, no easy way to just visually host a bit of HTML from another site).

You'll note a few differences (super unfriendly diff). There are some small things, like attributes beinig ordered differently and such, but there are also a few very, very large things. Most important is the lack of the last six <img>s, and the entirety of the navigation and footer sections. Even in the raw HTML it looks like the page cut off abruptly.

Why is this happening, and is there a way to fix it? I've thought of a bunch of things already, none of which have been fruitful:

• Request headers interfering? Nope, I tried copying the headers my browser sends, User-Agent and all, 1:1 into the requests request, but nothing changed.
• JavaScript loading content after the HTML is loaded? Nah. Even with JS disabled, Firefox gives me the "good" page.
• Uh... well... what else could there be?

It'd be amazing if you know a way this could happen and a way to fix it. Thank you!

Answer 1

I had a similar issue:

• Identical headers with Python and through the browser
• JavaScript definitely ruled out as a cause

To resolve the issue, I ended up swapping out the requests library for urllib.request.

Basically, I replaced:

import requests  session = requests.Session() r = session.get(URL) 

with:

import urllib.request  r = urllib.request.urlopen(URL) 

and then it worked.

Maybe one of those libraries is doing something strange behind the scenes? Not sure if that's an option for you or not.