python requests not working with google app engine

Question

I'm trying to fetch a page using python requests with this code in views.py:

s = requests.Session()
r = s.get("https://www.23andme.com/")

I get the error Exceeded 30 redirects. My app is a Google App Engine Django app. I've gotten this code to work in the python console and a django server on pythonanywhere.com, but for some reason it isn't working on google app engine. What could be causing this? Thanks

Edit: It seems like there is another issue with the Requests module in my app. I have this code to add an email to a mailchimp list:

m = mailchimp.Mailchimp(MAILCHIMP_API_KEY)
list_response = m.lists.list()

but if fails with the error HTTPS/SSL is required

Answer 1

Try installing urllib3 (and other stuff, see below).

See, historically there were tons of issues for requests with google app engine (and see issue #498). Those were mostly resolved with urllib3 support for GAE that came with v1.3. It came out a long time ago (the current release is 1.7), so its probably not the issue, however when you initially install requests, it includes urllib3 in a folder called packages and maybe it doesn't include the entirey of it.

I also tried searching the source code for requests and found this interesting:

# Attempt to enable urllib3's SNI support, if possible
try:
    from requests.packages.urllib3.contrib import pyopenssl
    pyopenssl.inject_into_urllib3()
except ImportError:
    pass

Going deeper, the contrib package includes a pyopenssl.py script which requires:

SSL with SNI-support for Python 2.

This needs the following packages installed:

• pyOpenSSL (tested with 0.13)
• ndg-httpsclient (tested with 0.3.2)
• pyasn1 (tested with 0.1.6)

So, to sum up:

1. Install urllib3 and other SSL packages mentioned above, then try running that request you're doing again and see if anything has changed. My guess is that this will (at the very least) help with the mailchimp as it's also complaining about SSL/HTTPS issues.

2. If that doesn't work, try using urllib3 api instead of requests to do the same task and see if that's working. If so, the problem is specifically with the packaged urllib3 that requests is using, which might need some patching.

   import urllib3
   http = urllib3.PoolManager()
   r = http.request('GET', 'https://www.23andme.com/')
   

Sorry this isn't a definite solution, hopefully one of my suggestions will help. Update me as to the progress I'll try and help out as much as I can

Answer 2

According to this discussion, the issue is with netrc on Google App Engine. I was able to work around the problem by using an older version of Requests (1.2.3 in my case, but others may work too.)

Edit: According to this answer, Requests 2.1.0 should work as well.