Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why is this python request returning the same value each time?

I'm using they python requests library for the first time, and I am confused. When I run the function below in a for loop, with different base URLs, it appears to receive a response, but the content that is returned is the same for both URLs.

If I look at the API URL in my browser, then I see that it's the content for the first URL that's being returned both times. What am I missing?

base_urls = ['http://kaweki.wikia.com/','http://solarmovie.wikia.com/']


def getEdits(wikiObj, limit=500):     
    payload = {'limit': limit}                             
    r = requests.get('{}api/v1/Activity/LatestActivity'.format(wikiObj),
                     params=payload)
    edits = r.json()
    return edits['items']

for url in base_urls:
    print getEdits(url)  
like image 607
Jeremy Avatar asked Mar 17 '23 20:03

Jeremy


2 Answers

There is a bug on the server side which ignores cache-control headers and such for a period of time.

Introducing sleep of 5 secs (maybe even shorter periods) works around the bug. I've marked the lines that were added below:


import requests
import json
from time import sleep #ADDED

base_urls = ['http://kaweki.wikia.com/', 'http://solarmovie.wikia.com/']


def getEdits(wikiObj, limit=500):       
    payload = {'limit': limit}   
    url = '{}api/v1/Activity/LatestActivity'.format(wikiObj)
    r = requests.get(url, params=payload) 
    edits = json.loads(r.content)
    return edits['items']

for url in base_urls:    
    print getEdits(url)  
    sleep(5) # ADDED

OUTPUT

[{u'article': 1461, u'revisionId': 14, u'user': 26127114, u'timestamp': 1424389645}, {u'article': 1461, u'revisionId': 13, u'user': 26127114, u'timestamp': 1424389322}, {u'article': 1461, u'revisionId': 12, u'user': 26127114, u'timestamp': 1424389172}, {u'article': 1461, u'revisionId': 5, u'user': 26127114, u'timestamp': 1424388924}]
[{u'article': 1461, u'revisionId': 14, u'user': 26127165, u'timestamp': 1424389107}, {u'article': 1461, u'revisionId': 7, u'user': 26127165, u'timestamp': 1424388706}]
like image 197
Nir Alfasi Avatar answered Mar 24 '23 05:03

Nir Alfasi


The API endpoints are "broken". Refreshing the two endpoints in a browser repeatedly has them switching back and forth between two responses. You can replicate it by making refreshing one request half a dozen times, and then refreshing the other request half a dozen times and switching back and forth every half a dozen requests.

Request A:

http://solarmovie.wikia.com/api/v1/Activity/LatestActivity

Request B:

http://kaweki.wikia.com/api/v1/Activity/LatestActivity

Response 1:

{
    items: [
        {
            article: 1461,
            user: 26127114,
            revisionId: 14,
            timestamp: 1424389645
        },
        {
            article: 1461,
            user: 26127114,
            revisionId: 13,
            timestamp: 1424389322
        },
        {
            article: 1461,
            user: 26127114,
            revisionId: 12,
            timestamp: 1424389172
        },
        {
            article: 1461,
            user: 26127114,
            revisionId: 5,
            timestamp: 1424388924
        }
    ],
    basepath: "http://kaweki.wikia.com"
}

Response 2:

{
    items: [
        {
            article: 1461,
            user: 26127165,
            revisionId: 14,
            timestamp: 1424389107
        },
        {
            article: 1461,
            user: 26127165,
            revisionId: 7,
            timestamp: 1424388706
        }
    ],
    basepath: "http://solarmovie.wikia.com"
}
like image 31
abraham Avatar answered Mar 24 '23 03:03

abraham