Twitter Streaming on GAE

Question

Why am i getting this error? and here is how i call the streaming API.

import tweepy
import sys

creds = json.loads(open('credential.json').read())
tw_consumer_key = creds['tw_consumer_key']
tw_consumer_secret = creds['tw_consumer_secret']
tw_access_token = creds['tw_access_token']
tw_access_token_secret = creds['tw_access_token_secret']

try:
    auth = tweepy.OAuthHandler(tw_consumer_key, tw_consumer_secret)
    auth.set_access_token(tw_access_token, tw_access_token_secret)
    api = tweepy.API(auth)
except Exception:
    service = None
    api = None    

# Query terms
Q = "Better"

class CustomStreamListener(tweepy.StreamListener):
    def on_status(self, status):
        try:
            print "%s\n%s\n%s\n%s\n\n" % (status.text, 
                                      status.author.screen_name, 
                                      status.created_at, 
                                      status.source,)
        except Exception, e:
            print >> sys.stderr, 'Encountered Exception:', e
            pass

    def on_error(self, status_code):
        print >> sys.stderr, 'Encountered error with status code:', status_code
        return True # Don't kill the stream

    def on_timeout(self):
        print >> sys.stderr, 'Timeout...'
        return True # Don't kill the stream



class LiveStream(webapp2.RequestHandler):
    def get(self):
        streaming_api = tweepy.streaming.Stream(auth, CustomStreamListener(), timeout=60)
        self.response.out.write(streaming_api.filter(track=Q))

Probably cause by GAE doesn't allowed the socket, i'm not sure how to apply the query term to get specific filtered streaming tweets. My purpose with this portion of code is to getting live stream with designate keywords. If there any alternative methods, please guide.

Answer 1

On App Engine, httplib connections (and urllib) will use the Google URL fetch service. URL Fetch service means other servers (not the app engine serving instance) perform the request and return the response to the app engine instance, instead of the process itself opening a socket.

I believe you're noticing that the httplib variant running on GAE doesn't provide the sock attribute used to set timeout. The fundamental problem though is that there is no way to get a stream, you will not get any results back into app engine until the response is complete. I haven't tested how it fails, but I expect you would get a DeadlineExceededError from the URL service as Twitter won't close the streaming response.

There are no alternative methods for getting the stream on GAE at the moment. It might work with outbound socket support on a backend. Socket support is currently only available to testers.