"IncompleteRead" Error when retrieving Twitter Data using Python

Question

While running this program to retrieve Twitter data using Python 2.7.8 :

#imports
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener

#setting up the keys
consumer_key = '…………...'
consumer_secret = '………...'
access_token = '…………...'
access_secret = '……………..'

class TweetListener(StreamListener):
# A listener handles tweets are the received from the stream.
#This is a basic listener that just prints received tweets to standard output

def on_data(self, data):
    print (data)
    return True

def on_error(self, status):
    print (status)

#printing all the tweets to the standard output
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)



stream = Stream(auth, TweetListener())

t = u"سوريا"
stream.filter(track=[t])

after running this program for 5 hours i got this Error message:

Traceback (most recent call last):
  File "/Users/Mona/Desktop/twitter.py", line 32, in <module>
    stream.filter(track=[t])
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 316, in filter
    self._start(async)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 237, in _start
    self._run()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 173, in _run
    self._read_loop(resp)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 225, in _read_loop
    next_status_obj = resp.read( int(delimited_string) )
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 543, in read
    return self._read_chunked(amt)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 612, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 660, in _safe_read
    raise IncompleteRead(''.join(s), amt)
IncompleteRead: IncompleteRead(0 bytes read, 976 more expected)
>>> 

Actually i don't know what to do with this problem !!!

Answer 1

I've just had this problem. The other answer is factually correct, in that it's almost certainly:

• Your program isn't keeping up with the stream
• you get a stall warning if that's the case.

In my case, I was reading the tweets into postgres for later analysis, across a fairly dense geographic area, as well as keywords (London, in fact, and about 100 keywords). It's quite possible that, even though you're just printing it, your local machine is doing a bunch of other things, and system processes get priority, so the tweets will back up until Twitter disconnects you. (This is typically manifests as an apparent memory leak - the program increases in size until it gets killed, or twitter disconnects - whichever is first.)

The thing that made sense here was to push off the processing to a queue. So, I used a redis and django-rq solution - it took about 3 hours to implement on dev and then my production server, including researching, installing, rejigging existing code, being stupid about my installation, testing, and misspelling things as I went.

• Install redis on your machine
• Start the redis server
• Install Django-RQ (or just Install RQ if you're working solely in python)

Now, in your django directory (where appropriate - ymmv for straight python applications) run: python manage.py rqworker &

You now have a queue! You can add jobs to that like by changing your handler like this: (At top of file)

import django_rq

Then in your handler section:

def on_data(self, data):
    django_rq.enqueue(print, data)
    return True

As an aside - if you're interested in stuff emanating from Syria, rather than just mentioning Syria, then you could add to the filter like this:

stream.filter(track=[t], locations=[35.6626, 32.7930, 42.4302, 37.2182]

That's a very rough geobox centred on Syria, but which will pick up bits of Iraq/Turkey around the edges. Since this is an optional extra, it's worth pointing this out:

Bounding boxes do not act as filters for other filter parameters. For example track=twitter&locations=-122.75,36.8,-121.75,37.8 would match any tweets containing the term Twitter (even non-geo tweets) OR coming from the San Francisco area.

From this answer, which helped me, and the twitter docs.

Edit: I see from your subsequent posts that you're still going down the road of using Twitter API, so hopefully you got this sorted anyway, but hopefully this will be useful for someone else! :)

Answer 2

You should check to see if you're failing to process tweets quickly enough using the stall_warnings parameter.

stream.filter(track=[t], stall_warnings=True)

These messages are handled by Tweepy (check out implementation here) and will inform you if you're falling behind. Falling behind means that you're unable to process tweets as quickly as the Twitter API is sending them to you. From the Twitter docs:

Setting this parameter to the string true will cause periodic messages to be delivered if the client is in danger of being disconnected. These messages are only sent when the client is falling behind, and will occur at a maximum rate of about once every 5 minutes.

In theory, you should receive a disconnect message from the API in this situation. However, that is not always the case:

The streaming API will attempt to deliver a message indicating why a stream was closed. Note that if the disconnect was due to network issues or a client reading too slowly, it is possible that this message will not be received.

The IncompleteRead could also be due to a temporary network issue and may never happen again. If it happens reproducibly after about 5 hours though, falling behind is a pretty good bet.