Invalid argument error from background process when main script stops

Question

I have this code to get tweets by running a background process. The following script is run from main script using subprocess.Popen function. So that the main script will stop executing after invoking the background process script.

def start_listner(unique_id, keyword, limit=200):
    class CustomStreamListener(tweepy.StreamListener):

        def __init__(self, api):
            logger.info('runnning')
            self.api = api
            super(tweepy.StreamListener, self).__init__()

            #setup rabbitMQ Connection


        def on_status(self, status):
            print status.text.encode('utf-8'), "\n"
            #queue the tweet and writes the tweet to the log

        def on_error(self, status_code):
          #some code to not kill the stream

        def on_timeout(self):
          #some code to not kill the stream

    sapi = tweepy.streaming.Stream(auth, CustomStreamListener(api))
    try:
        logger.info('tracking started')
        logger.info(keyword)
        logger.info(type(keyword))
        kw = keyword
        sapi.filter(track=[kw])  # keeps listening to the streaming api
    except Exception, err:
        logger.info(kw) # fails at this place when main py stops
        logger.info(err)

if __name__ == "__main__":
    logger.info("just now started")
    try:
        a = str(sys.argv[1])
        b = str(sys.argv[2])
        #c = int(sys.argv[5])
        logger.info(a)
        logger.info(b)
    except Exception, err:
        logger.info("inside main")
    start_listner(a, b)

According to the highest voted answer here I use the following main script to invoke the StreamingAnalytics.py(above code)

import time
import subprocess
subprocess.Popen(["python", "StreamingAnalytics.py", 'SriLankaQ', 'lanka'])

print 'I could escape.........'
time.sleep( 15 )

I have added a sleep so the tweets will be successfully added to the RabbitMQ queue during that time. But as soon as the main script stops the background process prints the following error.

2015-12-22 16:28:16,559 - main - INFO - {'text': 'RT @Dory: lanka singing Hotline bling \xf0\x9f\x98\x82\xf0\x9f\x98\x82 'source': u'Twitter for iPhone'}

2015-12-22 16:28:17,752 - main - INFO - lanka

2015-12-22 16:28:17,752 - main - INFO - [Errno 22] Invalid argument

UPDATE: Since I thought its an issue in passing arguments I removed the use of arguments by writing them to a file by main script and reading the file from background process file. So,

subprocess.Popen(["python", "StreamingAnalytics.py"])

But still the same error comes. Using the traceback module I could print more information on this error.

2015-12-24 11:01:16,562 - __main__ - INFO - Traceback (most recent call last):
File "StreamingAnalytics.py", line 84, in <module>
    sapi.filter(track=[keyword])
File "C:\Python27\lib\site-packages\tweepy\streaming.py", line 445, in filter
    self._start(async)
File "C:\Python27\lib\site-packages\tweepy\streaming.py", line 361, in
_start
    self._run()
File "C:\Python27\lib\site-packages\tweepy\streaming.py", line 294, in _run
    raise exception IOError: [Errno 22] Invalid argument

Answer 1

Your traceback is obscured by tweetpy.

My recommendation:

• edit tweepy/streaming.py
• find the two lines exception = ...
• add logging.exception("foobar") right before or after
• run again
• post complete traceback