Pydev PyUnit issue when using thread.join to ensure all threads are joined

Question

I'm having a issue when using pydev for testing where my tests keep hanging. I've dug into the issue and know what the root cause is. I've provided samples of the code below that can be used to reproduce the issue.

I'm mainly testing on Centos 6.3, python 2.7, eclipse juno, pydev 2.7.1, however the issue also occours on windows 7 with similar setup.

I have a python script that operates as a process launcher for server different threads (all inside third party libraries, so I cant resign that side of the system).

To ensure that all the threads have finished at the end of my process.py i have a section of code that attempts to join all threads before exiting.

 for t in threading.enumerate():               
     if t.getName() != 'MainThread':
         t.join()   

This works fine in normal production code.

The issue occours when running tests in PyUnit in eclipse with pydev. Extra threads are being added to python that result in my tests hanging.

If i launch my program using Run As -> Python Run , my code runs as expected and exits fine. If I launch my program using Run As -> Python unit-test, the test always hangs.

If i look at whats threads are available the issue becomes clear. Using the test code sample provided, I can see when just running the test as a python run, the following threads are shown (as expected)

Thread:  <bound method _MainThread.getName of <_MainThread(MainThread, started 140268135126784)>> 
Thread:  <bound method ThreadClass.getName of <ThreadClass(Thread A, started 140268006471424)>>
Thread:  <bound method ThreadClass.getName of <ThreadClass(Thread B, started 140267927631616)>>

When I run my tests as a unit-test

Thread:  <bound method _MainThread.getName of <_MainThread(MainThread, started 139904571213568)>>
Thread:  <bound method WriterThread.getName of <WriterThread(pydevd.Writer, started daemon 139904379361024)>>
Thread:  <bound method ThreadClass.getName of <ThreadClass(Thread A, started 139904130479872)>>
Thread:  <bound method ThreadClass.getName of <ThreadClass(Thread B, started 139904119990016)>>
Thread:  <bound method PyDBCommandThread.getName of <PyDBCommandThread(pydevd.CommandThread, started daemon 139904358381312)>>
Thread:  <bound method ReaderThread.getName of <ReaderThread(pydevd.Reader, started daemon 139904368871168)>>
Thread:  <bound method ServerComm.getName of <ServerComm(Thread-4, started 139904345736960)>>

The extra threads added by python appear to break this code. When my code attempts to join the ServerComm or pydev.Writer it hangs.

I know I could attempt to not join these threads by name, however that way i'm changing production code to deal with this and I'm not too keen on that solution. Has anyone else come across this before and found a good workaround? Any help in this would be much appreciated. Below is sample code for the issue.

Sample test_process.py

import sys
import traceback
import unittest

class TestUBProcessManager(unittest.TestCase):
    def setUp(self):
        pass

    def runTest(self):
        globalsDict = {'sys':sys, '__name__':'__main__'}

        haveException = False
        try:
            execfile('Process.py', globalsDict)
        except Exception, detail:
            haveException = True
            traceback.print_exc()

        self.assertFalse(haveException)

if __name__ == '__main__':
    unittest.main()    

Sample Process.py

import threading                                                                                                    
import time                                                                                                         

class ThreadClass(threading.Thread):                                                                                

    def __init__(self, threadName, threadRunCount,threadSleep):                                                     
        threading.Thread.__init__(self)                                                                             
        self.name = threadName;                                                                                     
        self.runCount = threadRunCount;                                                                             
        self.sleepTime = threadSleep;                                                                               

    def run(self):                                                                                                  
        for i in range (1,self.runCount):                                                                           
            print "\t",self.name, "Working";                                                                        
            time.sleep(self.sleepTime);                                                                             

class Launcher(object):                                                                                             
    def __init__(self):                                                                                             
        print "Init Threads";                                                                                       
        self.threadA = ThreadClass("Thread A",3,2)                                                                  
        self.threadB = ThreadClass("Thread B",7,2)                                                                  

    def launchProcess(self):                                                                                        
        print "Starting Threads";                                                                                   
        self.threadA.start();                                                                                       
        self.threadB.start();                                                                                       
        time.sleep(2);                                                                                              

if __name__ == '__main__':                                                                                          
    launcher = Launcher()                                                                                           
    try:                                                                                                            
        launcher.launchProcess()                                                                                    
    finally:                                                                                                        
        print "Available Threads Needed To Finish"                                                                  
        for t in threading.enumerate():                                                                             
            print "Thread: ",t.getName                                                                              

        print "Attempt to join threads to ensure all threads are finished"                                          
        for t in threading.enumerate():                                                                             
            print "About To Join : ",t.getName                                                                      
            if t.getName() != 'MainThread':                                                                         
                    t.join()                                                                                        
    print "All Done"                                                                                                

Answer 1

Just incase someone else comes accross this issue. I decided to change my code to check what threads are present when the test starts.

if __name__ == '__main__':                                                                                          
    initialThreads = [];    
    for t in threading.enumerate():
        initialThreads.append(t);

    launcher = Launcher()                                                                                           
    try:                                                                                                            
        launcher.launchProcess()                                                                                    
    finally:                                                                                                        
        print "Available Threads Needed To Finish"                                                                  
        for t in threading.enumerate():                                                                             
        print "Thread: ",t.getName                                                                              

        print "Attempt to join threads to ensure all threads are finished"                                          
        for t in threading.enumerate():                                                                             
            print "About To Join : ",t.getName                                                                      
            if t not in initialThreads:                                                                         
                t.join()                                                                                        
    print "All Done"