Program does not exit. How to find out what python is doing?

Question

I have a python script which is working fine so far. However, my program does not exit properly. I can debug until and I'm returning to the end, but the programm keeps running.

main.main() does a lot of stuff: it downloads (http, ftp, sftp, ...) some csv files from a data provider, converts the data into a standardized file format and loads everyting into the database.

This works fine. However, the program does not exit. How can I find out, where the programm is "waiting"? There exist more than one provider - the script terminates correctly for all providers except for one (sftp download, I'm using paramiko)

if __name__ == "__main__":

    main.log = main.log2both
    filestoconvert = []
    #filestoconvert = glob.glob(r'C:\Data\Feed\ProviderName\download\*.csv')
    main.main(['ProviderName'], ['download', 'convert', 'load'], filestoconvert)

I'm happy for any thoughts and ideas!

Answer 1

If your program does not terminate it most likely means you have a thread still working.

To list all the running threads you can use :

threading.enumerate()

This function lists all Thread that are currently running (see documentation)

If this is not enough you might need a bit of script along with the function (see documentation):

sys._current_frames()

So to print stacktrace of all alive threads you would do something like :

import sys, traceback, threading
thread_names = {t.ident: t.name for t in threading.enumerate()}
for thread_id, frame in sys._current_frames().iteritems():
    print("Thread %s:" % thread_names.get(thread_id, thread_id))
    traceback.print_stack(frame)
    print()

Good luck !

Answer 2

You can involve the python debugger for a script.py with

python -m pdb script.py

You find the pdb commands at http://docs.python.org/library/pdb.html#debugger-commands