Python multiprocess debugging

Question

I'm trying to debug a simple python application but no luck so far.

import multiprocessing

def worker(num):
    for a in range(0, 10):
        print a

if __name__ == '__main__':
    for i in range(5):
        p = multiprocessing.Process(target=worker, args=(i,))
        p.start()

I want to set a breakpoint inside the for-loop to track the values of 'a' but non of the tools that I tried are able to do that. So far I tried debuging with:

• PyCharm and get the following error: ImportError: No module named pydevd - http://youtrack.jetbrains.com/issue/PY-6649 It looks like they are still working on a fix for this and from what I understand, no ETA on this
• I also tried debuging with Winpdb - http://winpdb.org but it simply won't go inside my 'worker' method and just print the values of 'a'

I would really appreciate any help with this!

Answer 1

I found it very useful to replace multiprocessing.Process() with threading.Thread() when I'm going to set breakpoints. Both classes have similar arguments so in most cases they are interchangeable.

Usually my scripts use Process() until I specify command line argument --debug which effectively replaces those calls with Thread(). That allows me to debug those scripts with pdb.

Answer 2

I've rarely needed to use a traditional debugger when attempting to debug Python code, preferring instead to liberally sprinkle my code with trace statements. I'd change your code to the following:

import multiprocessing
import logging

def worker(num):
    for a in range(0, 10):
        logging.debug("(%d, %d)" % (num, a))

if __name__ == '__main__':
    logging.basicConfig(level=logging.DEBUG)
    for i in range(5):
        p = multiprocessing.Process(target=worker, args=(i,))
        logging.info("Starting process %d" % i)
        p.start()

In production, you disable the debug trace statements by setting the trace level to logging.WARNING so you only log warnings and errors.

There's a good basic and advanced logging tutorial on the official Python site.

Answer 3

I copied everything in /Applications/PyCharm\ 2.6\ EAP.app/helpers/pydev/*.py to site-packages in my virtualenv and it worked for my (I'm debugging celery/kombu, breakpoints work as expected).

Answer 4

You should be able to do it with remote-pdb.

from multiprocessing import Pool


def test(thing):
  from remote_pdb import set_trace
  set_trace()
  s = thing*2
  print(s)
  return s



if __name__ == '__main__':
  with Pool(5) as p:
    print(p.map(test,['dog','cat','bird']))

Then just telnet to the port thats listed in the log.

Example:

RemotePdb session open at 127.0.0.1:54273, waiting for connection ...

telnet 127.0.0.1 54273
<telnet junk>
-> s = thing*2
(Pdb) 

or

nc -tC 127.0.0.1 54273

-> s = thing * 2
(Pdb)

You should be able to debug the process at that point.