Debug slow program; Restart from middle

Question

I have a program which has slow computations and I wish to debug the algorithm. Now it's very tedious to always rerun everything and I'd rather restart from the middle of the program. Can you think of some clever way to achieve this?

The first vague idea is to define checkpoints (where I make a function call) where I save locals and large data with pickle and/or sqlite (sqlite to be able to inspect intermediate data). Later I could try to call the program telling it to restart at a specific checkpoint. However I don't want to split all code chunks between checkpoints just for this purpose.

Has someone a clever idea how to solve this debugging issue?

Answer 1

Unit Tests

This is why Unit Tests exist. Try pyunit with small "sample data", or doctest for extremely simple functions.

Mini Test-Programs

If for some reason you really need interactivity, I usually write an interactive mini-program as what is effectively a unit test.

def _interactiveTest():
    ...
    import code
    code.interact(local=locals())

if __name__=='__main__':
    _interactiveTest()

You can often afford to ignore loading large chunks of the main program if you're only testing a specific part; adjust your architecture as necessary to avoid initializing unneeded parts of the program. This is the reason people might be saying "make your program more modular", and that is what modularity means: small chunks of the program stand alone, letting you reuse them or (in this case) load them separately.

Invoking Interpreter in Program

You can also drop down into an interpreter and pass in the locals (as demonstrated above), at any point in your program. This is sort of "a poor man's debugger" but I find it efficient enough. =)

Monolithic Algorithms

Been there, done that. Sometimes your workflow can't be modularized any further, and things start getting unwieldy.

Your intuition for making checkpoints is a very good one, and the same one I use: If you work in an interpreter environment, or embed an interpreter, you will not have to deal with this issue as often as if you just reran your scripts. Serializing your data can work, but it introduces a large overhead of reading and writing from disk; you want your dataset to remain in memory. Then you can do something like test1 = algorithm(data), test2 = algorithm(data) (this assumes your algorithm is not an in-place algorithm; if it is, use copy-on-write or make a copy of your datastructures before each test).

If you are still having issues after trying all the above, then either perhaps you are either:

• using your real dataset; you should just a smaller test dataset for prototyping!
• using an inefficient algorithm.

As a last resort, you can profile your code to find bottlenecks.

Other

There are probably powerful python debuggers out there. Eclipse has one I think.

Also I'd personally avoid reload <modulename>, which I've always found caused more headaches than it has solved problems.

Answer 2

Make your program more modular. Ideally, the main block of code should look something like

import config
import my_numerics
import post_processing

my_numerics.configure(config.numerics)
values = my_numerics.run()

post_processing.run(values, config.post_processing)

You get the idea. It's then easy to make a 'mock' numerics object which returns pre-computed data, and pass that into post-processing.

---

EDIT: I still don't understand. Is the following accurate pseudocode for your problem?

for _ in range(lots):
    do_slow_thing_one()

for _ in range(many):
    do_slow_thing_two()

for _ in range(lots_many)
    do_slow_thing_three()

That is, you want to interrupt the numerics halfway through their run (not at the end), say at the beginning of the third loop, without having to rerun the first two?

If so, even if the loops don't contain much code, you should modularise the design:

input_data = np.load(some_stuff)
stage_one = do_thing_one(input_data)
stage_two = do_thing_two(stage_one)
stage_three = do_thing_three(stage_two)

The first way of doing this is transferring data between distinct stages through an implicit interface; namely, the dictionary of local variables. This is bad, because you haven't defined which variables are being used and hence you can't mock them for testing/debugging purposes. The second way defines a (rudimentary) interface between your functions. You now no longer care what do_thing_one does, as long as it takes some input data and returns some output data. This means that to debug do_thing_three you can just do

stage_two = np.load(intermediate_stuff)
stage_three = do_thing_three(stage_two)

As long as the data in stage_two is of the correct format, it doesn't matter where it came from.

Answer 3

Joblib handles result caching in a quite transparent way. Here is an example, from their documentation:

>>> from joblib import Memory
>>> mem = Memory(cachedir='/tmp/joblib')
>>> import numpy as np
>>> a = np.vander(np.arange(3))
>>> square = mem.cache(np.square)
>>> b = square(a)
________________________________________________________________________________
[Memory] Calling square...
square(array([[0, 0, 1],
       [1, 1, 1],
       [4, 2, 1]]))
___________________________________________________________square - 0.0s, 0.0min

>>> c = square(a)
>>> # The above call did not trigger an evaluation because the result is cached

The calculation results are automatically saved on disk, so Joblib might suit your needs.