How to use Python/NumPy for a fairly large project?

Question

So far I've been writing one-script Python programs, mainly for text processing and data analysis.

Now I want to bring one of my old Matlab simulation project to Python/NumPy. In that project I have one main program (in a .m file) with a few tens of functions (each in a separate .m file). There were global variables that were used across all functions so that I don't have to input them as arguments for every function. But then I can't run/test individual function without running the main program, because the global variables will then be undefined, or it calls another function in another file. Plus the file organizing is a mess. It is painful to add new functions, changing existing ones, especially the main program.

This time I want to do things properly. I want the program to have a proper architecture, if it's the right word. First I need to know to organize all those functions. I don't think each little function having its own file is a good idea. I think maybe I can divide those functions into several groups and each group can be one file? Will it be a .py file or some other kind of file? Second I would like it to be easily expandable, that I can add new functions easily.

I believe there must be some standard way of doing this, but I have no clue.

One more question: when I run a Matlab program, after it finishes, I still have all the variables in the workspace, so I can still check the numbers, make plots, etc. But when I run my python script through IPython shell, it clears everything afterwards. Is there a similar thing as the workspace?

Answer 1

Any time you have a bunch of functions sharing the same global state:

foo = 1
def do_something1():
    print foo

def do_something2():
    global foo
    foo += 1

you're better off defining a class:

class NoGlobal(object):
     """docstring -- Pick a better name for your class please :)"""

     def __init__(self):
          self.foo = 1

     def do_something1(self):
          print self.foo

     def do_something2(self):
          self.foo += 1

Now you're not sharing global state and you can run your "simulation" as much as you'd like without mucking about with your globals -- Simply instantiate a new class instance and you're ready to start the new simulation.

As far as leaving it so that you can play around with things after your script terminates, that's what the -i option is for:

python -i yourscript.py