Best practices on managing complexity/visualizing components in your software?

Question

We're building tools to mine information from the web. We have several pieces, such as

• Crawl data from the web
• Extract information based on templates & business rules
• Parse results into database
• Apply normalization & filtering rules
• Etc, etc.

The problem is troubleshooting issues & having a good "high-level picture" of what's happening at each stage.

What techniques have helped you understand and manage complex processes?

• Use workflow tools like Windows Workflow foundation
• Encapsulate separate functions into command-line tools & use scripting tools to link them together
• Write a Domain-Specific Language (DSL) to specify what order things should happen at a higher level.

Just curious how you get a handle on a system with many interacting components. We'd like document/understand how the system works at a higher level than tracing through the source code.

Answer 1

I use AT&T's famous Graphviz, its simple and does the work nicely. Its the same library Doxygen uses too.

Also if you make a little effort you can get very nice looking graphs.

Forgot to mention, the way I use it is as follows (because Graphviz parses Graphviz scripts), I use an alternative system to log events in Graphviz format, so I then just parse the Logs file and get a nice graph.

Answer 2

The code says what happens at each stage. Using a DSL would be a boon, but possibly not if it comes at the cost of writing your own scripting-language and/or compiler.

Higher level documentation should not include details of what happens at each step; it should provide an overview of the steps and how they relate together.

Good tips:

• Visualize your database schema relations.
• Use visio or other tools (like the one you mentioned - haven't used it) for process overviews (imho it belongs to the specification of your project).
• Make sure your code is properly structured / compartmentalized / etc.
• Make sure you have some sort of project specification (or some other "general" documentation that explains what the system does on an abstract level).

I wouldn't recommend building command-line tools unless you actually have a use for them. No need in maintaining tools you don't use. (That's not the same as saying it can't be useful; but most of what you do sounds more like it belongs in a library rather than executing external processes).

Answer 3

I find a dependency structure matrix a helpful way to analyze the structure of an application. A tool like lattix could help.

Depending on your platform and toolchain there are many really useful static analysis packages that could help you to document the relationships between subsystems or components of your application. For the .NET platform, NDepend is a good example. There are many other for other platforms though.

Having a good design or model before building the system is the best way to have an understanding for the team of how the application should be structured but tools like those I mentioned can help enforce architectural rules and will often give you insights into the design that just trawling through the code cannot.