How to build large applications

Question

I think I've become quite good at the basics of programming (for a variety of languages). I can write a good line of code. I can write a good method. I can write a good class. I can write a good group of classes. I can write good small or medium application.

I do not however know how to build a good large application. Particularly in the case where multiple technologies are involved and more are likely to become involved with time. Say a project with a large web front-end, a large server back-end that connects to some other integration back-end and finally a large and complex database. Oh, I've been involved in a few of these applications and I could build one I'm sure. I'm not so sure however that it could qualify as "good".

My question is thus for a reference to a book or other good source of reading where I could learn how to distribute and organize code and data for general large projects. For example, would I want to layer things very strictly or would I want to encapsulate it independent units instead? Would I want to try to keep most of the logic in the same pool, or should it just be distributed as it seems most logical when adding whatever feature I'm adding?

I've seen lots of general principals on these issues (e.g. No spaghetti code, meatball code...) and read a few excellent articles that discuss the matter but I've never encountered a source which would lead me to concrete practical knowledge. I realize the difficulty of the question and so I'd be happy to just hear about the readings that others have found to help them in their quest for such knowledge.

As always, thank you for your replies.

****Given the debated nature of the definition of "good" code, the term "good" in this context won't be defined (it means whatever you think it ought to mean).

Answer 1

As programmers, we like to believe we are smart people, so it's hard to admit that something is too big and complex to even think about all at once. But for a large-scale software project it's true, and the sooner you acknowledge your finite brain capacity and start coming up with ways to simplify the problem, the better off you'll be.

The other main thing to realise is that you will spend most of your time changing existing code. Building the initial codebase is just the honeymoon period -- you need to design your code with the idea in mind that, 6 months later you will be sitting in front of it trying to solve some problem without a clue how this particular module works, even though you wrote it yourself.

So, what can we do?

Minimise coupling between unrelated parts of your code. Code is going to change over time in ways you can't anticipate -- there will be showstopper problems integrating with unfamiliar products, requirements changes -- and those will cause ripple-on changes. If you have established stable interfaces and coded to them, you can make any changes you need in the implementation without those changes affecting code that uses the interface. You need to spend time and effort developing interfaces that will stand the test of time -- if an interface needs to change too, you're back to square one.

Establish automated tests that you can use for regression testing. Yes, it's a lot of work up front. But it will pay off in the future when you can make a change, run the tests, and establish that it still works without that anxious feeling of wondering if everything will fall over if you commit your latest change to source control.

Lay off the tricky stuff. Every now and then I see some clever C++ template trick and think, "Wow! That's just what my code needs!" But the truth is, the decrease in how readable and readily understandable the code becomes is often simply not worth the increased genericity. If you're someone like me whose natural inclination is to try to solve every problem in as general a manner as possible, you need to learn to restrain it until you actually come across the need for that general solution. If that need arises, you might have to rewrite some code -- it's no big deal.