Complicated application stack refactor/redesign strategies

Question

I am involved in a project maintenance plan that involves either major refactoring or complete redesign. We have a very complicated existing technology stack that has evolved organically over a period of five years. I myself joined on three years ago, and was 'caught up' in the madness. This stack was put together based on a client's evolving needs, and is fast becoming an unmanageable behemoth consisting of many technologies and uncontrollable collection of data.

Our primary goal is to make the stack more workable and manageable, and come up with a better system to manage data in the future. We would like to convert a tightly integrated Business Model and REST Controller system written primarily in PHP into a coherent REST API.

Our PHP Business Model relies on the eXist XML database, which contains most of the data, and MySQL which supports 2 of our sub-applications' data. We have a library of SQL and XQuery source codes to manage the data in the data stores dynamically. There are also parts of the business model that relies on Java code by using the PHP-Java Bridge and SAXON and FOP to create PDF facsimiles of documents stored in the XML database. Since this is a web application, we use a combination of PHPTAL, XSLT, CSS, XHTML, and JavaScript to facilitate the client UI. Lastly, we have a bunch of housekeeping scripts that manage PHP, Apache, functions written in Perl, that are managed using Ant tasks.

The basic functions of the stack are to facilitate electronic forms for a variety of user models. The simple philosophy behind the application was undermined over the years by a combination of client fiddling, excessive end-user hand holding, lack of systems analyses oversight, bad testing methodologies and lack of future goals and well-defined project scope.

We are currently planning to arrest any further development, better describe the system which may or may not include better unit test plans, and create the foundation for an API using the same technology components. My gut reaction is to come up with a solid scope and redesign the system using a well-supported programming language and framework that is well suited for writing web APIs, possibly loose some functionality, but retain the functionality that matters the most. I would also recommend migrating the existing data to a separate 'read-only' platform.

My question to those who have been in similar situations are:

• how did you get out of this kind of situation?
• what sort of planning steps would be useful to employ in these kinds of situation?
• are there source code management techniques for these kinds of stacks? what do I look for? - search keywords

I realize that there are no perfect solutions, and that to answer these questions, I would need to specify our project issues further. I can not really do that for a variety of reasons. I am having problems finding useful resources to help me approach this issue, and I would appreciate advice on how to proceed.

Answer 1

Here are some questions I would be asking or thinking about if I was in your shoes:

Database design:

• What information do you need to capture?
• How does it all need to be tied together?
• What kind of reporting do you need to do?
• What kind of metrics do you want to collect?
• When should old data be purged permanently to long-term archival?

Application design:

• What can you design into the system now that will make it less likely clients will need to be "hand held"?
• What functionality is currently missing or deficient?
• Are there any easy changes you could make that would expand your customer base?
• Do you have a formal coding standard? Is it enforced?
• What kind of testing do you want to require? (Unit testing, formal QA, etc.)
• What kind of documentation do you want to require? (Source code, end-user, etc.)
• What kind of metrics are you using to determine how well the application(s) are working?
• Looking back three years to when you started - knowing what you know now, what design decisions do you wish you had implemented back then?

Software:

• Is it possible to provide the same services at the same level using a smaller set of technologies?
• What technologies are most familiar to your existing staff?
• If you need to hire more people, what technologies are easiest to bring in on a permanent or consultant basis?

Hardware:

• Does your existing hardware meet your needs? If not, what do you need? More servers? Bigger servers? Load balancers? etc.
• Are you currently taking advantage of virtualization?

Personnel:

• Do you have "buy in" from all the stakeholders? If not all of them, do you at least have buy in from the decision makers?
• Do the stakeholders understand how big an undertaking this will be?

Disaster prevention/recovery:

• How will you provide redundancy in the event of the primary server(s)/network(s) being unavailable?
• How will you backup the data (onsite, offsite, short-term, long-term)?
• What kind of monitoring do you need to set up? How should you be notified in the event of a problem? Who should be notified?

Sorry, I couldn't fit that into a comment :-) And I'm out of time to elaborate on any of it, I was just killing time waiting for a meeting.